refactor(8.0)!: remove orphaned zero-config subsystem + dead cloud/progressive-init storage vestige
The old config-generation subsystem (src/config/ + autoConfiguration.ts) was
superseded during the 8.0 rework and never wired into init(): it emitted settings
for a partitioning subsystem that no longer exists and probed deleted cloud env
vars. The live zero-config path is inline — recall preset → HNSW knobs, storage
auto-detect, auto persistMode, container-memory-aware cache sizing.
The storage progressive-init / cloud-detection cluster was equally dead after the
cloud adapters were dropped: isCloudStorage() is permanently false (no overriders),
scheduleBackgroundInit/runBackgroundInit were never called (the latter an empty
body), initMode was never assigned, and Brainy.isFullyInitialized()/
awaitBackgroundInit() were always-trivial with zero callers. scheduleCountPersist()
collapses to its only-ever-taken immediate write-through path.
Removed:
- src/config/{index,zeroConfig,storageAutoConfig,modelAutoConfig,sharedConfigManager}.ts
- src/utils/autoConfiguration.ts + the inert BrainyZeroConfig export
- Brainy.isFullyInitialized()/awaitBackgroundInit() (+ BrainyInterface decls)
- InitMode type, isCloudStorage/detectCloudEnvironment/resolveInitMode,
scheduleBackgroundInit/runBackgroundInit/ensureValidatedForWrite and their state
- Dead cloud env-var probes (K_SERVICE/K_REVISION/AWS_LAMBDA_FUNCTION_NAME/
FUNCTIONS_TARGET/AZURE_FUNCTIONS_ENVIRONMENT)
Kept (verified live): production-detection logging (environment.ts), container-
memory cache sizing (memoryDetection/paramValidation), on-disk hash bucketing
(sharding.ts).
Docs: scrubbed deleted-subsystem references (JS quantization knobs, cloud/OPFS
adapters, partitioning, old zero-config API) across 14 files; deleted two wholly-
obsolete feature docs (complete-feature-list, v3-features); rewrote
architecture/zero-config for 8.0.
~3,700 LOC removed. Build clean; 1392 unit + 24 db-mvcc green.
This commit is contained in:
parent
00d3203d68
commit
35b9d7ef43
28 changed files with 596 additions and 3752 deletions
|
|
@ -428,8 +428,7 @@ brainy inspect health /data/brain
|
|||
|
||||
See [the multi-process model](docs/concepts/multi-process.md) and the
|
||||
[inspection guide](docs/guides/inspection.md) for the full story, including
|
||||
stale-lock detection, the cross-process flush RPC, and what's not yet
|
||||
enforced on cloud storage backends.
|
||||
stale-lock detection and the cross-process flush RPC.
|
||||
|
||||
## Contributing
|
||||
|
||||
|
|
|
|||
|
|
@ -391,13 +391,12 @@ await brain.init() // Instant (0-10ms)
|
|||
|
||||
### Vector Index Tuning Knobs
|
||||
|
||||
Brainy 8.0 exposes exactly three knobs on `config.vector`:
|
||||
Brainy 8.0 exposes two knobs on `config.vector`:
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
vector: {
|
||||
recall: 'fast', // 'fast' | 'balanced' | 'accurate'
|
||||
quantization: { bits: 8 }, // 4 | 8 (SQ4 / SQ8)
|
||||
persistMode: 'deferred' // 'immediate' | 'deferred'
|
||||
}
|
||||
})
|
||||
|
|
|
|||
|
|
@ -35,10 +35,9 @@ Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery,
|
|||
The three knobs that matter most:
|
||||
|
||||
1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
|
||||
2. **`config.vector.quantization`** — `{ bits: 4 | 8 }` for memory savings on the open-core JS vector index
|
||||
3. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
|
||||
2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
|
||||
|
||||
The native vector provider (via the optional `@soulcraft/cortex` package) extends this with a higher-performing index when installed.
|
||||
The native vector provider (via the optional `@soulcraft/cortex` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
|
||||
|
||||
## Storage Configurations
|
||||
|
||||
|
|
@ -95,7 +94,6 @@ const brain = new Brainy({
|
|||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'fast', // Trade recall for latency
|
||||
quantization: { bits: 8 }, // SQ8 quantization
|
||||
persistMode: 'deferred' // Batch persistence
|
||||
}
|
||||
})
|
||||
|
|
@ -142,8 +140,7 @@ Your service layer handles routing and isolation; Brainy stays simple.
|
|||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'accurate',
|
||||
quantization: { bits: 8 }
|
||||
recall: 'accurate'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
|
@ -153,7 +150,6 @@ const brain = new Brainy({
|
|||
| Setting | Values | When to change |
|
||||
|---------|--------|----------------|
|
||||
| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
|
||||
| `vector.quantization.bits` | `4` / `8` | Smaller index, lower memory |
|
||||
| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
|
||||
| `storage.cache.maxSize` | integer | Hot-path read cache size |
|
||||
| `storage.cache.ttl` | ms | Cache freshness |
|
||||
|
|
@ -174,14 +170,13 @@ const stats = await brain.stats()
|
|||
|
||||
### Issue: Slow queries
|
||||
1. Switch to `vector.recall: 'fast'`
|
||||
2. Enable SQ8 quantization
|
||||
3. Increase the read cache (`storage.cache.maxSize`)
|
||||
4. Consider the optional native vector provider via `@soulcraft/cortex`
|
||||
2. Increase the read cache (`storage.cache.maxSize`)
|
||||
3. Consider the optional native vector provider via `@soulcraft/cortex`
|
||||
|
||||
### Issue: Memory pressure
|
||||
1. Enable `vector.quantization: { bits: 4 }` or `{ bits: 8 }`
|
||||
2. Reduce `storage.cache.maxSize`
|
||||
3. Move to `vector.persistMode: 'deferred'` to batch writes
|
||||
1. Reduce `storage.cache.maxSize`
|
||||
2. Move to `vector.persistMode: 'deferred'` to batch writes
|
||||
3. Consider the optional native vector provider via `@soulcraft/cortex` for at-scale index acceleration
|
||||
|
||||
### Issue: Slow startup after a crash
|
||||
1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
|
||||
|
|
@ -198,5 +193,5 @@ const stats = await brain.stats()
|
|||
|
||||
- Brainy 8.0 is a **library**, not a cluster
|
||||
- Storage adapters: `filesystem`, `memory`, `auto`
|
||||
- Vector tuning: `recall`, `quantization`, `persistMode`
|
||||
- Vector tuning: `recall`, `persistMode`
|
||||
- Backup is an operator-layer concern — snapshot `rootDirectory`
|
||||
|
|
|
|||
|
|
@ -1441,10 +1441,9 @@ const brain = new Brainy({
|
|||
rootDirectory: './brainy-data'
|
||||
},
|
||||
|
||||
// Vector index configuration (3 knobs)
|
||||
// Vector index configuration (2 knobs)
|
||||
vector: {
|
||||
recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
|
||||
quantization: { bits: 8 }, // 4 | 8 (SQ4 / SQ8)
|
||||
persistMode: 'immediate' // 'immediate' | 'deferred'
|
||||
},
|
||||
|
||||
|
|
|
|||
|
|
@ -47,16 +47,13 @@ await visualizationAugmentation.graphRelationships(authors)
|
|||
|
||||
#### 2. **Data Portability**
|
||||
```typescript
|
||||
// Export from one Brainy instance
|
||||
const data = await brain1.export()
|
||||
// Snapshot from one Brainy instance, restore into another —
|
||||
// types are universally understood
|
||||
const pin = brain1.now()
|
||||
await pin.persist('/snapshots/brain1')
|
||||
await pin.release()
|
||||
|
||||
// Import to another—types are universally understood
|
||||
await brain2.import(data)
|
||||
|
||||
// Or sync between different storage backends
|
||||
const cloudBrain = new Brainy({ storage: 's3' })
|
||||
const localBrain = new Brainy({ storage: 'filesystem' })
|
||||
await cloudBrain.sync(localBrain) // Types match perfectly
|
||||
const brain2 = await Brainy.load('/snapshots/brain1') // Types match perfectly
|
||||
```
|
||||
|
||||
#### 3. **AI Model Compatibility**
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ brainy-data/
|
|||
### Vector Index
|
||||
Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph:
|
||||
- **Performance**: O(log n) search complexity
|
||||
- **Memory Efficient**: SQ4/SQ8 scalar quantization support
|
||||
- **Configurable recall**: `fast` / `balanced` / `accurate` presets trade recall for latency
|
||||
- **Scalable**: Handles millions of vectors per process
|
||||
- **Persistent**: Serializable to storage
|
||||
- **Swappable**: Replace with a native implementation (such as `@soulcraft/cortex`) via the plugin system without changing application code
|
||||
|
|
|
|||
|
|
@ -13,776 +13,145 @@ next:
|
|||
|
||||
# Zero Configuration & Auto-Adaptation
|
||||
|
||||
> **Status (8.0):** This document predates Brainy 8.0 and needs a rewrite before
|
||||
> republication. Large parts describe storage backends and environments that 8.0
|
||||
> removed (browser/OPFS/IndexedDB, edge KV, S3) or features that were never built
|
||||
> (model auto-selection, workload detection). Brainy 8.0 is server-only (Node.js/Bun)
|
||||
> with two storage adapters: `memory` and `filesystem` — see
|
||||
> [Storage Adapters](../concepts/storage-adapters.md) for the accurate story.
|
||||
> Basic zero-config (`new Brainy()` with auto-selected storage) works as described.
|
||||
> **"Zero config by default, fully tunable when you need it."** Construct a
|
||||
> `Brainy()` with no options and it picks sensible, environment-aware defaults.
|
||||
> Every default below is overridable through the constructor — see the
|
||||
> [API Reference](../api/README.md#configuration).
|
||||
|
||||
## Overview
|
||||
|
||||
Brainy is designed with **"Zero Config by Default, Infinite Tunability"** philosophy. It automatically detects your environment, adapts to available resources, learns from usage patterns, and optimizes itself for your specific workload—all without any configuration.
|
||||
Brainy 8.0 is server-only (Node.js 22+ / Bun). With no configuration it:
|
||||
|
||||
## Zero Configuration Magic
|
||||
- selects a storage adapter from the runtime,
|
||||
- initializes the embedding model (all-MiniLM-L6-v2, 384 dimensions),
|
||||
- builds and maintains the metadata, graph, and vector indexes,
|
||||
- sizes its caches and write buffers to the detected memory budget,
|
||||
- chooses a persistence mode that matches the storage backend, and
|
||||
- quiets its own logging when it detects a production environment.
|
||||
|
||||
### Instant Start
|
||||
There is no public config-generation function — adaptation happens inside the
|
||||
constructor and `init()`.
|
||||
|
||||
## Instant Start
|
||||
|
||||
```typescript
|
||||
import { Brainy } from 'brainy'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// That's it. No config needed.
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Brainy automatically:
|
||||
// ✓ Detects environment (Node.js, Browser, Edge, Deno)
|
||||
// ✓ Chooses optimal storage (FileSystem, OPFS, Memory)
|
||||
// ✓ Downloads required models (if needed)
|
||||
// ✓ Configures vector dimensions (384 optimal)
|
||||
// ✓ Sets up indexing strategies
|
||||
// ✓ Enables appropriate augmentations
|
||||
// ✓ Configures caching layers
|
||||
// ✓ Optimizes for your hardware
|
||||
await brain.add({ data: 'First entity', type: 'concept' })
|
||||
const results = await brain.find('first')
|
||||
```
|
||||
|
||||
### Environment Detection ✅ Available
|
||||
## What Auto-Adaptation Covers
|
||||
|
||||
Brainy automatically detects and adapts to your runtime:
|
||||
### 1. Storage auto-detection
|
||||
|
||||
With no `storage` option, Brainy uses `type: 'auto'`:
|
||||
|
||||
- **Filesystem** when running on a runtime with a writable Node filesystem and a
|
||||
resolvable root directory. This is the default for typical Node/Bun servers and
|
||||
persists across restarts.
|
||||
- **In-memory** otherwise (no filesystem access, or an explicit memory request).
|
||||
Fast, zero I/O, discarded on process exit — ideal for tests and ephemeral
|
||||
caches.
|
||||
|
||||
8.0 ships exactly two storage adapters — `memory` and `filesystem` — plus the
|
||||
`auto` selector that resolves to one of them. See
|
||||
[Storage Adapters](../concepts/storage-adapters.md) for the full contract.
|
||||
|
||||
```typescript
|
||||
// Brainy's environment detection
|
||||
const environment = {
|
||||
// Runtime detection
|
||||
isNode: typeof process !== 'undefined',
|
||||
isBrowser: typeof window !== 'undefined',
|
||||
isDeno: typeof Deno !== 'undefined',
|
||||
isEdge: typeof EdgeRuntime !== 'undefined',
|
||||
isWebWorker: typeof WorkerGlobalScope !== 'undefined',
|
||||
|
||||
// Capability detection
|
||||
hasFileSystem: /* auto-detected */,
|
||||
hasIndexedDB: /* auto-detected */,
|
||||
hasOPFS: /* auto-detected */,
|
||||
hasWebGPU: /* auto-detected */,
|
||||
hasWASM: /* auto-detected */,
|
||||
|
||||
// Resource detection
|
||||
cpuCores: /* auto-detected */,
|
||||
memory: /* auto-detected */,
|
||||
storage: /* auto-detected */
|
||||
}
|
||||
```
|
||||
|
||||
## Auto-Adaptive Storage ✅ Available
|
||||
|
||||
> **Current**: Brainy automatically selects the best storage adapter for your environment.
|
||||
|
||||
### Storage Selection Logic
|
||||
|
||||
```typescript
|
||||
// Brainy's intelligent storage selection
|
||||
async function autoSelectStorage() {
|
||||
// Server environments
|
||||
if (environment.isNode) {
|
||||
if (await hasWritePermission('./data')) {
|
||||
return 'filesystem' // Best for servers
|
||||
} else if (process.env.S3_BUCKET) {
|
||||
return 's3' // Cloud deployment
|
||||
} else {
|
||||
return 'memory' // Fallback for restricted environments
|
||||
}
|
||||
}
|
||||
|
||||
// Browser environments
|
||||
if (environment.isBrowser) {
|
||||
if (await navigator.storage.estimate() > 1GB) {
|
||||
return 'opfs' // Best for modern browsers
|
||||
} else if (indexedDB) {
|
||||
return 'indexeddb' // Fallback for older browsers
|
||||
} else {
|
||||
return 'memory' // In-memory for restricted contexts
|
||||
}
|
||||
}
|
||||
|
||||
// Edge environments
|
||||
if (environment.isEdge) {
|
||||
return 'kv' // Use edge KV stores (Cloudflare, Vercel)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Storage Migration
|
||||
|
||||
Brainy seamlessly migrates between storage types:
|
||||
|
||||
```typescript
|
||||
// Start with memory storage (development)
|
||||
const brain = new Brainy() // Auto-selects memory
|
||||
|
||||
// Later, migrate to production storage
|
||||
await brain.migrate({
|
||||
to: 'filesystem',
|
||||
path: './production-data'
|
||||
// Explicit override when you want a specific root
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './brainy-data' }
|
||||
})
|
||||
// All data seamlessly transferred
|
||||
```
|
||||
|
||||
## Learning & Optimization 🚧 Coming Soon
|
||||
### 2. HNSW quality from the `recall` preset
|
||||
|
||||
> **Note**: These features are planned for Q2 2025. Currently, Brainy uses static optimizations.
|
||||
|
||||
### Query Pattern Learning 🚧 Planned
|
||||
|
||||
Brainy learns from your query patterns and optimizes accordingly:
|
||||
Vector-index quality comes from a single preset rather than hand-tuned graph
|
||||
parameters. `config.vector.recall` accepts `'fast'`, `'balanced'`, or
|
||||
`'accurate'` and defaults to `'balanced'`. The preset maps internally to the
|
||||
HNSW construction and search parameters (`M` / `efConstruction` / `efSearch`),
|
||||
so you trade recall against latency with one knob instead of three.
|
||||
|
||||
```typescript
|
||||
// Brainy observes query patterns
|
||||
class QueryPatternLearner {
|
||||
analyze(queries: Query[]) {
|
||||
return {
|
||||
// Frequency analysis
|
||||
mostCommonFields: this.getTopFields(queries),
|
||||
avgResultSize: this.getAvgSize(queries),
|
||||
temporalPatterns: this.getTimePatterns(queries),
|
||||
|
||||
// Relationship analysis
|
||||
commonTraversals: this.getGraphPatterns(queries),
|
||||
typicalDepth: this.getAvgDepth(queries),
|
||||
|
||||
// Performance analysis
|
||||
slowQueries: this.getSlowQueries(queries),
|
||||
cacheability: this.getCacheability(queries)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Automatic optimizations based on learning:
|
||||
// - Creates indexes for frequently queried fields
|
||||
// - Pre-computes common graph traversals
|
||||
// - Adjusts cache sizes based on working set
|
||||
// - Optimizes vector search parameters
|
||||
const brain = new Brainy({
|
||||
vector: { recall: 'fast' } // favor latency over recall
|
||||
})
|
||||
```
|
||||
|
||||
### Auto-Indexing 🚧 Planned
|
||||
The default JS index is `JsHnswVectorIndex`. An optional native acceleration
|
||||
provider (the `@soulcraft/cortex` package) can replace it with a
|
||||
higher-performing implementation; the public knobs stay the same. Quantization
|
||||
and other index-internal acceleration are the native provider's concern, not a
|
||||
Brainy configuration option.
|
||||
|
||||
Brainy automatically creates indexes based on usage:
|
||||
### 3. Persistence mode follows the backend
|
||||
|
||||
`config.vector.persistMode` accepts `'immediate'` or `'deferred'`. Left unset,
|
||||
Brainy chooses for you:
|
||||
|
||||
- **Immediate** on filesystem storage, so the index file stays in lock-step with
|
||||
the data and survives a crash.
|
||||
- **Deferred** on in-memory storage, where there is nothing durable to sync to,
|
||||
so writes are batched for throughput.
|
||||
|
||||
```typescript
|
||||
// No manual index configuration needed
|
||||
await brain.find({ where: { category: "tech" } }) // First query
|
||||
// Brainy notices 'category' field usage
|
||||
|
||||
await brain.find({ where: { category: "science" } }) // Second query
|
||||
// Pattern detected - auto-creates category index
|
||||
|
||||
await brain.find({ where: { category: "tech" } }) // Third query
|
||||
// Now using index - 100x faster!
|
||||
const brain = new Brainy({
|
||||
vector: { persistMode: 'deferred' } // batch persistence for write-heavy loads
|
||||
})
|
||||
```
|
||||
|
||||
### Adaptive Caching 🚧 Planned
|
||||
### 4. Memory-aware cache and buffer sizing
|
||||
|
||||
Cache strategies adapt to your access patterns:
|
||||
Brainy reads the container's memory budget — `CLOUD_RUN_MEMORY`, `MEMORY_LIMIT`,
|
||||
or the cgroup memory limit when running in a container — and sizes its read
|
||||
caches and write buffers to fit. On a small instance it stays conservative; on a
|
||||
large one it uses more of the available headroom. Query-result limits are capped
|
||||
against the same budget (roughly 25 KB per result) to keep a single oversized
|
||||
query from exhausting memory.
|
||||
|
||||
You can pin the cache explicitly:
|
||||
|
||||
```typescript
|
||||
class AdaptiveCache {
|
||||
async adapt(metrics: AccessMetrics) {
|
||||
if (metrics.hitRate < 0.3) {
|
||||
// Low hit rate - switch strategy
|
||||
this.strategy = 'lfu' // Least Frequently Used
|
||||
} else if (metrics.workingSet > this.size) {
|
||||
// Working set too large - increase size
|
||||
this.size = Math.min(metrics.workingSet * 1.5, maxMemory)
|
||||
} else if (metrics.temporalLocality > 0.8) {
|
||||
// High temporal locality - use time-based eviction
|
||||
this.strategy = 'ttl'
|
||||
this.ttl = metrics.avgAccessInterval * 2
|
||||
}
|
||||
}
|
||||
}
|
||||
const brain = new Brainy({
|
||||
cache: { maxSize: 10000, ttl: 3_600_000 }
|
||||
})
|
||||
```
|
||||
|
||||
## Performance Auto-Scaling 🚧 Coming Soon
|
||||
### 5. Logging quiets in production
|
||||
|
||||
### Dynamic Batch Sizing
|
||||
|
||||
Brainy adjusts batch sizes based on system load:
|
||||
|
||||
```typescript
|
||||
class DynamicBatcher {
|
||||
calculateOptimalBatch() {
|
||||
const cpuUsage = process.cpuUsage()
|
||||
const memoryUsage = process.memoryUsage()
|
||||
|
||||
if (cpuUsage < 30 && memoryUsage < 50) {
|
||||
return 1000 // System idle - large batches
|
||||
} else if (cpuUsage < 60 && memoryUsage < 70) {
|
||||
return 100 // Moderate load - medium batches
|
||||
} else {
|
||||
return 10 // High load - small batches
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Automatically applied during bulk operations
|
||||
for (const item of millionItems) {
|
||||
await brain.add(item) // Internally batched optimally
|
||||
}
|
||||
```
|
||||
|
||||
### Memory Management
|
||||
|
||||
Automatic memory pressure handling:
|
||||
|
||||
```typescript
|
||||
class MemoryManager {
|
||||
async handlePressure() {
|
||||
const usage = process.memoryUsage()
|
||||
const available = os.freemem()
|
||||
|
||||
if (available < 100 * 1024 * 1024) { // Less than 100MB free
|
||||
// Emergency mode
|
||||
await this.flushCaches()
|
||||
await this.compactIndexes()
|
||||
await this.offloadToDisk()
|
||||
} else if (usage.heapUsed / usage.heapTotal > 0.9) {
|
||||
// Preventive mode
|
||||
await this.reduceCacheSizes()
|
||||
await this.pauseBackgroundTasks()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Connection Pooling
|
||||
|
||||
Automatic connection management for storage backends:
|
||||
|
||||
```typescript
|
||||
class ConnectionPool {
|
||||
async getOptimalPoolSize() {
|
||||
// Adapts based on workload
|
||||
const metrics = await this.getMetrics()
|
||||
|
||||
if (metrics.waitTime > 100) {
|
||||
// Queries waiting - increase pool
|
||||
this.size = Math.min(this.size * 1.5, this.maxSize)
|
||||
} else if (metrics.idleConnections > this.size * 0.5) {
|
||||
// Too many idle - decrease pool
|
||||
this.size = Math.max(this.size * 0.7, this.minSize)
|
||||
}
|
||||
|
||||
return this.size
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Model Auto-Selection
|
||||
|
||||
### Embedding Model Selection
|
||||
|
||||
Brainy chooses the best embedding model for your use case:
|
||||
|
||||
```typescript
|
||||
async function autoSelectModel(data: Sample[]) {
|
||||
const analysis = {
|
||||
languages: detectLanguages(data),
|
||||
domainSpecific: detectDomain(data),
|
||||
averageLength: getAvgLength(data),
|
||||
requiresMultilingual: languages.length > 1
|
||||
}
|
||||
|
||||
if (analysis.requiresMultilingual) {
|
||||
return 'multilingual-e5-base' // Handles 100+ languages
|
||||
} else if (analysis.domainSpecific === 'code') {
|
||||
return 'codebert-base' // Optimized for code
|
||||
} else if (analysis.averageLength > 512) {
|
||||
return 'all-mpnet-base-v2' // Better for long text
|
||||
} else {
|
||||
return 'all-MiniLM-L6-v2' // Fast and efficient default
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Model Downloading
|
||||
|
||||
Models are automatically downloaded when needed:
|
||||
|
||||
```typescript
|
||||
// First use - model auto-downloads
|
||||
const brain = new Brainy()
|
||||
await brain.init() // Downloads model if not cached
|
||||
|
||||
// Intelligent model caching
|
||||
const modelCache = {
|
||||
location: process.env.MODEL_CACHE || '~/.brainy/models',
|
||||
maxSize: 5 * 1024 * 1024 * 1024, // 5GB max
|
||||
strategy: 'lru', // Least recently used eviction
|
||||
|
||||
// CDN selection based on location
|
||||
cdn: await selectFastestCDN([
|
||||
'https://cdn.brainy.io',
|
||||
'https://brainy.b-cdn.net',
|
||||
'https://models.huggingface.co'
|
||||
])
|
||||
}
|
||||
```
|
||||
|
||||
## Workload Detection
|
||||
|
||||
### Pattern Recognition
|
||||
|
||||
Brainy identifies your workload type and optimizes:
|
||||
|
||||
```typescript
|
||||
enum WorkloadType {
|
||||
OLTP = 'oltp', // Many small transactions
|
||||
OLAP = 'olap', // Analytical queries
|
||||
STREAMING = 'streaming', // Real-time ingestion
|
||||
BATCH = 'batch', // Bulk processing
|
||||
HYBRID = 'hybrid' // Mixed workload
|
||||
}
|
||||
|
||||
class WorkloadDetector {
|
||||
detect(metrics: OperationMetrics): WorkloadType {
|
||||
if (metrics.writesPerSecond > 1000 && metrics.avgWriteSize < 1024) {
|
||||
return WorkloadType.STREAMING
|
||||
} else if (metrics.avgQueryComplexity > 0.8 && metrics.avgResultSize > 10000) {
|
||||
return WorkloadType.OLAP
|
||||
} else if (metrics.batchOperations > metrics.singleOperations) {
|
||||
return WorkloadType.BATCH
|
||||
} else if (metrics.writeReadRatio > 0.3 && metrics.writeReadRatio < 0.7) {
|
||||
return WorkloadType.HYBRID
|
||||
} else {
|
||||
return WorkloadType.OLTP
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Optimization Strategies
|
||||
|
||||
Different optimizations for different workloads:
|
||||
|
||||
```typescript
|
||||
class WorkloadOptimizer {
|
||||
optimize(workload: WorkloadType) {
|
||||
switch (workload) {
|
||||
case WorkloadType.STREAMING:
|
||||
return {
|
||||
entityRegistry: true, // Deduplication
|
||||
batchSize: 1000,
|
||||
walEnabled: true,
|
||||
cacheSize: 'small',
|
||||
indexStrategy: 'lazy'
|
||||
}
|
||||
|
||||
case WorkloadType.OLAP:
|
||||
return {
|
||||
entityRegistry: false,
|
||||
batchSize: 10000,
|
||||
walEnabled: false,
|
||||
cacheSize: 'large',
|
||||
indexStrategy: 'eager',
|
||||
parallelQueries: true
|
||||
}
|
||||
|
||||
case WorkloadType.BATCH:
|
||||
return {
|
||||
entityRegistry: false,
|
||||
batchSize: 50000,
|
||||
walEnabled: false,
|
||||
cacheSize: 'minimal',
|
||||
indexStrategy: 'deferred'
|
||||
}
|
||||
|
||||
default:
|
||||
return this.defaultConfig
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Hardware Adaptation 🚧 Coming Soon
|
||||
|
||||
> **Note**: GPU acceleration and hardware optimization planned for Q3 2025.
|
||||
|
||||
### CPU Optimization
|
||||
|
||||
Adapts to available CPU resources:
|
||||
|
||||
```typescript
|
||||
class CPUAdapter {
|
||||
async optimize() {
|
||||
const cores = os.cpus().length
|
||||
const type = os.cpus()[0].model
|
||||
|
||||
// Parallel processing based on cores
|
||||
this.parallelism = Math.max(1, cores - 1) // Leave one core free
|
||||
|
||||
// SIMD detection for vector operations
|
||||
if (type.includes('Intel') || type.includes('AMD')) {
|
||||
this.enableSIMD = await checkSIMDSupport()
|
||||
}
|
||||
|
||||
// Thread pool sizing
|
||||
this.threadPoolSize = cores * 2 // Optimal for I/O bound
|
||||
|
||||
// Vector search optimization
|
||||
if (cores >= 8) {
|
||||
this.hnswConstruction = 200 // Higher quality index
|
||||
this.hnswSearch = 100 // More accurate search
|
||||
} else {
|
||||
this.hnswConstruction = 100 // Balanced
|
||||
this.hnswSearch = 50 // Faster search
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Memory Adaptation
|
||||
|
||||
Intelligent memory allocation:
|
||||
|
||||
```typescript
|
||||
class MemoryAdapter {
|
||||
async configure() {
|
||||
const totalMemory = os.totalmem()
|
||||
const availableMemory = os.freemem()
|
||||
|
||||
// Allocate based on available memory
|
||||
const allocation = {
|
||||
cache: Math.min(availableMemory * 0.25, 2 * GB),
|
||||
vectors: Math.min(availableMemory * 0.30, 4 * GB),
|
||||
indexes: Math.min(availableMemory * 0.20, 2 * GB),
|
||||
working: Math.min(availableMemory * 0.25, 2 * GB)
|
||||
}
|
||||
|
||||
// Adjust for low memory systems
|
||||
if (totalMemory < 4 * GB) {
|
||||
allocation.cache *= 0.5
|
||||
allocation.vectors *= 0.7
|
||||
this.enableSwapping = true
|
||||
}
|
||||
|
||||
return allocation
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### GPU Acceleration
|
||||
|
||||
Automatic GPU detection and utilization:
|
||||
|
||||
```typescript
|
||||
class GPUAdapter {
|
||||
async detect() {
|
||||
// WebGPU in browsers
|
||||
if (navigator?.gpu) {
|
||||
const adapter = await navigator.gpu.requestAdapter()
|
||||
return {
|
||||
available: true,
|
||||
type: 'webgpu',
|
||||
memory: adapter.limits.maxBufferSize,
|
||||
compute: adapter.limits.maxComputeWorkgroupsPerDimension
|
||||
}
|
||||
}
|
||||
|
||||
// CUDA in Node.js
|
||||
if (process.platform === 'linux' || process.platform === 'win32') {
|
||||
const hasCuda = await checkCudaSupport()
|
||||
if (hasCuda) {
|
||||
return {
|
||||
available: true,
|
||||
type: 'cuda',
|
||||
memory: await getCudaMemory(),
|
||||
compute: await getCudaCores()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { available: false }
|
||||
}
|
||||
|
||||
async optimize(gpu: GPUInfo) {
|
||||
if (gpu.available) {
|
||||
// Offload vector operations to GPU
|
||||
this.vectorOps = 'gpu'
|
||||
this.embeddingGeneration = 'gpu'
|
||||
this.matrixMultiplication = 'gpu'
|
||||
|
||||
// Larger batch sizes for GPU
|
||||
this.batchSize = gpu.memory > 8 * GB ? 10000 : 1000
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Network Adaptation
|
||||
|
||||
### Bandwidth Detection
|
||||
|
||||
Optimizes for available network bandwidth:
|
||||
|
||||
```typescript
|
||||
class NetworkAdapter {
|
||||
async measureBandwidth() {
|
||||
const testSize = 1 * MB
|
||||
const start = Date.now()
|
||||
await this.transfer(testSize)
|
||||
const duration = Date.now() - start
|
||||
|
||||
const bandwidth = (testSize / duration) * 1000 // bytes/sec
|
||||
|
||||
if (bandwidth < 1 * MB) {
|
||||
// Low bandwidth - optimize
|
||||
this.compression = 'aggressive'
|
||||
this.batchTransfers = true
|
||||
this.cacheRemote = true
|
||||
} else if (bandwidth > 100 * MB) {
|
||||
// High bandwidth
|
||||
this.compression = 'minimal'
|
||||
this.parallelTransfers = true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Latency Optimization
|
||||
|
||||
Adapts to network latency:
|
||||
|
||||
```typescript
|
||||
class LatencyOptimizer {
|
||||
async optimize() {
|
||||
const latency = await this.measureLatency()
|
||||
|
||||
if (latency > 100) { // High latency
|
||||
// Batch operations
|
||||
this.minBatchSize = 100
|
||||
|
||||
// Aggressive prefetching
|
||||
this.prefetchDepth = 3
|
||||
|
||||
// Local caching
|
||||
this.cacheStrategy = 'aggressive'
|
||||
|
||||
// Connection pooling
|
||||
this.connectionPool = Math.min(latency / 10, 50)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Cloud Provider Detection 🚧 Coming Soon
|
||||
|
||||
> **Note**: Cloud provider auto-detection planned for Q3 2025.
|
||||
|
||||
### Automatic Cloud Optimization
|
||||
|
||||
Detects and optimizes for cloud providers:
|
||||
|
||||
```typescript
|
||||
class CloudDetector {
|
||||
async detect() {
|
||||
// AWS Detection
|
||||
if (process.env.AWS_REGION || await canReachMetadata('169.254.169.254')) {
|
||||
return {
|
||||
provider: 'aws',
|
||||
instance: await getEC2InstanceType(),
|
||||
region: process.env.AWS_REGION,
|
||||
services: {
|
||||
storage: 's3',
|
||||
cache: 'elasticache',
|
||||
compute: 'lambda'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Google Cloud Detection
|
||||
if (process.env.GOOGLE_CLOUD_PROJECT || await canReachMetadata('metadata.google.internal')) {
|
||||
return {
|
||||
provider: 'gcp',
|
||||
instance: await getGCEInstanceType(),
|
||||
region: process.env.GOOGLE_CLOUD_REGION,
|
||||
services: {
|
||||
storage: 'gcs',
|
||||
cache: 'memorystore',
|
||||
compute: 'cloud-run'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Vercel Edge Detection
|
||||
if (process.env.VERCEL) {
|
||||
return {
|
||||
provider: 'vercel',
|
||||
region: process.env.VERCEL_REGION,
|
||||
services: {
|
||||
storage: 'vercel-kv',
|
||||
cache: 'edge-config',
|
||||
compute: 'edge-runtime'
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Development vs Production
|
||||
|
||||
### Automatic Environment Detection
|
||||
|
||||
```typescript
|
||||
class EnvironmentDetector {
|
||||
detect() {
|
||||
const indicators = {
|
||||
// Development indicators
|
||||
isDevelopment:
|
||||
process.env.NODE_ENV === 'development' ||
|
||||
process.env.DEBUG ||
|
||||
process.argv.includes('--dev') ||
|
||||
isLocalhost() ||
|
||||
hasDevTools(),
|
||||
|
||||
// Test indicators
|
||||
isTest:
|
||||
process.env.NODE_ENV === 'test' ||
|
||||
process.env.CI ||
|
||||
isTestRunner(),
|
||||
|
||||
// Production indicators
|
||||
isProduction:
|
||||
process.env.NODE_ENV === 'production' ||
|
||||
process.env.VERCEL ||
|
||||
process.env.NETLIFY ||
|
||||
!isLocalhost()
|
||||
}
|
||||
|
||||
return indicators
|
||||
}
|
||||
}
|
||||
|
||||
// Different defaults for different environments
|
||||
const config = environment.isProduction ? {
|
||||
storage: 'filesystem',
|
||||
wal: true,
|
||||
monitoring: true,
|
||||
compression: true,
|
||||
caching: 'aggressive'
|
||||
} : {
|
||||
storage: 'memory',
|
||||
wal: false,
|
||||
monitoring: false,
|
||||
compression: false,
|
||||
caching: 'minimal'
|
||||
}
|
||||
```
|
||||
|
||||
## Error Recovery
|
||||
|
||||
### Automatic Fallbacks
|
||||
|
||||
Brainy automatically recovers from errors:
|
||||
|
||||
```typescript
|
||||
class AutoRecovery {
|
||||
async handleStorageFailure() {
|
||||
try {
|
||||
await this.primaryStorage.write(data)
|
||||
} catch (error) {
|
||||
console.warn('Primary storage failed, trying fallback')
|
||||
|
||||
// Try secondary storage
|
||||
if (this.secondaryStorage) {
|
||||
await this.secondaryStorage.write(data)
|
||||
} else {
|
||||
// Fall back to memory
|
||||
await this.memoryStorage.write(data)
|
||||
|
||||
// Schedule retry
|
||||
this.scheduleRetry(data)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async handleModelFailure() {
|
||||
try {
|
||||
return await this.primaryModel.embed(text)
|
||||
} catch (error) {
|
||||
// Fall back to simpler model
|
||||
return await this.fallbackModel.embed(text)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Brainy detects production-style environments (for example `NODE_ENV` set to a
|
||||
non-development value) and reduces its own log verbosity automatically. This is
|
||||
logging-only behavior — it does not change indexing, storage, or query results.
|
||||
|
||||
## Configuration Override
|
||||
|
||||
While zero-config is default, you can override when needed:
|
||||
Zero-config is the default, not a ceiling. Every adaptive decision above has an
|
||||
explicit constructor option:
|
||||
|
||||
```typescript
|
||||
// Explicit configuration when needed
|
||||
const brain = new Brainy({
|
||||
// Override auto-detection
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: '/custom/path'
|
||||
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
|
||||
vector: {
|
||||
recall: 'accurate',
|
||||
persistMode: 'immediate'
|
||||
},
|
||||
|
||||
// Override auto-optimization
|
||||
optimization: {
|
||||
autoIndex: false,
|
||||
autoCache: false,
|
||||
autoBatch: false
|
||||
},
|
||||
|
||||
// Override auto-scaling
|
||||
scaling: {
|
||||
maxMemory: 2 * GB,
|
||||
maxConnections: 100,
|
||||
maxBatchSize: 1000
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Monitoring Auto-Adaptation
|
||||
|
||||
Brainy provides visibility into its auto-adaptation:
|
||||
|
||||
```typescript
|
||||
brain.on('adaptation', (event) => {
|
||||
console.log(`Brainy adapted: ${event.type}`)
|
||||
console.log(`Reason: ${event.reason}`)
|
||||
console.log(`Before: ${JSON.stringify(event.before)}`)
|
||||
console.log(`After: ${JSON.stringify(event.after)}`)
|
||||
cache: { maxSize: 50000, ttl: 600_000 }
|
||||
})
|
||||
|
||||
// Example events:
|
||||
// - Index created for frequently queried field
|
||||
// - Cache strategy changed due to low hit rate
|
||||
// - Batch size increased due to high throughput
|
||||
// - Storage migrated due to space constraints
|
||||
// - Model switched due to multilingual content
|
||||
await brain.init()
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
Brainy's zero-configuration and auto-adaptation capabilities mean you can focus on your application logic while Brainy handles:
|
||||
|
||||
- Environment detection and optimization
|
||||
- Storage selection and migration
|
||||
- Performance tuning and scaling
|
||||
- Resource management
|
||||
- Error recovery
|
||||
- Workload optimization
|
||||
|
||||
Just create a Brainy instance and start using it. Brainy will learn, adapt, and optimize itself for your specific use case—no configuration required.
|
||||
See the [API Reference](../api/README.md#configuration) for the complete option
|
||||
list.
|
||||
|
||||
## See Also
|
||||
|
||||
- [Architecture Overview](./overview.md)
|
||||
- [Storage Architecture](./storage.md)
|
||||
- [Performance Guide](../guides/performance.md)
|
||||
- [Augmentations System](./augmentations.md)
|
||||
- [Storage Adapters](../concepts/storage-adapters.md)
|
||||
- [Scaling Guide](../SCALING.md)
|
||||
- [API Reference](../api/README.md)
|
||||
|
|
|
|||
|
|
@ -135,11 +135,12 @@ The CLI `brainy inspect` subcommands all do this for you by default
|
|||
|
||||
## What's not enforced (yet)
|
||||
|
||||
- **Cloud storage backends** (S3, GCS, R2, Azure) do not currently enforce
|
||||
multi-process locking. Two processes pointing at the same bucket can both
|
||||
succeed at `init()` in writer mode and clobber each other's writes. A
|
||||
best-effort warning is logged in writer mode against a non-filesystem
|
||||
backend. Lock semantics for cloud backends will land in a future release.
|
||||
- **Non-filesystem backends** are out of scope in 8.0, which ships only the
|
||||
filesystem and memory adapters. A custom `BaseStorage` subclass that is not
|
||||
filesystem-backed does not enforce multi-process locking by default: two
|
||||
processes can both succeed at `init()` in writer mode and clobber each
|
||||
other's writes. A best-effort warning is logged in writer mode against a
|
||||
non-filesystem backend.
|
||||
- **Long-running readers** do not automatically pick up new Cortex segments
|
||||
the writer publishes. One-shot inspector calls re-open the store and see
|
||||
fresh segments; a reader that stays open for hours sees its column store
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ That's the full ceremony for inheriting multi-process safety.
|
|||
|
||||
## When NOT to extend FileSystemStorage
|
||||
|
||||
If your storage is **not filesystem-backed** (S3, GCS, R2, Azure, a custom
|
||||
If your storage is **not filesystem-backed** (a custom
|
||||
network backend), extend `BaseStorage` directly:
|
||||
|
||||
```typescript
|
||||
|
|
|
|||
|
|
@ -1,414 +0,0 @@
|
|||
# 🚀 Brainy 2.0 - Complete Feature List
|
||||
|
||||
> **The Truth**: Brainy is MORE powerful than previously documented! This is the complete list of ALL implemented features.
|
||||
|
||||
## 🧠 Core Intelligence Engine
|
||||
|
||||
### Triple Intelligence System ✅
|
||||
Unified query system that automatically combines:
|
||||
- **Vector Search**: HNSW-indexed semantic similarity (O(log n) performance)
|
||||
- **Graph Traversal**: Relationship-based discovery
|
||||
- **Field Filtering**: Metadata and attribute queries
|
||||
- **Auto-optimization**: Queries are automatically optimized based on data patterns
|
||||
|
||||
```typescript
|
||||
// All three intelligences work together automatically
|
||||
const results = await brain.find({
|
||||
like: 'AI research', // Vector search
|
||||
where: { year: 2024 }, // Metadata filtering
|
||||
connected: { to: authorId } // Graph traversal
|
||||
})
|
||||
```
|
||||
|
||||
### Neural Query Understanding ✅
|
||||
- **220+ embedded patterns** for query intent detection
|
||||
- Natural language query processing
|
||||
- Automatic query type detection
|
||||
- Query rewriting and optimization
|
||||
|
||||
## 🔧 12+ Production Augmentations
|
||||
|
||||
```typescript
|
||||
// Full crash recovery, checkpointing, replay
|
||||
```
|
||||
|
||||
### 2. Entity Registry ✅
|
||||
```typescript
|
||||
import { EntityRegistryAugmentation } from 'brainy'
|
||||
// Bloom filter-based deduplication for streaming data
|
||||
// Handles millions of entities with minimal memory
|
||||
```
|
||||
|
||||
### 3. Auto-Register Entities ✅
|
||||
```typescript
|
||||
import { AutoRegisterEntitiesAugmentation } from 'brainy'
|
||||
// Automatically extracts and registers entities from text
|
||||
```
|
||||
|
||||
### 4. Intelligent Verb Scoring ✅
|
||||
```typescript
|
||||
import { IntelligentVerbScoringAugmentation } from 'brainy'
|
||||
// Multi-factor relationship strength:
|
||||
// - Semantic similarity
|
||||
// - Temporal decay
|
||||
// - Frequency amplification
|
||||
// - Context awareness
|
||||
```
|
||||
|
||||
### 5. Batch Processing ✅
|
||||
```typescript
|
||||
import { BatchProcessingAugmentation } from 'brainy'
|
||||
// Adaptive batching with backpressure
|
||||
// Dynamically adjusts batch size based on load
|
||||
```
|
||||
|
||||
### 6. Connection Pool ✅
|
||||
```typescript
|
||||
import { ConnectionPoolAugmentation } from 'brainy'
|
||||
// Auto-scaling connection management
|
||||
// Optimized for high-concurrency workloads
|
||||
```
|
||||
|
||||
### 7. Request Deduplicator ✅
|
||||
```typescript
|
||||
import { RequestDeduplicatorAugmentation } from 'brainy'
|
||||
// In-flight request deduplication
|
||||
// 3x performance boost for concurrent operations
|
||||
```
|
||||
|
||||
### 8. WebSocket Conduit ✅
|
||||
```typescript
|
||||
import { WebSocketConduitAugmentation } from 'brainy'
|
||||
// Real-time bidirectional streaming
|
||||
// Auto-reconnection and heartbeat
|
||||
```
|
||||
|
||||
### 9. WebRTC Conduit ✅
|
||||
```typescript
|
||||
import { WebRTCConduitAugmentation } from 'brainy'
|
||||
// Peer-to-peer data channels
|
||||
// Direct browser-to-browser communication
|
||||
```
|
||||
|
||||
### 10. Memory Storage Optimization ✅
|
||||
```typescript
|
||||
import { MemoryStorageAugmentation } from 'brainy'
|
||||
// Memory-specific optimizations
|
||||
// Circular buffers, compression
|
||||
```
|
||||
|
||||
### 11. Server Search Conduit ✅
|
||||
```typescript
|
||||
import { ServerSearchConduitAugmentation } from 'brainy'
|
||||
// Forwards queries to a remote Brainy server
|
||||
```
|
||||
|
||||
### 12. Neural Import ✅
|
||||
```typescript
|
||||
import { NeuralImportAugmentation } from 'brainy'
|
||||
// AI-powered data understanding
|
||||
// Automatic entity detection and classification
|
||||
// Relationship discovery
|
||||
```
|
||||
|
||||
## 🤖 Neural Import Capabilities (FULLY IMPLEMENTED!)
|
||||
|
||||
```typescript
|
||||
const neuralImport = new NeuralImport(brain)
|
||||
|
||||
// ALL of these work TODAY:
|
||||
await neuralImport.neuralImport('data.csv')
|
||||
await neuralImport.detectEntitiesWithNeuralAnalysis(data)
|
||||
await neuralImport.detectNounType(entity)
|
||||
await neuralImport.detectRelationships(entities)
|
||||
await neuralImport.generateInsights(data)
|
||||
```
|
||||
|
||||
### Features:
|
||||
- **Auto-detects file format** (CSV, JSON, XML, etc.)
|
||||
- **Identifies entity types** using AI
|
||||
- **Discovers relationships** between entities
|
||||
- **Generates insights** about the data
|
||||
- **Creates optimal graph structure** automatically
|
||||
|
||||
## 🎯 Zero-Config Model Loading Cascade
|
||||
|
||||
Brainy automatically loads models with ZERO configuration required:
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy() // That's it!
|
||||
await brain.init()
|
||||
// Models load automatically from best available source
|
||||
```
|
||||
|
||||
### Loading Priority:
|
||||
1. **Local Cache** (./models) - Instant, no network
|
||||
2. **CDN** (models.soulcraft.com) - Fast, global [Coming Soon]
|
||||
3. **GitHub Releases** - Reliable backup
|
||||
4. **HuggingFace** - Ultimate fallback
|
||||
|
||||
### Key Features:
|
||||
- **Automatic fallback** if sources fail
|
||||
- **Model verification** with checksums
|
||||
- **Offline support** with bundled models
|
||||
- **No environment variables needed**
|
||||
- **Works in all environments** (Node, Browser, Workers)
|
||||
|
||||
## 🏢 Operation Modes
|
||||
|
||||
Run many reader processes against one shared on-disk store with a single writer.
|
||||
|
||||
### Reader Mode ✅
|
||||
```typescript
|
||||
const brain = new Brainy({ mode: 'reader' })
|
||||
// Optimized for read-heavy workloads
|
||||
// 80% cache ratio, aggressive prefetch
|
||||
// 1 hour TTL, minimal writes
|
||||
```
|
||||
|
||||
### Writer Mode ✅
|
||||
```typescript
|
||||
const brain = new Brainy({ mode: 'writer' })
|
||||
// Optimized for write-heavy workloads
|
||||
// Large write buffers, batch writes
|
||||
// Minimal caching, fast ingestion
|
||||
```
|
||||
|
||||
### Hybrid Mode ✅
|
||||
```typescript
|
||||
const brain = new Brainy({ mode: 'hybrid' })
|
||||
// Balanced for mixed workloads
|
||||
// Adaptive caching and batching
|
||||
```
|
||||
|
||||
## 💾 Advanced Caching System
|
||||
|
||||
### 3-Level Cache Architecture ✅
|
||||
```typescript
|
||||
const cacheConfig = {
|
||||
hotCache: {
|
||||
size: 1000, // L1 - RAM
|
||||
ttl: 60000 // 1 minute
|
||||
},
|
||||
warmCache: {
|
||||
size: 10000, // L2 - Fast storage
|
||||
ttl: 300000 // 5 minutes
|
||||
},
|
||||
coldCache: {
|
||||
size: 100000, // L3 - Persistent
|
||||
ttl: null // No expiry
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cache Features:
|
||||
- **Automatic promotion/demotion** between levels
|
||||
- **LRU eviction** within each level
|
||||
- **Compression** for cold cache
|
||||
- **Statistics tracking** for optimization
|
||||
|
||||
## 📊 Comprehensive Statistics
|
||||
|
||||
```typescript
|
||||
const stats = await brain.getStats()
|
||||
// Returns detailed metrics:
|
||||
{
|
||||
nouns: {
|
||||
count, created, updated, deleted,
|
||||
size, avgSize
|
||||
},
|
||||
verbs: {
|
||||
count, created, types,
|
||||
weights: { min, max, avg }
|
||||
},
|
||||
vectors: {
|
||||
dimensions: 384,
|
||||
indexSize, partitions,
|
||||
avgSearchTime
|
||||
},
|
||||
cache: {
|
||||
hits, misses, evictions,
|
||||
hitRate, sizes
|
||||
},
|
||||
performance: {
|
||||
operations, avgTimes,
|
||||
p95Latency, p99Latency
|
||||
},
|
||||
storage: {
|
||||
used, available,
|
||||
compression, files
|
||||
},
|
||||
throttling: {
|
||||
delays, rateLimited,
|
||||
backoffMs, retries
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 GPU Acceleration Support
|
||||
|
||||
```typescript
|
||||
// Automatic GPU detection
|
||||
const device = await detectBestDevice()
|
||||
// Returns: 'cpu' | 'webgpu' | 'cuda'
|
||||
|
||||
// WebGPU in browser (when available)
|
||||
if (device === 'webgpu') {
|
||||
// Transformer models use WebGPU automatically
|
||||
}
|
||||
|
||||
// CUDA in Node.js (future GPU support)
|
||||
if (device === 'cuda') {
|
||||
// Future: GPU acceleration for embeddings
|
||||
}
|
||||
```
|
||||
|
||||
## 🔄 Adaptive Systems
|
||||
|
||||
### Adaptive Backpressure ✅
|
||||
```typescript
|
||||
// Automatically adjusts flow based on system load
|
||||
// Prevents OOM and maintains throughput
|
||||
```
|
||||
|
||||
### Adaptive Socket Manager ✅
|
||||
```typescript
|
||||
// Dynamic connection pooling
|
||||
// Scales connections based on traffic patterns
|
||||
```
|
||||
|
||||
### Cache Auto-Configuration ✅
|
||||
```typescript
|
||||
// Sizes cache based on available memory
|
||||
// Adjusts strategies based on usage patterns
|
||||
```
|
||||
|
||||
### S3 Throttling Protection ✅
|
||||
```typescript
|
||||
// Built-in exponential backoff
|
||||
// Rate limit detection and adaptation
|
||||
// Automatic retry with jitter
|
||||
```
|
||||
|
||||
## 🛠️ Storage Adapters
|
||||
|
||||
All included, auto-selected based on environment:
|
||||
|
||||
### FileSystem Storage ✅
|
||||
- Default for Node.js
|
||||
- Efficient file-based storage
|
||||
- Automatic directory management
|
||||
|
||||
### Memory Storage ✅
|
||||
- Ultra-fast in-memory operations
|
||||
- Perfect for testing and temporary data
|
||||
- Circular buffer support
|
||||
|
||||
### OPFS Storage ✅
|
||||
- Browser persistent storage
|
||||
- Survives page refreshes
|
||||
- Quota management
|
||||
|
||||
### S3 Storage ✅
|
||||
- AWS S3 compatible
|
||||
- Automatic multipart uploads
|
||||
- Throttling protection
|
||||
- Batch operations
|
||||
|
||||
## 🎨 Natural Language Processing
|
||||
|
||||
### Built-in Patterns (220+)
|
||||
- Question types (what, why, how, when, where)
|
||||
- Temporal queries (yesterday, last week, 2024)
|
||||
- Comparative queries (better than, similar to)
|
||||
- Aggregations (count, sum, average)
|
||||
- Filters (only, except, without)
|
||||
- Relationships (related to, connected with)
|
||||
|
||||
### Coverage: 94-98% of typical queries!
|
||||
|
||||
## 🔐 Security Features
|
||||
|
||||
### Built-in Security ✅
|
||||
- Automatic input sanitization
|
||||
- SQL injection prevention
|
||||
- XSS protection for web contexts
|
||||
- Rate limiting support
|
||||
|
||||
### Encryption Ready ✅
|
||||
```typescript
|
||||
import { crypto } from 'brainy/utils'
|
||||
// AES-256-GCM encryption utilities
|
||||
// Key derivation functions
|
||||
// Secure random generation
|
||||
```
|
||||
|
||||
## 🎯 Key Design Principles
|
||||
|
||||
### 1. Zero Configuration
|
||||
```typescript
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
// Everything else is automatic!
|
||||
```
|
||||
|
||||
### 2. Fixed Dimensions (384)
|
||||
- **ALWAYS** uses all-MiniLM-L6-v2 model
|
||||
- **ALWAYS** 384 dimensions
|
||||
- **NOT** configurable (by design)
|
||||
- Ensures everything works together
|
||||
|
||||
### 3. Progressive Enhancement
|
||||
- Starts simple, scales automatically
|
||||
- Adapts to workload patterns
|
||||
- Optimizes based on usage
|
||||
|
||||
### 4. Universal Compatibility
|
||||
- Works in Node.js 18+
|
||||
- Works in modern browsers
|
||||
- Works in Web Workers
|
||||
- Works in Edge environments
|
||||
|
||||
## 📦 What Ships in Core (MIT Licensed)
|
||||
|
||||
**EVERYTHING** is included in the core package:
|
||||
- ✅ All engines (vector, graph, field, neural)
|
||||
- ✅ All augmentations (12+)
|
||||
- ✅ All storage adapters
|
||||
- ✅ Reader / writer / hybrid operation modes
|
||||
- ✅ Complete statistics
|
||||
- ✅ GPU support
|
||||
- ✅ No feature limitations
|
||||
- ✅ No premium tiers
|
||||
- ✅ 100% MIT licensed
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
```typescript
|
||||
import { Brainy } from 'brainy'
|
||||
|
||||
// Zero config required!
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Add data (auto-detects type)
|
||||
await brain.add('Content here')
|
||||
|
||||
// Search with natural language
|
||||
const results = await brain.find('related content from last week')
|
||||
|
||||
// Everything else is automatic!
|
||||
```
|
||||
|
||||
## 📈 Performance Characteristics
|
||||
|
||||
- **Vector Search**: O(log n) with HNSW indexing
|
||||
- **Graph Traversal**: O(k) for k-hop queries
|
||||
- **Field Filtering**: O(1) with metadata index
|
||||
- **Memory Usage**: ~100MB base + data
|
||||
- **Embedding Speed**: ~100ms for batch of 10
|
||||
- **Query Speed**: <10ms for most queries
|
||||
|
||||
## 🎉 Summary
|
||||
|
||||
Brainy 2.0 is a **complete**, **production-ready** AI database that requires **ZERO configuration**. Every feature listed here is **implemented and working** today. No configuration, no setup, no complexity - just powerful AI capabilities that work out of the box!
|
||||
|
|
@ -1,242 +0,0 @@
|
|||
# 🚀 Brainy - Production-Ready Features
|
||||
|
||||
> **Status**: All features listed here are IMPLEMENTED and TESTED
|
||||
|
||||
## 📊 Performance Metrics
|
||||
- **Search Latency**: <10ms for 10,000+ items
|
||||
- **Write Throughput**: 10,000+ ops/sec
|
||||
- **Memory Efficiency**: <500MB for 10K items
|
||||
- **Concurrent Operations**: 100+ simultaneous operations
|
||||
|
||||
## 🧠 Core Intelligence Features
|
||||
|
||||
### Triple Intelligence System ✅
|
||||
Unified query system combining three types of intelligence:
|
||||
```typescript
|
||||
const results = await brain.find({
|
||||
like: 'AI research', // Vector similarity search
|
||||
where: { year: 2024 }, // Metadata filtering
|
||||
connected: { to: authorId } // Graph relationships
|
||||
})
|
||||
```
|
||||
|
||||
### Intelligent Type Mapping ✅
|
||||
Prevents semantic degradation by intelligently inferring types:
|
||||
```typescript
|
||||
// Automatically infers 'person' from email field
|
||||
brain.add({ name: "John", email: "john@example.com" }, 'entity')
|
||||
// → Stored as type: 'person', not generic 'entity'
|
||||
```
|
||||
|
||||
### Neural Query Understanding ✅
|
||||
- 220+ embedded patterns for intent detection
|
||||
- Natural language query processing
|
||||
- Automatic query optimization
|
||||
- Pattern-based query rewriting
|
||||
|
||||
## 🔐 Security & Compliance
|
||||
|
||||
### Rate Limiting ✅
|
||||
Per-operation configurable limits:
|
||||
```typescript
|
||||
const rateLimiter = createRateLimitAugmentation({
|
||||
limits: {
|
||||
searches: 1000, // per minute
|
||||
writes: 100,
|
||||
reads: 5000,
|
||||
deletes: 50
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Audit Logging ✅
|
||||
Comprehensive operation tracking:
|
||||
```typescript
|
||||
const auditLogger = createAuditLogAugmentation({
|
||||
logLevel: 'detailed',
|
||||
retention: 90, // days
|
||||
includeMetadata: true
|
||||
})
|
||||
|
||||
// Query audit logs
|
||||
const logs = auditLogger.queryLogs({
|
||||
operation: 'add',
|
||||
startTime: Date.now() - 3600000
|
||||
})
|
||||
```
|
||||
|
||||
## 📦 Storage & Persistence
|
||||
|
||||
Full crash recovery and replay:
|
||||
```typescript
|
||||
enabled: true,
|
||||
checkpointInterval: 1000,
|
||||
maxLogSize: 100 * 1024 * 1024 // 100MB
|
||||
}))
|
||||
```
|
||||
|
||||
### Multi-Tenancy ✅
|
||||
Service-based data isolation:
|
||||
```typescript
|
||||
// Isolated data per service
|
||||
await brain.add(data, 'document', { service: 'tenant-1' })
|
||||
await brain.find('query', { service: 'tenant-1' })
|
||||
```
|
||||
|
||||
### Write-Only Mode ✅
|
||||
For dedicated write nodes:
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
mode: 'write-only',
|
||||
storage: 's3://bucket/path'
|
||||
})
|
||||
```
|
||||
|
||||
## 🚀 Performance Features
|
||||
|
||||
### Batch Operations ✅
|
||||
Optimized bulk processing:
|
||||
```typescript
|
||||
// Parallel processing with automatic batching
|
||||
await brain.addMany(items) // <10ms per item
|
||||
await brain.updateMany(updates)
|
||||
await brain.removeMany(filters)
|
||||
```
|
||||
|
||||
### Request Deduplication ✅
|
||||
Automatic duplicate request handling:
|
||||
```typescript
|
||||
brain.use(new RequestDeduplicatorAugmentation())
|
||||
// Identical concurrent requests return same result
|
||||
```
|
||||
|
||||
### Smart Caching ✅
|
||||
Intelligent search result caching:
|
||||
```typescript
|
||||
brain.use(new CacheAugmentation({
|
||||
maxSize: 10000,
|
||||
ttl: 300000, // 5 minutes
|
||||
invalidateOnWrite: true
|
||||
}))
|
||||
```
|
||||
|
||||
## 🔄 Data Processing
|
||||
|
||||
### Entity Registry ✅
|
||||
Bloom filter-based deduplication:
|
||||
```typescript
|
||||
brain.use(new EntityRegistryAugmentation())
|
||||
// Handles millions of entities with minimal memory
|
||||
```
|
||||
|
||||
### Neural Import ✅
|
||||
Intelligent data import with type inference:
|
||||
```typescript
|
||||
await brain.import({
|
||||
source: 'data.json',
|
||||
autoDetectTypes: true,
|
||||
batchSize: 1000
|
||||
})
|
||||
```
|
||||
|
||||
### Streaming Pipeline ✅
|
||||
Real-time data processing:
|
||||
```typescript
|
||||
brain.stream()
|
||||
.pipe(transform)
|
||||
.pipe(enrich)
|
||||
.pipe(brain.writer())
|
||||
```
|
||||
|
||||
## 📊 Analytics & Monitoring
|
||||
|
||||
### Metrics Collection ✅
|
||||
Built-in performance metrics:
|
||||
```typescript
|
||||
const metrics = brain.getMetrics()
|
||||
// {
|
||||
// operations: { add: 1000, find: 5000 },
|
||||
// performance: { p95: 8, p99: 12 },
|
||||
// cache: { hits: 4500, misses: 500 }
|
||||
// }
|
||||
```
|
||||
|
||||
### Health Monitoring ✅
|
||||
Automatic health checks:
|
||||
```typescript
|
||||
const health = brain.getHealth()
|
||||
// {
|
||||
// status: 'healthy',
|
||||
// storage: 'connected',
|
||||
// memory: { used: 245, limit: 512 }
|
||||
// }
|
||||
```
|
||||
|
||||
## 🛠️ Developer Experience
|
||||
|
||||
### Zero Configuration ✅
|
||||
Works out of the box:
|
||||
```typescript
|
||||
import Brainy from '@soulcraft/brainy'
|
||||
const brain = new Brainy() // Auto-configures everything
|
||||
```
|
||||
|
||||
### TypeScript First ✅
|
||||
Full type safety and inference:
|
||||
```typescript
|
||||
// Types are automatically inferred
|
||||
const results = await brain.find<MyType>('query')
|
||||
```
|
||||
|
||||
### Augmentation System ✅
|
||||
Extensible plugin architecture:
|
||||
```typescript
|
||||
class CustomAugmentation extends BaseAugmentation {
|
||||
execute(operation, params, next) {
|
||||
// Your custom logic
|
||||
return next()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 Operational Features
|
||||
|
||||
### Graceful Shutdown ✅
|
||||
Clean shutdown with data persistence:
|
||||
```typescript
|
||||
process.on('SIGTERM', async () => {
|
||||
await brain.shutdown() // Saves all pending data
|
||||
})
|
||||
```
|
||||
|
||||
### Hot Reload ✅
|
||||
Configuration updates without restart:
|
||||
```typescript
|
||||
brain.updateConfig({
|
||||
cache: { enabled: false }
|
||||
})
|
||||
```
|
||||
|
||||
### Backup & Restore ✅
|
||||
Full data backup capabilities:
|
||||
```typescript
|
||||
await brain.backup('backup.bin')
|
||||
await brain.restore('backup.bin')
|
||||
```
|
||||
|
||||
## 📈 Proven at Scale
|
||||
|
||||
- **10,000+ items**: Sub-10ms search
|
||||
- **1M+ operations**: Stable memory usage
|
||||
- **100+ concurrent users**: No performance degradation
|
||||
|
||||
## 🚫 NOT Implemented (Planned)
|
||||
|
||||
These features are documented but NOT yet implemented:
|
||||
- GraphQL API (use REST API instead)
|
||||
- Kubernetes operators (use Docker)
|
||||
|
||||
---
|
||||
|
||||
*Last Updated: Latest Version*
|
||||
*All features listed above are production-ready and tested*
|
||||
|
|
@ -1,15 +1,17 @@
|
|||
# Framework Integration Guide
|
||||
|
||||
Brainy 3.0 is **framework-first** - designed from the ground up to work seamlessly with modern JavaScript frameworks. This guide shows you how to integrate Brainy into any framework.
|
||||
Brainy is **framework-friendly** - designed to drop into the server side of any modern JavaScript framework. This guide shows you how to integrate Brainy into framework-based apps.
|
||||
|
||||
## 🎯 Why Framework-First?
|
||||
> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun (server-side only). It is not a browser library. In framework apps, run Brainy in API routes, server components, server actions, loaders, or a dedicated backend service - never in client-side bundles.
|
||||
|
||||
Traditional AI databases require complex browser polyfills and bundler configurations. Brainy 3.0 trusts your framework to handle this:
|
||||
## 🎯 Why Server-Side?
|
||||
|
||||
Brainy embeds an HNSW vector index, a graph engine, and a filesystem-backed persistence layer. These belong on the server:
|
||||
|
||||
- **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'`
|
||||
- **Framework responsibility**: Let Next.js, Vite, Webpack handle Node.js polyfills
|
||||
- **Cleaner code**: No browser-specific entry points or conditional imports
|
||||
- **Better DX**: Same API everywhere - browser, server, edge
|
||||
- **Auto storage detection**: `new Brainy()` auto-selects filesystem persistence on Node
|
||||
- **Cleaner code**: No browser polyfills, no conditional client/server imports
|
||||
- **Better DX**: One instance shared across your server routes
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
|
|
@ -24,7 +26,8 @@ npm install @soulcraft/brainy
|
|||
```javascript
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
// Works in any framework!
|
||||
// Run on the server (API route, server component, backend service)
|
||||
// new Brainy() auto-detects filesystem persistence on Node
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
|
|
@ -41,52 +44,48 @@ const results = await brain.find("framework integration")
|
|||
|
||||
## ⚛️ React Integration
|
||||
|
||||
Brainy runs on the server, so a React client component talks to it through an API endpoint (see the Next.js API route below). The hook fetches results; it never instantiates Brainy in the browser.
|
||||
|
||||
### Basic Hook Pattern
|
||||
|
||||
```jsx
|
||||
import { useState, useEffect, useCallback } from 'react'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { useState, useCallback } from 'react'
|
||||
|
||||
function useBrainy() {
|
||||
const [brain, setBrain] = useState(null)
|
||||
const [isReady, setIsReady] = useState(false)
|
||||
function useBrainySearch(endpoint = '/api/search') {
|
||||
const [results, setResults] = useState([])
|
||||
const [loading, setLoading] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
const initBrain = async () => {
|
||||
const newBrain = new Brainy({
|
||||
storage: { type: 'opfs' } // Browser storage
|
||||
const search = useCallback(async (query) => {
|
||||
if (!query) return
|
||||
setLoading(true)
|
||||
try {
|
||||
const res = await fetch(endpoint, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query })
|
||||
})
|
||||
await newBrain.init()
|
||||
setBrain(newBrain)
|
||||
setIsReady(true)
|
||||
const { results } = await res.json()
|
||||
setResults(results)
|
||||
} finally {
|
||||
setLoading(false)
|
||||
}
|
||||
}, [endpoint])
|
||||
|
||||
initBrain()
|
||||
}, [])
|
||||
|
||||
return { brain, isReady }
|
||||
return { results, loading, search }
|
||||
}
|
||||
|
||||
// Usage in component
|
||||
function SearchComponent() {
|
||||
const { brain, isReady } = useBrainy()
|
||||
const [results, setResults] = useState([])
|
||||
|
||||
const handleSearch = useCallback(async (query) => {
|
||||
if (!isReady) return
|
||||
const searchResults = await brain.find(query)
|
||||
setResults(searchResults)
|
||||
}, [brain, isReady])
|
||||
|
||||
if (!isReady) return <div>Loading AI...</div>
|
||||
const { results, loading, search } = useBrainySearch()
|
||||
|
||||
return (
|
||||
<div>
|
||||
<input
|
||||
type="text"
|
||||
placeholder="Search..."
|
||||
onChange={(e) => handleSearch(e.target.value)}
|
||||
onChange={(e) => search(e.target.value)}
|
||||
/>
|
||||
{loading && <div>Searching...</div>}
|
||||
<div>
|
||||
{results.map(result => (
|
||||
<div key={result.id}>
|
||||
|
|
@ -100,48 +99,34 @@ function SearchComponent() {
|
|||
}
|
||||
```
|
||||
|
||||
### React Context Pattern
|
||||
### Shared Server Instance
|
||||
|
||||
```jsx
|
||||
import React, { createContext, useContext, useEffect, useState } from 'react'
|
||||
On the server, create one Brainy instance and reuse it across requests. This module is imported only by server code (API routes, server components), never by client components:
|
||||
|
||||
```javascript
|
||||
// lib/brain.server.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const BrainyContext = createContext()
|
||||
let brainPromise
|
||||
|
||||
export function BrainyProvider({ children }) {
|
||||
const [brain, setBrain] = useState(null)
|
||||
const [isReady, setIsReady] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
const initBrain = async () => {
|
||||
const newBrain = new Brainy()
|
||||
await newBrain.init()
|
||||
setBrain(newBrain)
|
||||
setIsReady(true)
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
// new Brainy() auto-detects filesystem persistence on Node
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
}
|
||||
|
||||
initBrain()
|
||||
}, [])
|
||||
|
||||
return (
|
||||
<BrainyContext.Provider value={{ brain, isReady }}>
|
||||
{children}
|
||||
</BrainyContext.Provider>
|
||||
)
|
||||
}
|
||||
|
||||
export function useBrainContext() {
|
||||
const context = useContext(BrainyContext)
|
||||
if (!context) {
|
||||
throw new Error('useBrainContext must be used within BrainyProvider')
|
||||
}
|
||||
return context
|
||||
return brainPromise
|
||||
}
|
||||
```
|
||||
|
||||
## 🟢 Vue.js Integration
|
||||
|
||||
### Composition API
|
||||
Vue components call a server endpoint (see the Nuxt server route in the [Vue.js Integration Guide](vue-integration.md)); Brainy itself runs on the server.
|
||||
|
||||
### Composition API (client component)
|
||||
|
||||
```vue
|
||||
<template>
|
||||
|
|
@ -155,100 +140,70 @@ export function useBrainContext() {
|
|||
</template>
|
||||
|
||||
<script setup>
|
||||
import { ref, onMounted } from 'vue'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { ref } from 'vue'
|
||||
|
||||
const brain = ref(null)
|
||||
const isReady = ref(false)
|
||||
const query = ref('')
|
||||
const results = ref([])
|
||||
|
||||
onMounted(async () => {
|
||||
brain.value = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
await brain.value.init()
|
||||
isReady.value = true
|
||||
})
|
||||
|
||||
const search = async () => {
|
||||
if (!isReady.value || !query.value) return
|
||||
results.value = await brain.value.find(query.value)
|
||||
if (!query.value) return
|
||||
const res = await fetch('/api/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query: query.value })
|
||||
})
|
||||
results.value = (await res.json()).results
|
||||
}
|
||||
</script>
|
||||
```
|
||||
|
||||
### Vue 3 Plugin
|
||||
### Shared Server Instance
|
||||
|
||||
On the server, create one Brainy instance and reuse it across requests:
|
||||
|
||||
```javascript
|
||||
// plugins/brainy.js
|
||||
// server/brain.js (server-only module)
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export default {
|
||||
install(app, options) {
|
||||
const brain = new Brainy(options)
|
||||
let brainPromise
|
||||
|
||||
app.config.globalProperties.$brain = brain
|
||||
app.provide('brain', brain)
|
||||
|
||||
// Initialize on app mount
|
||||
brain.init()
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
// new Brainy() auto-detects filesystem persistence on Node
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
}
|
||||
return brainPromise
|
||||
}
|
||||
|
||||
// main.js
|
||||
import { createApp } from 'vue'
|
||||
import BrainyPlugin from './plugins/brainy'
|
||||
|
||||
const app = createApp(App)
|
||||
app.use(BrainyPlugin, {
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
```
|
||||
|
||||
## 🅰️ Angular Integration
|
||||
|
||||
### Service Pattern
|
||||
The Angular service calls your backend over HTTP; Brainy lives in that backend, not in the browser.
|
||||
|
||||
### Service Pattern (calls the backend)
|
||||
|
||||
```typescript
|
||||
// brainy.service.ts
|
||||
import { Injectable } from '@angular/core'
|
||||
import { BehaviorSubject, Observable } from 'rxjs'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { HttpClient } from '@angular/common/http'
|
||||
import { Observable } from 'rxjs'
|
||||
|
||||
@Injectable({
|
||||
providedIn: 'root'
|
||||
})
|
||||
export class BrainyService {
|
||||
private brain: Brainy
|
||||
private readySubject = new BehaviorSubject<boolean>(false)
|
||||
constructor(private http: HttpClient) {}
|
||||
|
||||
ready$: Observable<boolean> = this.readySubject.asObservable()
|
||||
|
||||
constructor() {
|
||||
this.initBrain()
|
||||
search(query: string): Observable<{ results: any[] }> {
|
||||
return this.http.post<{ results: any[] }>('/api/search', { query })
|
||||
}
|
||||
|
||||
private async initBrain() {
|
||||
this.brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
await this.brain.init()
|
||||
this.readySubject.next(true)
|
||||
}
|
||||
|
||||
async search(query: string): Promise<any[]> {
|
||||
if (!this.readySubject.value) {
|
||||
throw new Error('Brain not ready')
|
||||
}
|
||||
return await this.brain.find(query)
|
||||
}
|
||||
|
||||
async add(data: any, type: string, metadata?: any): Promise<string> {
|
||||
if (!this.readySubject.value) {
|
||||
throw new Error('Brain not ready')
|
||||
}
|
||||
return await this.brain.add({ data, type, metadata })
|
||||
add(data: any, type: string, metadata?: any): Observable<{ id: string }> {
|
||||
return this.http.post<{ id: string }>('/api/add', { data, type, metadata })
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
@ -280,64 +235,52 @@ export class SearchComponent {
|
|||
|
||||
constructor(private brainyService: BrainyService) {}
|
||||
|
||||
async search() {
|
||||
search() {
|
||||
if (!this.query) return
|
||||
this.results = await this.brainyService.search(this.query)
|
||||
this.brainyService.search(this.query).subscribe(({ results }) => {
|
||||
this.results = results
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The matching backend endpoint uses Brainy directly (Node/Bun):
|
||||
|
||||
```typescript
|
||||
// server: api/search
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy() // auto-detects filesystem persistence on Node
|
||||
await brain.init()
|
||||
|
||||
export async function handleSearch(query: string) {
|
||||
return await brain.find(query)
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 Next.js Integration
|
||||
|
||||
### App Router (Next.js 13+)
|
||||
In Next.js, Brainy lives in server code only: API routes, server components, or server actions. Create one shared instance in a server-only module.
|
||||
|
||||
```jsx
|
||||
// app/providers.jsx
|
||||
'use client'
|
||||
import { createContext, useContext, useEffect, useState } from 'react'
|
||||
### Shared Server Instance
|
||||
|
||||
```javascript
|
||||
// lib/brain.server.js (imported only by server code)
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const BrainyContext = createContext()
|
||||
let brainPromise
|
||||
|
||||
export function BrainyProvider({ children }) {
|
||||
const [brain, setBrain] = useState(null)
|
||||
const [isReady, setIsReady] = useState(false)
|
||||
|
||||
useEffect(() => {
|
||||
const initBrain = async () => {
|
||||
const newBrain = new Brainy()
|
||||
await newBrain.init()
|
||||
setBrain(newBrain)
|
||||
setIsReady(true)
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './data' }
|
||||
})
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
}
|
||||
|
||||
initBrain()
|
||||
}, [])
|
||||
|
||||
return (
|
||||
<BrainyContext.Provider value={{ brain, isReady }}>
|
||||
{children}
|
||||
</BrainyContext.Provider>
|
||||
)
|
||||
}
|
||||
|
||||
export const useBrainy = () => useContext(BrainyContext)
|
||||
```
|
||||
|
||||
```jsx
|
||||
// app/layout.jsx
|
||||
import { BrainyProvider } from './providers'
|
||||
|
||||
export default function RootLayout({ children }) {
|
||||
return (
|
||||
<html>
|
||||
<body>
|
||||
<BrainyProvider>
|
||||
{children}
|
||||
</BrainyProvider>
|
||||
</body>
|
||||
</html>
|
||||
)
|
||||
return brainPromise
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -345,45 +288,78 @@ export default function RootLayout({ children }) {
|
|||
|
||||
```javascript
|
||||
// app/api/search/route.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'filesystem', rootDirectory: './data' }
|
||||
})
|
||||
await brain.init()
|
||||
import { getBrain } from '@/lib/brain.server'
|
||||
|
||||
export async function POST(request) {
|
||||
const { query } = await request.json()
|
||||
const brain = await getBrain()
|
||||
const results = await brain.find(query)
|
||||
|
||||
return Response.json({ results })
|
||||
}
|
||||
```
|
||||
|
||||
## 🔷 Svelte Integration
|
||||
### Server Action
|
||||
|
||||
```javascript
|
||||
// app/actions.js
|
||||
'use server'
|
||||
import { getBrain } from '@/lib/brain.server'
|
||||
|
||||
export async function search(query) {
|
||||
const brain = await getBrain()
|
||||
return await brain.find(query)
|
||||
}
|
||||
```
|
||||
|
||||
## 🔷 SvelteKit Integration
|
||||
|
||||
Brainy runs in a server-only module (`*.server.js`); the component fetches results from an endpoint.
|
||||
|
||||
```javascript
|
||||
// src/lib/server/brain.js (server-only — note the .server suffix)
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
let brainPromise
|
||||
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
const brain = new Brainy() // auto-detects filesystem persistence
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
}
|
||||
return brainPromise
|
||||
}
|
||||
```
|
||||
|
||||
```javascript
|
||||
// src/routes/api/search/+server.js
|
||||
import { json } from '@sveltejs/kit'
|
||||
import { getBrain } from '$lib/server/brain'
|
||||
|
||||
export async function POST({ request }) {
|
||||
const { query } = await request.json()
|
||||
const brain = await getBrain()
|
||||
return json({ results: await brain.find(query) })
|
||||
}
|
||||
```
|
||||
|
||||
```svelte
|
||||
<!-- SearchComponent.svelte -->
|
||||
<script>
|
||||
import { onMount } from 'svelte'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
let brain = null
|
||||
let isReady = false
|
||||
let query = ''
|
||||
let results = []
|
||||
|
||||
onMount(async () => {
|
||||
brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
await brain.init()
|
||||
isReady = true
|
||||
})
|
||||
|
||||
async function search() {
|
||||
if (!isReady || !query) return
|
||||
results = await brain.find(query)
|
||||
if (!query) return
|
||||
const res = await fetch('/api/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query })
|
||||
})
|
||||
results = (await res.json()).results
|
||||
}
|
||||
</script>
|
||||
|
||||
|
|
@ -401,27 +377,23 @@ export async function POST(request) {
|
|||
|
||||
## 🌟 Solid.js Integration
|
||||
|
||||
The component calls a server endpoint (use SolidStart server routes, or any backend, to host Brainy):
|
||||
|
||||
```jsx
|
||||
import { createSignal, onMount } from 'solid-js'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { createSignal } from 'solid-js'
|
||||
|
||||
function SearchComponent() {
|
||||
const [brain, setBrain] = createSignal(null)
|
||||
const [isReady, setIsReady] = createSignal(false)
|
||||
const [query, setQuery] = createSignal('')
|
||||
const [results, setResults] = createSignal([])
|
||||
|
||||
onMount(async () => {
|
||||
const newBrain = new Brainy()
|
||||
await newBrain.init()
|
||||
setBrain(newBrain)
|
||||
setIsReady(true)
|
||||
})
|
||||
|
||||
const search = async () => {
|
||||
if (!isReady() || !query()) return
|
||||
const searchResults = await brain().find(query())
|
||||
setResults(searchResults)
|
||||
if (!query()) return
|
||||
const res = await fetch('/api/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query: query() })
|
||||
})
|
||||
setResults((await res.json()).results)
|
||||
}
|
||||
|
||||
return (
|
||||
|
|
@ -450,57 +422,25 @@ function SearchComponent() {
|
|||
|
||||
## 📦 Bundler Configuration
|
||||
|
||||
### Vite (Recommended)
|
||||
Brainy is a server-side dependency, so keep it out of client bundles. Import it only from server-only modules (`*.server.js`, API routes, server components, server actions). If your bundler ever tries to pull Brainy into a client bundle, that's a sign it's being imported from a client component — move the import to a server module.
|
||||
|
||||
For server builds, mark Brainy as external so the bundler doesn't inline it:
|
||||
|
||||
```javascript
|
||||
// vite.config.js
|
||||
// vite.config.js (SSR build)
|
||||
import { defineConfig } from 'vite'
|
||||
|
||||
export default defineConfig({
|
||||
define: {
|
||||
global: 'globalThis'
|
||||
},
|
||||
optimizeDeps: {
|
||||
include: ['@soulcraft/brainy']
|
||||
ssr: {
|
||||
external: ['@soulcraft/brainy']
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Webpack
|
||||
|
||||
```javascript
|
||||
// webpack.config.js
|
||||
module.exports = {
|
||||
resolve: {
|
||||
fallback: {
|
||||
"fs": false,
|
||||
"path": require.resolve("path-browserify"),
|
||||
"crypto": require.resolve("crypto-browserify")
|
||||
}
|
||||
},
|
||||
plugins: [
|
||||
new webpack.ProvidePlugin({
|
||||
global: 'global'
|
||||
})
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Rollup
|
||||
|
||||
```javascript
|
||||
// rollup.config.js
|
||||
import { nodeResolve } from '@rollup/plugin-node-resolve'
|
||||
import commonjs from '@rollup/plugin-commonjs'
|
||||
|
||||
// rollup.config.js (server bundle)
|
||||
export default {
|
||||
plugins: [
|
||||
nodeResolve({
|
||||
browser: true,
|
||||
preferBuiltins: false
|
||||
}),
|
||||
commonjs()
|
||||
]
|
||||
external: ['@soulcraft/brainy', 'node:fs', 'node:path', 'node:crypto']
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -508,30 +448,24 @@ export default {
|
|||
|
||||
### Server-Side Rendering
|
||||
|
||||
Instantiate Brainy on the server and feed its results into the rendered page. Never construct it in client code.
|
||||
|
||||
```javascript
|
||||
// Check if running in browser
|
||||
if (typeof window !== 'undefined') {
|
||||
// Browser-only code
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
}
|
||||
// Server-side data loading (framework loader / getServerSideProps / load fn)
|
||||
import { getBrain } from './brain.server'
|
||||
|
||||
// Or use dynamic imports
|
||||
const initBrainForBrowser = async () => {
|
||||
if (typeof window === 'undefined') return null
|
||||
|
||||
const { Brainy } = await import('@soulcraft/brainy')
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
return brain
|
||||
export async function load({ url }) {
|
||||
const brain = await getBrain()
|
||||
const query = url.searchParams.get('q') ?? ''
|
||||
const results = query ? await brain.find(query) : []
|
||||
return { results }
|
||||
}
|
||||
```
|
||||
|
||||
### Static Site Generation
|
||||
|
||||
```javascript
|
||||
// For build-time usage
|
||||
// For build-time usage (runs in Node during the build)
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export async function generateStaticProps() {
|
||||
|
|
@ -552,67 +486,46 @@ export async function generateStaticProps() {
|
|||
## 🔧 Framework-Specific Tips
|
||||
|
||||
### React
|
||||
- Use `useCallback` for search functions to prevent re-renders
|
||||
- Consider `useMemo` for expensive brain operations
|
||||
- Implement cleanup in `useEffect` for proper memory management
|
||||
- Keep components client-side and call a Brainy-backed API route
|
||||
- Use `useCallback` for fetch handlers to prevent re-renders
|
||||
- Debounce keystroke-driven searches before hitting the endpoint
|
||||
|
||||
### Vue
|
||||
- Use `shallowRef` for the brain instance (it's not reactive data)
|
||||
- Consider Pinia for global brain state management
|
||||
- Use `watchEffect` for reactive search queries
|
||||
- Components call an endpoint; the shared instance lives in a server module
|
||||
- Consider Pinia for caching results client-side
|
||||
- Debounce reactive search queries
|
||||
|
||||
### Angular
|
||||
- Implement proper dependency injection with services
|
||||
- Use RxJS observables for reactive search
|
||||
- Consider lazy loading brain in feature modules
|
||||
- Use `HttpClient` and RxJS to call the backend
|
||||
- Hold the shared Brainy instance in your Node backend, not the app
|
||||
- Consider lazy loading search features in feature modules
|
||||
|
||||
### Next.js
|
||||
- Use dynamic imports for client-side only features
|
||||
- Consider API routes for server-side brain operations
|
||||
- Implement proper error boundaries
|
||||
- Put Brainy in server-only modules (`*.server.js`), API routes, or server actions
|
||||
- Reuse one shared instance across requests
|
||||
- Implement proper error boundaries for failed fetches
|
||||
|
||||
## 🚨 Common Issues & Solutions
|
||||
|
||||
### Issue: "crypto is not defined"
|
||||
**Solution**: Your framework should handle this automatically. If not:
|
||||
```javascript
|
||||
// Add to your bundle config
|
||||
define: {
|
||||
global: 'globalThis'
|
||||
}
|
||||
```
|
||||
### Issue: "fs module not found" / "crypto is not defined" in the browser
|
||||
**Cause**: Brainy was imported into a client bundle. Brainy 8.0 is a server-side library (Node 22+/Bun) and uses Node built-ins like `fs` and `crypto`.
|
||||
**Solution**: Import Brainy only from server code — server-only modules (`*.server.js`), API routes, server components, or server actions. From client components, call those endpoints instead.
|
||||
|
||||
### Issue: "fs module not found"
|
||||
**Solution**: This is expected in browsers. Use browser-compatible storage:
|
||||
```javascript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'opfs' } // Or 'memory' for development
|
||||
})
|
||||
```
|
||||
|
||||
### Issue: Large bundle size
|
||||
**Solution**: Use dynamic imports for optional features:
|
||||
```javascript
|
||||
const brain = await import('@soulcraft/brainy').then(m => new m.Brainy())
|
||||
```
|
||||
### Issue: Large client bundle size
|
||||
**Cause**: A client module is pulling in Brainy.
|
||||
**Solution**: Move the `import { Brainy } from '@soulcraft/brainy'` into a server-only module so it never reaches the browser bundle.
|
||||
|
||||
### Issue: SSR hydration mismatch
|
||||
**Solution**: Initialize brain only on client:
|
||||
```javascript
|
||||
useEffect(() => {
|
||||
// Browser-only initialization
|
||||
initBrain()
|
||||
}, [])
|
||||
```
|
||||
**Solution**: Run the search on the server (loader / server action / API route) and pass the results down as props, so server and client render the same markup.
|
||||
|
||||
## 🎯 Best Practices
|
||||
|
||||
1. **Initialize Once**: Create brain instance at app level, not component level
|
||||
2. **Use Context**: Share brain instance across components with context/providers
|
||||
3. **Handle Loading**: Always show loading states during brain initialization
|
||||
4. **Error Boundaries**: Implement proper error handling for brain operations
|
||||
5. **Memory Management**: Clean up brain instances on unmount
|
||||
6. **Storage Strategy**: Choose appropriate storage for your deployment target
|
||||
1. **Initialize Once**: Create one shared Brainy instance per server process, not per request
|
||||
2. **Server-Only**: Import Brainy only from server modules — never from client components
|
||||
3. **Endpoint Boundary**: Expose search/add through API routes or server actions
|
||||
4. **Handle Loading**: Show loading states in the client while the fetch is in flight
|
||||
5. **Error Handling**: Catch and surface failed endpoint calls gracefully
|
||||
6. **Storage**: Use `filesystem` for persistence (the default on Node) or `memory` for ephemeral/tests
|
||||
|
||||
## 📚 Next Steps
|
||||
|
||||
|
|
|
|||
|
|
@ -1743,7 +1743,7 @@ For off-site backup, snapshot `rootDirectory` from your scheduler with `gsutil r
|
|||
- HNSW Index: O(log n) search (1B entities = ~30 hops)
|
||||
- Metadata Index: O(1) filtering
|
||||
- Graph Adjacency: O(1) relationship lookups
|
||||
- Storage: Unlimited (cloud buckets)
|
||||
- Storage: Bounded by the filesystem volume backing `rootDirectory`
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
|
|
@ -1887,7 +1887,7 @@ groupBy: 'type'
|
|||
- ✅ O(1) metadata filtering
|
||||
- ✅ O(1) relationship traversal
|
||||
- ✅ Human-readable VFS structure
|
||||
- ✅ Cloud storage support (GCS/S3/R2)
|
||||
- ✅ Filesystem-backed persistence (snapshot/sync the directory for off-site backup)
|
||||
- ✅ Billion-scale performance
|
||||
- ✅ Zero mocks, production-ready!
|
||||
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ console.log('Brainy ready.')
|
|||
|
||||
## Native Acceleration (Optional)
|
||||
|
||||
For production workloads, add Cortex for Rust-accelerated SIMD distance calculations, vector quantization, and native embeddings:
|
||||
For production workloads, add Cortex for Rust-accelerated SIMD distance calculations and native embeddings:
|
||||
|
||||
```bash
|
||||
npm install @soulcraft/cortex
|
||||
|
|
|
|||
|
|
@ -108,4 +108,4 @@ await brain.find({ query: 'people who work at Anthropic' })
|
|||
- [Triple Intelligence](/docs/concepts/triple-intelligence) — understand how the query engine works
|
||||
- [The Find System](/docs/guides/find-system) — advanced queries, operators, and graph traversal
|
||||
- [API Reference](/docs/api/reference) — complete method documentation
|
||||
- [Storage Adapters](/docs/guides/storage-adapters) — S3, GCS, Azure, filesystem, OPFS
|
||||
- [Storage Adapters](/docs/guides/storage-adapters) — filesystem, memory
|
||||
|
|
|
|||
|
|
@ -271,7 +271,7 @@ Progress is reported after each batch (batch size is determined by the storage a
|
|||
|
||||
## Storage Backend Compatibility
|
||||
|
||||
Migrations work identically across all storage backends (Memory, FileSystem, S3, R2, GCS, OPFS). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter.
|
||||
Migrations work identically across all storage backends (Memory, FileSystem). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter.
|
||||
|
||||
Batch size and rate limiting are automatically configured per adapter — no tuning required.
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,8 @@
|
|||
|
||||
Complete guide to integrating Brainy with Vue.js applications, covering Vue 3, Nuxt.js, Composition API, Options API, and advanced patterns.
|
||||
|
||||
> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun, server-side only — it is not a browser library. In Vue apps, Brainy lives behind an HTTP endpoint (a Nuxt server route, or any Node/Bun backend), and your components call that endpoint. The composables, plugin, and store below are wrappers around `fetch`; the single Brainy instance is created once on the server (see [Nuxt Server Route](#nuxt-server-route)).
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Installation
|
||||
|
|
@ -15,34 +17,47 @@ npm install @soulcraft/brainy
|
|||
|
||||
### Basic Setup
|
||||
|
||||
The composable talks to a Brainy-backed endpoint (see [Nuxt Server Route](#nuxt-server-route)). It is always "ready" because there is no client-side initialization — the server owns the Brainy instance.
|
||||
|
||||
```javascript
|
||||
// src/composables/useBrainy.js
|
||||
import { ref, onMounted } from 'vue'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { ref } from 'vue'
|
||||
|
||||
const brain = ref(null)
|
||||
const isReady = ref(false)
|
||||
export function useBrainy(endpoint = '/api/brain') {
|
||||
const error = ref(null)
|
||||
|
||||
export function useBrainy() {
|
||||
onMounted(async () => {
|
||||
const search = async (query, options = {}) => {
|
||||
error.value = null
|
||||
try {
|
||||
brain.value = new Brainy({
|
||||
storage: { type: 'opfs' } // Browser storage
|
||||
const res = await fetch(`${endpoint}/search`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, options })
|
||||
})
|
||||
await brain.value.init()
|
||||
isReady.value = true
|
||||
if (!res.ok) throw new Error(`Search failed: ${res.status}`)
|
||||
return (await res.json()).results
|
||||
} catch (err) {
|
||||
error.value = err.message
|
||||
console.error('Brainy initialization failed:', err)
|
||||
console.error('Brainy search failed:', err)
|
||||
return []
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
return {
|
||||
brain: readonly(brain),
|
||||
isReady: readonly(isReady),
|
||||
error: readonly(error)
|
||||
const add = async (data, type, metadata) => {
|
||||
const res = await fetch(`${endpoint}/add`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ data, type, metadata })
|
||||
})
|
||||
return (await res.json()).id
|
||||
}
|
||||
|
||||
const stats = async () => {
|
||||
const res = await fetch(`${endpoint}/stats`)
|
||||
return await res.json()
|
||||
}
|
||||
|
||||
return { search, add, stats, error: readonly(error) }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -54,17 +69,11 @@ export function useBrainy() {
|
|||
<!-- src/components/Search.vue -->
|
||||
<template>
|
||||
<div class="search-container">
|
||||
<div v-if="!isReady" class="loading">
|
||||
<div class="spinner"></div>
|
||||
<span>Initializing AI...</span>
|
||||
</div>
|
||||
|
||||
<div v-else>
|
||||
<div class="search-input">
|
||||
<input
|
||||
v-model="query"
|
||||
@input="handleSearch"
|
||||
placeholder="Search with AI..."
|
||||
placeholder="Search..."
|
||||
class="input"
|
||||
/>
|
||||
</div>
|
||||
|
|
@ -93,7 +102,6 @@ export function useBrainy() {
|
|||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
|
|
@ -101,22 +109,21 @@ import { ref, watch } from 'vue'
|
|||
import { useBrainy } from '../composables/useBrainy'
|
||||
import { debounce } from '../utils/debounce'
|
||||
|
||||
const { brain, isReady } = useBrainy()
|
||||
const { search: searchBrain } = useBrainy()
|
||||
|
||||
const query = ref('')
|
||||
const results = ref([])
|
||||
const loading = ref(false)
|
||||
|
||||
const search = async (searchQuery) => {
|
||||
if (!isReady.value || !searchQuery.trim()) {
|
||||
if (!searchQuery.trim()) {
|
||||
results.value = []
|
||||
return
|
||||
}
|
||||
|
||||
loading.value = true
|
||||
try {
|
||||
const searchResults = await brain.value.find(searchQuery)
|
||||
results.value = searchResults
|
||||
results.value = await searchBrain(searchQuery)
|
||||
} catch (error) {
|
||||
console.error('Search error:', error)
|
||||
results.value = []
|
||||
|
|
@ -228,11 +235,7 @@ watch(query, (newQuery) => {
|
|||
<div class="data-manager">
|
||||
<h2>Data Management</h2>
|
||||
|
||||
<div v-if="!isReady" class="loading">
|
||||
Initializing...
|
||||
</div>
|
||||
|
||||
<div v-else>
|
||||
<div>
|
||||
<!-- Add Data Form -->
|
||||
<form @submit.prevent="addData" class="add-form">
|
||||
<h3>Add New Data</h3>
|
||||
|
|
@ -289,7 +292,7 @@ watch(query, (newQuery) => {
|
|||
import { ref, reactive, onMounted } from 'vue'
|
||||
import { useBrainy } from '../composables/useBrainy'
|
||||
|
||||
const { brain, isReady } = useBrainy()
|
||||
const { add, stats: fetchStats } = useBrainy()
|
||||
|
||||
const newItem = reactive({
|
||||
data: '',
|
||||
|
|
@ -299,18 +302,7 @@ const newItem = reactive({
|
|||
|
||||
const stats = ref(null)
|
||||
|
||||
onMounted(async () => {
|
||||
if (isReady.value) {
|
||||
await loadStats()
|
||||
}
|
||||
})
|
||||
|
||||
// Watch for brain readiness
|
||||
watch(isReady, async (ready) => {
|
||||
if (ready) {
|
||||
await loadStats()
|
||||
}
|
||||
})
|
||||
onMounted(loadStats)
|
||||
|
||||
const addData = async () => {
|
||||
try {
|
||||
|
|
@ -319,11 +311,7 @@ const addData = async () => {
|
|||
tags: newItem.tags.split(',').map(tag => tag.trim()).filter(Boolean)
|
||||
}
|
||||
|
||||
await brain.value.add({
|
||||
data: newItem.data,
|
||||
type: newItem.type,
|
||||
metadata
|
||||
})
|
||||
await add(newItem.data, newItem.type, metadata)
|
||||
|
||||
// Reset form
|
||||
newItem.data = ''
|
||||
|
|
@ -339,7 +327,7 @@ const addData = async () => {
|
|||
|
||||
const loadStats = async () => {
|
||||
try {
|
||||
stats.value = await brain.value.stats()
|
||||
stats.value = await fetchStats()
|
||||
} catch (error) {
|
||||
console.error('Failed to load stats:', error)
|
||||
}
|
||||
|
|
@ -435,11 +423,6 @@ button:disabled {
|
|||
<!-- src/components/SearchOptions.vue -->
|
||||
<template>
|
||||
<div class="search-container">
|
||||
<div v-if="!isReady" class="loading">
|
||||
Initializing AI...
|
||||
</div>
|
||||
|
||||
<div v-else>
|
||||
<input
|
||||
v-model="query"
|
||||
@input="handleSearch"
|
||||
|
|
@ -460,48 +443,35 @@ button:disabled {
|
|||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script>
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { debounce } from '../utils/debounce'
|
||||
|
||||
export default {
|
||||
name: 'SearchOptions',
|
||||
data() {
|
||||
return {
|
||||
brain: null,
|
||||
isReady: false,
|
||||
query: '',
|
||||
results: [],
|
||||
loading: false
|
||||
}
|
||||
},
|
||||
async mounted() {
|
||||
await this.initBrain()
|
||||
},
|
||||
methods: {
|
||||
async initBrain() {
|
||||
try {
|
||||
this.brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
await this.brain.init()
|
||||
this.isReady = true
|
||||
} catch (error) {
|
||||
console.error('Brain initialization failed:', error)
|
||||
}
|
||||
},
|
||||
async search(query) {
|
||||
if (!this.isReady || !query.trim()) {
|
||||
if (!query.trim()) {
|
||||
this.results = []
|
||||
return
|
||||
}
|
||||
|
||||
this.loading = true
|
||||
try {
|
||||
this.results = await this.brain.find(query)
|
||||
const res = await fetch('/api/brain/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query })
|
||||
})
|
||||
this.results = (await res.json()).results
|
||||
} catch (error) {
|
||||
console.error('Search error:', error)
|
||||
this.results = []
|
||||
|
|
@ -520,10 +490,6 @@ export default {
|
|||
this.loading = false
|
||||
}
|
||||
}
|
||||
},
|
||||
beforeUnmount() {
|
||||
// Cleanup if needed
|
||||
this.brain = null
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
|
@ -533,31 +499,22 @@ export default {
|
|||
|
||||
### Global Brainy Plugin
|
||||
|
||||
The plugin exposes a global `$searchBrain` helper that calls your Brainy-backed endpoint. Brainy itself runs on the server (see [Nuxt Server Route](#nuxt-server-route)).
|
||||
|
||||
```javascript
|
||||
// src/plugins/brainy.js
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export default {
|
||||
install(app, options = {}) {
|
||||
const defaultOptions = {
|
||||
storage: { type: 'opfs' },
|
||||
...options
|
||||
}
|
||||
const endpoint = options.endpoint ?? '/api/brain'
|
||||
|
||||
const brain = new Brainy(defaultOptions)
|
||||
|
||||
// Make brain available globally
|
||||
app.config.globalProperties.$brain = brain
|
||||
app.provide('brain', brain)
|
||||
|
||||
// Auto-initialize
|
||||
brain.init().catch(error => {
|
||||
console.error('Brainy plugin initialization failed:', error)
|
||||
// Add global search method (calls the server endpoint)
|
||||
app.config.globalProperties.$searchBrain = async (query, searchOptions) => {
|
||||
const res = await fetch(`${endpoint}/search`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, options: searchOptions })
|
||||
})
|
||||
|
||||
// Add global method
|
||||
app.config.globalProperties.$searchBrain = async (query, options) => {
|
||||
return await brain.find(query, options)
|
||||
return (await res.json()).results
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -574,7 +531,7 @@ import BrainyPlugin from './plugins/brainy'
|
|||
const app = createApp(App)
|
||||
|
||||
app.use(BrainyPlugin, {
|
||||
storage: { type: 'opfs' }
|
||||
endpoint: '/api/brain'
|
||||
})
|
||||
|
||||
app.mount('#app')
|
||||
|
|
@ -611,24 +568,58 @@ export default {
|
|||
|
||||
## 🏰 Nuxt.js Integration
|
||||
|
||||
### Nuxt Plugin
|
||||
Nuxt's server engine (Nitro) is the natural home for Brainy: it runs on Node/Bun, so the Brainy instance lives in `server/`, and pages/components call its routes.
|
||||
|
||||
### Nuxt Server Route
|
||||
|
||||
```javascript
|
||||
// plugins/brainy.client.js
|
||||
// server/utils/brain.js (server-only — Nitro never bundles this into the client)
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export default defineNuxtPlugin(async () => {
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'opfs' }
|
||||
})
|
||||
let brainPromise
|
||||
|
||||
export function getBrain() {
|
||||
if (!brainPromise) {
|
||||
brainPromise = (async () => {
|
||||
// new Brainy() auto-detects filesystem persistence on Node
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
return brain
|
||||
})()
|
||||
}
|
||||
return brainPromise
|
||||
}
|
||||
```
|
||||
|
||||
return {
|
||||
provide: {
|
||||
brain
|
||||
}
|
||||
}
|
||||
```javascript
|
||||
// server/api/brain/search.post.js
|
||||
import { getBrain } from '../../utils/brain'
|
||||
|
||||
export default defineEventHandler(async (event) => {
|
||||
const { query, options } = await readBody(event)
|
||||
const brain = await getBrain()
|
||||
return { results: await brain.find(query, options) }
|
||||
})
|
||||
```
|
||||
|
||||
```javascript
|
||||
// server/api/brain/add.post.js
|
||||
import { getBrain } from '../../utils/brain'
|
||||
|
||||
export default defineEventHandler(async (event) => {
|
||||
const { data, type, metadata } = await readBody(event)
|
||||
const brain = await getBrain()
|
||||
return { id: await brain.add({ data, type, metadata }) }
|
||||
})
|
||||
```
|
||||
|
||||
```javascript
|
||||
// server/api/brain/stats.get.js
|
||||
import { getBrain } from '../../utils/brain'
|
||||
|
||||
export default defineEventHandler(async () => {
|
||||
const brain = await getBrain()
|
||||
return await brain.stats()
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -637,26 +628,25 @@ export default defineNuxtPlugin(async () => {
|
|||
```javascript
|
||||
// composables/useBrainy.js
|
||||
export const useBrainy = () => {
|
||||
const { $brain } = useNuxtApp()
|
||||
|
||||
const search = async (query, options = {}) => {
|
||||
return await $brain.find(query, options)
|
||||
return (await $fetch('/api/brain/search', {
|
||||
method: 'POST',
|
||||
body: { query, options }
|
||||
})).results
|
||||
}
|
||||
|
||||
const add = async (data, type, metadata) => {
|
||||
return await $brain.add({ data, type, metadata })
|
||||
return (await $fetch('/api/brain/add', {
|
||||
method: 'POST',
|
||||
body: { data, type, metadata }
|
||||
})).id
|
||||
}
|
||||
|
||||
const stats = async () => {
|
||||
return await $brain.stats()
|
||||
return await $fetch('/api/brain/stats')
|
||||
}
|
||||
|
||||
return {
|
||||
brain: $brain,
|
||||
search,
|
||||
add,
|
||||
stats
|
||||
}
|
||||
return { search, add, stats }
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -666,13 +656,8 @@ export const useBrainy = () => {
|
|||
<!-- pages/search.vue -->
|
||||
<template>
|
||||
<div>
|
||||
<h1>AI Search</h1>
|
||||
<h1>Search</h1>
|
||||
|
||||
<div v-if="pending" class="loading">
|
||||
Initializing AI...
|
||||
</div>
|
||||
|
||||
<div v-else>
|
||||
<input
|
||||
v-model="query"
|
||||
@input="handleSearch"
|
||||
|
|
@ -688,7 +673,6 @@ export const useBrainy = () => {
|
|||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup>
|
||||
|
|
@ -697,12 +681,6 @@ const { search } = useBrainy()
|
|||
const query = ref('')
|
||||
const results = ref([])
|
||||
const searching = ref(false)
|
||||
const pending = ref(true)
|
||||
|
||||
// Initialize
|
||||
onMounted(() => {
|
||||
pending.value = false
|
||||
})
|
||||
|
||||
const handleSearch = debounce(async () => {
|
||||
if (!query.value.trim()) {
|
||||
|
|
@ -722,82 +700,58 @@ const handleSearch = debounce(async () => {
|
|||
</script>
|
||||
```
|
||||
|
||||
### Nuxt Configuration
|
||||
|
||||
```javascript
|
||||
// nuxt.config.ts
|
||||
export default defineNuxtConfig({
|
||||
ssr: false, // Disable SSR for browser-only features
|
||||
|
||||
// Or use client-side hydration
|
||||
nitro: {
|
||||
experimental: {
|
||||
wasm: true
|
||||
}
|
||||
},
|
||||
|
||||
vite: {
|
||||
define: {
|
||||
global: 'globalThis'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## 🛠️ Advanced Patterns
|
||||
|
||||
### Global State Management with Pinia
|
||||
|
||||
The store caches results and stats client-side; all Brainy work happens behind the server endpoints.
|
||||
|
||||
```javascript
|
||||
// stores/brainy.js
|
||||
import { defineStore } from 'pinia'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
|
||||
export const useBrainyStore = defineStore('brainy', () => {
|
||||
const brain = ref(null)
|
||||
const isReady = ref(false)
|
||||
const error = ref(null)
|
||||
const stats = ref(null)
|
||||
|
||||
const init = async (options = {}) => {
|
||||
const search = async (query, options = {}) => {
|
||||
error.value = null
|
||||
try {
|
||||
brain.value = new Brainy(options)
|
||||
await brain.value.init()
|
||||
isReady.value = true
|
||||
await loadStats()
|
||||
const res = await fetch('/api/brain/search', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, options })
|
||||
})
|
||||
return (await res.json()).results
|
||||
} catch (err) {
|
||||
error.value = err.message
|
||||
console.error('Brainy initialization failed:', err)
|
||||
throw err
|
||||
}
|
||||
}
|
||||
|
||||
const search = async (query, options = {}) => {
|
||||
if (!isReady.value) throw new Error('Brain not ready')
|
||||
return await brain.value.find(query, options)
|
||||
}
|
||||
|
||||
const add = async (data, type, metadata) => {
|
||||
if (!isReady.value) throw new Error('Brain not ready')
|
||||
const id = await brain.value.add({ data, type, metadata })
|
||||
const res = await fetch('/api/brain/add', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ data, type, metadata })
|
||||
})
|
||||
const { id } = await res.json()
|
||||
await loadStats() // Refresh stats
|
||||
return id
|
||||
}
|
||||
|
||||
const loadStats = async () => {
|
||||
if (!isReady.value) return
|
||||
try {
|
||||
stats.value = await brain.value.stats()
|
||||
const res = await fetch('/api/brain/stats')
|
||||
stats.value = await res.json()
|
||||
} catch (err) {
|
||||
console.error('Failed to load stats:', err)
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
brain: readonly(brain),
|
||||
isReady: readonly(isReady),
|
||||
error: readonly(error),
|
||||
stats: readonly(stats),
|
||||
init,
|
||||
search,
|
||||
add,
|
||||
loadStats
|
||||
|
|
@ -867,7 +821,7 @@ const showResults = ref(false)
|
|||
const selectedIndex = ref(-1)
|
||||
|
||||
const search = async (searchQuery) => {
|
||||
if (!brainyStore.isReady || !searchQuery.trim()) {
|
||||
if (!searchQuery.trim()) {
|
||||
results.value = []
|
||||
return
|
||||
}
|
||||
|
|
@ -1166,21 +1120,23 @@ onMounted(() => {
|
|||
|
||||
### Component Testing with Vitest
|
||||
|
||||
The component talks to the server over `fetch`, so mock the endpoint, not Brainy itself.
|
||||
|
||||
```javascript
|
||||
// tests/components/Search.test.js
|
||||
import { mount } from '@vue/test-utils'
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest'
|
||||
import Search from '../src/components/Search.vue'
|
||||
|
||||
// Mock Brainy
|
||||
vi.mock('@soulcraft/brainy', () => ({
|
||||
Brainy: vi.fn().mockImplementation(() => ({
|
||||
init: vi.fn().mockResolvedValue(undefined),
|
||||
find: vi.fn().mockResolvedValue([
|
||||
{ id: '1', data: 'Test result', score: 0.9 }
|
||||
])
|
||||
}))
|
||||
}))
|
||||
// Mock the server endpoint
|
||||
beforeEach(() => {
|
||||
global.fetch = vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
results: [{ id: '1', data: 'Test result', score: 0.9 }]
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('Search Component', () => {
|
||||
let wrapper
|
||||
|
|
@ -1193,14 +1149,7 @@ describe('Search Component', () => {
|
|||
expect(wrapper.find('input').exists()).toBe(true)
|
||||
})
|
||||
|
||||
it('shows loading state initially', () => {
|
||||
expect(wrapper.text()).toContain('Initializing AI')
|
||||
})
|
||||
|
||||
it('performs search when input changes', async () => {
|
||||
// Wait for brain to initialize
|
||||
await wrapper.vm.$nextTick()
|
||||
|
||||
const input = wrapper.find('input')
|
||||
await input.setValue('test query')
|
||||
await input.trigger('input')
|
||||
|
|
@ -1208,6 +1157,7 @@ describe('Search Component', () => {
|
|||
// Wait for debounced search
|
||||
await new Promise(resolve => setTimeout(resolve, 350))
|
||||
|
||||
expect(global.fetch).toHaveBeenCalled()
|
||||
expect(wrapper.text()).toContain('Test result')
|
||||
})
|
||||
})
|
||||
|
|
@ -1241,6 +1191,8 @@ test('search functionality works', async ({ page }) => {
|
|||
|
||||
### Bundle Optimization
|
||||
|
||||
Keep Brainy out of the client bundle — it belongs to the server build only. Mark it external for SSR so the bundler resolves it at runtime instead of inlining it.
|
||||
|
||||
```javascript
|
||||
// vite.config.js
|
||||
import { defineConfig } from 'vite'
|
||||
|
|
@ -1248,63 +1200,46 @@ import vue from '@vitejs/plugin-vue'
|
|||
|
||||
export default defineConfig({
|
||||
plugins: [vue()],
|
||||
define: {
|
||||
global: 'globalThis'
|
||||
},
|
||||
optimizeDeps: {
|
||||
include: ['@soulcraft/brainy']
|
||||
},
|
||||
build: {
|
||||
rollupOptions: {
|
||||
output: {
|
||||
manualChunks: {
|
||||
'brainy': ['@soulcraft/brainy']
|
||||
}
|
||||
}
|
||||
}
|
||||
ssr: {
|
||||
external: ['@soulcraft/brainy']
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
Wrap the endpoint call with retry/backoff so transient network failures don't surface to the user.
|
||||
|
||||
```javascript
|
||||
// src/composables/useBrainyWithErrorHandling.js
|
||||
import { ref, onMounted } from 'vue'
|
||||
import { Brainy } from '@soulcraft/brainy'
|
||||
import { ref } from 'vue'
|
||||
|
||||
export function useBrainyWithErrorHandling() {
|
||||
const brain = ref(null)
|
||||
const isReady = ref(false)
|
||||
export function useBrainyWithErrorHandling(endpoint = '/api/brain') {
|
||||
const error = ref(null)
|
||||
const retryCount = ref(0)
|
||||
|
||||
const init = async () => {
|
||||
try {
|
||||
brain.value = new Brainy()
|
||||
await brain.value.init()
|
||||
isReady.value = true
|
||||
const search = async (query, options = {}, maxRetries = 3) => {
|
||||
error.value = null
|
||||
retryCount.value = 0
|
||||
for (let attempt = 0; attempt <= maxRetries; attempt++) {
|
||||
try {
|
||||
const res = await fetch(`${endpoint}/search`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ query, options })
|
||||
})
|
||||
if (!res.ok) throw new Error(`Search failed: ${res.status}`)
|
||||
return (await res.json()).results
|
||||
} catch (err) {
|
||||
error.value = err.message
|
||||
console.error('Brainy initialization failed:', err)
|
||||
|
||||
// Retry logic
|
||||
if (retryCount.value < 3) {
|
||||
retryCount.value++
|
||||
setTimeout(init, 2000 * retryCount.value)
|
||||
if (attempt === maxRetries) throw err
|
||||
// Exponential backoff before retrying
|
||||
await new Promise(resolve => setTimeout(resolve, 2000 * (attempt + 1)))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
onMounted(init)
|
||||
|
||||
return {
|
||||
brain: readonly(brain),
|
||||
isReady: readonly(isReady),
|
||||
error: readonly(error),
|
||||
retry: init
|
||||
search
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
@ -1315,6 +1250,13 @@ Here's a complete Vue 3 application structure:
|
|||
|
||||
```
|
||||
my-brainy-vue-app/
|
||||
├── server/ # Server-side (Node/Bun) — hosts Brainy
|
||||
│ ├── utils/
|
||||
│ │ └── brain.js # Shared Brainy instance (getBrain)
|
||||
│ └── api/brain/
|
||||
│ ├── search.post.js
|
||||
│ ├── add.post.js
|
||||
│ └── stats.get.js
|
||||
├── src/
|
||||
│ ├── components/
|
||||
│ │ ├── Search.vue
|
||||
|
|
@ -1336,7 +1278,7 @@ my-brainy-vue-app/
|
|||
└── package.json
|
||||
```
|
||||
|
||||
This provides a complete, production-ready Vue.js application with comprehensive Brainy integration.
|
||||
This provides a complete, production-ready Vue.js application: Brainy runs on the server, and the client talks to it over HTTP.
|
||||
|
||||
## 📚 Next Steps
|
||||
|
||||
|
|
|
|||
|
|
@ -386,8 +386,7 @@ npm install @soulcraft/brainy
|
|||
|
||||
## Requirements
|
||||
|
||||
- Node.js 18+ (for server/desktop)
|
||||
- Modern browser (for web apps)
|
||||
- Node.js 22+ or Bun (server-only)
|
||||
- Brainy 3.0+
|
||||
|
||||
## API Reference
|
||||
|
|
|
|||
|
|
@ -172,19 +172,6 @@ interface MetadataIndexOptionalHooks {
|
|||
close?: () => Promise<void> | void
|
||||
}
|
||||
|
||||
/**
|
||||
* Optional progressive-initialization surface implemented by cloud storage
|
||||
* adapters (plugin-provided in 8.0 — the built-in filesystem/memory adapters
|
||||
* complete their full init synchronously inside `init()` and don't expose
|
||||
* these). Feature-detected with `typeof x === 'function'` before invoking.
|
||||
*/
|
||||
interface StorageBackgroundInitHooks {
|
||||
/** Whether all background init tasks (bucket validation, count sync) finished. */
|
||||
isBackgroundInitComplete?: () => boolean
|
||||
/** Resolves when all background init tasks complete. */
|
||||
awaitBackgroundInit?: () => Promise<void>
|
||||
}
|
||||
|
||||
/**
|
||||
* Brainy's internal resolved configuration. `normalizeConfig()` defaults every
|
||||
* field at construction except the genuinely-optional ones listed in the
|
||||
|
|
@ -1075,13 +1062,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* This Promise is created in the constructor and resolves when init() completes.
|
||||
* It can be awaited multiple times safely - the result is cached.
|
||||
*
|
||||
* This enables reliable readiness detection for consumers,
|
||||
* especially in cloud environments where progressive initialization means
|
||||
* init() returns quickly but background tasks may still be running.
|
||||
* This enables reliable readiness detection for consumers — await it
|
||||
* before issuing queries when init() was fired without awaiting.
|
||||
*
|
||||
* @example Waiting for readiness before API calls
|
||||
* ```typescript
|
||||
* const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||
* const brain = new Brainy()
|
||||
* brain.init() // Fire and forget
|
||||
*
|
||||
* // Elsewhere in your code (e.g., API handler)
|
||||
|
|
@ -1115,82 +1101,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
return this._readyPromise
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if Brainy is fully initialized including all background tasks
|
||||
*
|
||||
* This checks both:
|
||||
* 1. Basic initialization complete (init() returned)
|
||||
* 2. Storage background tasks complete (bucket validation, count sync)
|
||||
*
|
||||
* Useful for determining if all lazy/progressive initialization is done.
|
||||
*
|
||||
* @returns true if all initialization including background tasks is complete
|
||||
*
|
||||
* @example Health check with background status
|
||||
* ```typescript
|
||||
* app.get('/health', (req, res) => {
|
||||
* res.json({
|
||||
* ready: brain.isInitialized,
|
||||
* fullyInitialized: brain.isFullyInitialized(),
|
||||
* status: brain.isFullyInitialized() ? 'ready' : 'warming'
|
||||
* })
|
||||
* })
|
||||
* ```
|
||||
*/
|
||||
isFullyInitialized(): boolean {
|
||||
if (!this.initialized) return false
|
||||
|
||||
// Check if storage has background init methods (cloud storage adapters)
|
||||
const storage = this.storage as BaseStorage & StorageBackgroundInitHooks
|
||||
if (typeof storage?.isBackgroundInitComplete === 'function') {
|
||||
return storage.isBackgroundInitComplete()
|
||||
}
|
||||
|
||||
// Non-cloud storage adapters are fully initialized after init()
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for all background initialization tasks to complete
|
||||
*
|
||||
* For cloud storage adapters with progressive initialization,
|
||||
* this waits for:
|
||||
* - Bucket/container validation
|
||||
* - Count synchronization
|
||||
* - Any other background tasks
|
||||
*
|
||||
* For non-cloud storage, this resolves immediately.
|
||||
*
|
||||
* **Use Case**: Call this when you need guaranteed consistency, such as:
|
||||
* - Before running batch operations
|
||||
* - Before reporting full system health
|
||||
* - When transitioning from "initializing" to "ready" status
|
||||
*
|
||||
* @returns Promise that resolves when all background tasks complete
|
||||
*
|
||||
* @example Ensuring full initialization
|
||||
* ```typescript
|
||||
* const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||
* await brain.init() // Fast return in cloud (<200ms)
|
||||
*
|
||||
* // Optional: wait for background tasks if needed
|
||||
* await brain.awaitBackgroundInit()
|
||||
* console.log('All background tasks complete')
|
||||
* ```
|
||||
*/
|
||||
async awaitBackgroundInit(): Promise<void> {
|
||||
// Must be initialized first
|
||||
await this.ready
|
||||
|
||||
// Check if storage has background init methods (cloud storage adapters)
|
||||
const storage = this.storage as BaseStorage & StorageBackgroundInitHooks
|
||||
if (typeof storage?.awaitBackgroundInit === 'function') {
|
||||
await storage.awaitBackgroundInit()
|
||||
}
|
||||
|
||||
// Non-cloud storage: no background init to wait for
|
||||
}
|
||||
|
||||
// ============= CORE CRUD OPERATIONS =============
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -1,54 +0,0 @@
|
|||
/**
|
||||
* Zero-Configuration System
|
||||
* Main entry point for all auto-configuration features
|
||||
*/
|
||||
|
||||
// Model configuration (simplified - always Q8 WASM)
|
||||
export {
|
||||
getModelPrecision,
|
||||
shouldAutoDownloadModels,
|
||||
getModelPath
|
||||
} from './modelAutoConfig.js'
|
||||
|
||||
// Storage configuration
|
||||
export {
|
||||
autoDetectStorage,
|
||||
StorageType,
|
||||
StoragePreset,
|
||||
StorageConfigResult,
|
||||
logStorageConfig,
|
||||
// Backward compatibility types
|
||||
type StorageTypeString,
|
||||
type StoragePresetString
|
||||
} from './storageAutoConfig.js'
|
||||
|
||||
// Shared configuration for multi-instance
|
||||
export {
|
||||
SharedConfig,
|
||||
SharedConfigManager
|
||||
} from './sharedConfigManager.js'
|
||||
|
||||
// Main zero-config processor
|
||||
export {
|
||||
BrainyZeroConfig,
|
||||
processZeroConfig,
|
||||
createEmbeddingFunctionWithPrecision
|
||||
} from './zeroConfig.js'
|
||||
|
||||
/**
|
||||
* Main zero-config processor
|
||||
* This is what Brainy will call
|
||||
*/
|
||||
export async function applyZeroConfig(input?: string | any): Promise<any> {
|
||||
// Handle legacy config (full object) by detecting known legacy properties
|
||||
if (input && typeof input === 'object' &&
|
||||
(input.storage?.forceMemoryStorage || input.storage?.forceFileSystemStorage || input.storage?.s3Storage)) {
|
||||
// This is a legacy config object - pass through unchanged
|
||||
console.log('📦 Using legacy configuration format')
|
||||
return input
|
||||
}
|
||||
|
||||
// Process as zero-config (includes new object format)
|
||||
const { processZeroConfig } = await import('./zeroConfig.js')
|
||||
return processZeroConfig(input)
|
||||
}
|
||||
|
|
@ -1,71 +0,0 @@
|
|||
/**
|
||||
* Model Configuration
|
||||
* Brainy uses Q8 WASM embeddings - no configuration needed (zero-config)
|
||||
*/
|
||||
|
||||
import { isNode } from '../utils/environment.js'
|
||||
|
||||
interface ModelConfigResult {
|
||||
precision: 'q8'
|
||||
reason: string
|
||||
autoSelected: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Get model precision configuration
|
||||
* Always returns Q8 - the optimal balance of size and accuracy
|
||||
*/
|
||||
export function getModelPrecision(): ModelConfigResult {
|
||||
return {
|
||||
precision: 'q8',
|
||||
reason: 'Q8 WASM (23MB bundled, no downloads)',
|
||||
autoSelected: true
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if running in a serverless environment
|
||||
*/
|
||||
function isServerlessEnvironment(): boolean {
|
||||
if (!isNode()) return false
|
||||
|
||||
return !!(
|
||||
process.env.AWS_LAMBDA_FUNCTION_NAME ||
|
||||
process.env.VERCEL ||
|
||||
process.env.NETLIFY ||
|
||||
process.env.CLOUDFLARE_WORKERS ||
|
||||
process.env.FUNCTIONS_WORKER_RUNTIME ||
|
||||
process.env.K_SERVICE // Google Cloud Run
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if models need to be downloaded
|
||||
* With bundled WASM model, this is rarely needed
|
||||
*/
|
||||
export function shouldAutoDownloadModels(): boolean {
|
||||
// Model is bundled - no downloads needed in normal operation
|
||||
// This flag exists for edge cases only
|
||||
const explicitlyDisabled = process.env.BRAINY_ALLOW_REMOTE_MODELS === 'false'
|
||||
return !explicitlyDisabled
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the model path
|
||||
* With bundled WASM model, this points to the package assets
|
||||
*/
|
||||
export function getModelPath(): string {
|
||||
// Check if user explicitly set a path (for advanced users)
|
||||
if (process.env.BRAINY_MODELS_PATH) {
|
||||
return process.env.BRAINY_MODELS_PATH
|
||||
}
|
||||
|
||||
// Serverless - use /tmp for ephemeral storage
|
||||
if (isServerlessEnvironment()) {
|
||||
return '/tmp/.brainy/models'
|
||||
}
|
||||
|
||||
// Node-like runtime - use home directory for persistent storage
|
||||
const homeDir = process.env.HOME || process.env.USERPROFILE || '~'
|
||||
return `${homeDir}/.brainy/models`
|
||||
}
|
||||
|
|
@ -1,266 +0,0 @@
|
|||
/**
|
||||
* Shared Configuration Manager
|
||||
* Ensures configuration consistency across multiple instances using shared storage
|
||||
*/
|
||||
|
||||
import { StorageType } from './storageAutoConfig.js'
|
||||
import { getBrainyVersion } from '../utils/version.js'
|
||||
|
||||
export interface SharedConfig {
|
||||
// Critical parameters that MUST match across instances
|
||||
version: string
|
||||
precision: 'q8'
|
||||
dimensions: number
|
||||
hnswM: number
|
||||
hnswEfConstruction: number
|
||||
distanceFunction: string
|
||||
createdAt: string
|
||||
lastUpdated: string
|
||||
|
||||
// Informational (can differ between instances)
|
||||
instanceCount?: number
|
||||
lastAccessedBy?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Manages configuration consistency for shared storage
|
||||
*/
|
||||
export class SharedConfigManager {
|
||||
private static CONFIG_FILE = '.brainy/config.json'
|
||||
|
||||
/**
|
||||
* Load or create shared configuration
|
||||
* When connecting to existing data, this OVERRIDES auto-configuration!
|
||||
*/
|
||||
static async loadOrCreateSharedConfig(
|
||||
storage: any,
|
||||
localConfig: any
|
||||
): Promise<{ config: any; warnings: string[] }> {
|
||||
const warnings: string[] = []
|
||||
|
||||
try {
|
||||
// Check if we're using shared storage
|
||||
if (!this.isSharedStorage(localConfig.storageType)) {
|
||||
// Local storage - use local config
|
||||
return { config: localConfig, warnings: [] }
|
||||
}
|
||||
|
||||
// Try to load existing configuration from shared storage
|
||||
const existingConfig = await this.loadConfigFromStorage(storage)
|
||||
|
||||
if (existingConfig) {
|
||||
// EXISTING SHARED DATA - Must use its configuration!
|
||||
console.log('📁 Found existing shared data configuration')
|
||||
|
||||
// Check for critical mismatches
|
||||
const mismatches = this.checkCriticalMismatches(localConfig, existingConfig)
|
||||
|
||||
if (mismatches.length > 0) {
|
||||
console.warn('⚠️ Configuration override required for shared storage:')
|
||||
mismatches.forEach(m => {
|
||||
console.warn(` - ${m.param}: ${m.local} → ${m.shared} (using shared)`)
|
||||
warnings.push(`${m.param} overridden: ${m.local} → ${m.shared}`)
|
||||
})
|
||||
}
|
||||
|
||||
// Override critical parameters with shared values
|
||||
const mergedConfig = this.mergeWithSharedConfig(localConfig, existingConfig)
|
||||
|
||||
// Update last accessed
|
||||
await this.updateAccessInfo(storage, existingConfig)
|
||||
|
||||
return { config: mergedConfig, warnings }
|
||||
|
||||
} else {
|
||||
// NEW SHARED STORAGE - Save our configuration
|
||||
console.log('📝 Initializing new shared storage with configuration')
|
||||
|
||||
const sharedConfig = this.createSharedConfig(localConfig)
|
||||
await this.saveConfigToStorage(storage, sharedConfig)
|
||||
|
||||
return { config: localConfig, warnings: [] }
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('Failed to manage shared configuration:', error)
|
||||
warnings.push('Could not verify shared configuration - proceeding with caution')
|
||||
return { config: localConfig, warnings }
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if storage type is shared (multi-instance)
|
||||
*/
|
||||
private static isSharedStorage(storageType: StorageType): boolean {
|
||||
return ['s3', 'gcs', 'r2'].includes(storageType)
|
||||
}
|
||||
|
||||
/**
|
||||
* Load configuration from shared storage
|
||||
*/
|
||||
private static async loadConfigFromStorage(storage: any): Promise<SharedConfig | null> {
|
||||
try {
|
||||
const configData = await storage.get(this.CONFIG_FILE)
|
||||
if (!configData) return null
|
||||
|
||||
const config = JSON.parse(configData)
|
||||
return config as SharedConfig
|
||||
} catch (error) {
|
||||
// Config doesn't exist yet
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Save configuration to shared storage
|
||||
*/
|
||||
private static async saveConfigToStorage(storage: any, config: SharedConfig): Promise<void> {
|
||||
try {
|
||||
await storage.set(this.CONFIG_FILE, JSON.stringify(config, null, 2))
|
||||
} catch (error) {
|
||||
console.error('Failed to save shared configuration:', error)
|
||||
throw new Error('Cannot initialize shared storage without saving configuration')
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create shared configuration from local config
|
||||
*/
|
||||
private static createSharedConfig(localConfig: any): SharedConfig {
|
||||
return {
|
||||
version: getBrainyVersion(), // Brainy version
|
||||
precision: localConfig.embeddingOptions?.precision || 'fp32',
|
||||
dimensions: 384, // Fixed for all-MiniLM-L6-v2
|
||||
hnswM: localConfig.hnsw?.M || 16,
|
||||
hnswEfConstruction: localConfig.hnsw?.efConstruction || 200,
|
||||
distanceFunction: localConfig.distanceFunction || 'cosine',
|
||||
createdAt: new Date().toISOString(),
|
||||
lastUpdated: new Date().toISOString(),
|
||||
instanceCount: 1,
|
||||
lastAccessedBy: this.getInstanceIdentifier()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check for critical configuration mismatches
|
||||
*/
|
||||
private static checkCriticalMismatches(
|
||||
localConfig: any,
|
||||
sharedConfig: SharedConfig
|
||||
): Array<{ param: string; local: any; shared: any }> {
|
||||
const mismatches: Array<{ param: string; local: any; shared: any }> = []
|
||||
|
||||
// Model precision - CRITICAL!
|
||||
const localPrecision = localConfig.embeddingOptions?.precision || 'fp32'
|
||||
if (localPrecision !== sharedConfig.precision) {
|
||||
mismatches.push({
|
||||
param: 'Model Precision',
|
||||
local: localPrecision,
|
||||
shared: sharedConfig.precision
|
||||
})
|
||||
}
|
||||
|
||||
// HNSW parameters - Important for index consistency
|
||||
const localM = localConfig.hnsw?.M || 16
|
||||
if (localM !== sharedConfig.hnswM) {
|
||||
mismatches.push({
|
||||
param: 'HNSW M',
|
||||
local: localM,
|
||||
shared: sharedConfig.hnswM
|
||||
})
|
||||
}
|
||||
|
||||
const localEf = localConfig.hnsw?.efConstruction || 200
|
||||
if (localEf !== sharedConfig.hnswEfConstruction) {
|
||||
mismatches.push({
|
||||
param: 'HNSW efConstruction',
|
||||
local: localEf,
|
||||
shared: sharedConfig.hnswEfConstruction
|
||||
})
|
||||
}
|
||||
|
||||
// Distance function
|
||||
const localDistance = localConfig.distanceFunction || 'cosine'
|
||||
if (localDistance !== sharedConfig.distanceFunction) {
|
||||
mismatches.push({
|
||||
param: 'Distance Function',
|
||||
local: localDistance,
|
||||
shared: sharedConfig.distanceFunction
|
||||
})
|
||||
}
|
||||
|
||||
return mismatches
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge local config with shared config (shared takes precedence for critical params)
|
||||
*/
|
||||
private static mergeWithSharedConfig(localConfig: any, sharedConfig: SharedConfig): any {
|
||||
return {
|
||||
...localConfig,
|
||||
|
||||
// Override critical parameters with shared values
|
||||
embeddingOptions: {
|
||||
...localConfig.embeddingOptions,
|
||||
precision: sharedConfig.precision // MUST use shared precision!
|
||||
},
|
||||
|
||||
hnsw: {
|
||||
...localConfig.hnsw,
|
||||
M: sharedConfig.hnswM,
|
||||
efConstruction: sharedConfig.hnswEfConstruction
|
||||
},
|
||||
|
||||
distanceFunction: sharedConfig.distanceFunction,
|
||||
|
||||
// Add metadata about shared configuration
|
||||
_sharedConfig: {
|
||||
loaded: true,
|
||||
version: sharedConfig.version,
|
||||
createdAt: sharedConfig.createdAt,
|
||||
precision: sharedConfig.precision
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Update access information in shared config
|
||||
*/
|
||||
private static async updateAccessInfo(storage: any, config: SharedConfig): Promise<void> {
|
||||
try {
|
||||
config.lastUpdated = new Date().toISOString()
|
||||
config.instanceCount = (config.instanceCount || 0) + 1
|
||||
config.lastAccessedBy = this.getInstanceIdentifier()
|
||||
|
||||
await this.saveConfigToStorage(storage, config)
|
||||
} catch {
|
||||
// Non-critical - don't fail if we can't update access info
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get unique identifier for this instance
|
||||
*/
|
||||
private static getInstanceIdentifier(): string {
|
||||
if (process.env.HOSTNAME) return process.env.HOSTNAME
|
||||
if (process.env.CONTAINER_ID) return process.env.CONTAINER_ID
|
||||
if (process.env.K_SERVICE) return process.env.K_SERVICE
|
||||
if (process.env.AWS_LAMBDA_FUNCTION_NAME) return process.env.AWS_LAMBDA_FUNCTION_NAME
|
||||
|
||||
// Generate a random identifier
|
||||
return `instance-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate that a configuration is compatible with shared data
|
||||
*/
|
||||
static validateCompatibility(config1: SharedConfig, config2: SharedConfig): boolean {
|
||||
return (
|
||||
config1.precision === config2.precision &&
|
||||
config1.dimensions === config2.dimensions &&
|
||||
config1.hnswM === config2.hnswM &&
|
||||
config1.hnswEfConstruction === config2.hnswEfConstruction &&
|
||||
config1.distanceFunction === config2.distanceFunction
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,113 +0,0 @@
|
|||
/**
|
||||
* @module config/storageAutoConfig
|
||||
* @description Storage configuration auto-detection for Brainy 8.0.
|
||||
*
|
||||
* Brainy 8.0 ships **two storage adapters**:
|
||||
* - `FILESYSTEM` — persistent on-disk storage (Node, Bun, Deno).
|
||||
* - `MEMORY` — in-memory, ephemeral.
|
||||
*
|
||||
* Cloud storage adapters (GCS / S3 / R2 / Azure) and the browser-only OPFS
|
||||
* adapter were removed. Cloud backup is now an operator concern handled by
|
||||
* `gsutil` / `aws s3 cp` / `rclone` / `azcopy` against the on-disk artefact.
|
||||
*
|
||||
* This module is kept thin: a two-value enum, a tiny preset enum that mirrors
|
||||
* it, and an `autoDetectStorage()` that picks filesystem when available and
|
||||
* memory otherwise.
|
||||
*/
|
||||
|
||||
import { isNode } from '../utils/environment.js'
|
||||
|
||||
/**
|
||||
* Storage backend types shipped in Brainy 8.0.
|
||||
*/
|
||||
export enum StorageType {
|
||||
MEMORY = 'memory',
|
||||
FILESYSTEM = 'filesystem',
|
||||
}
|
||||
|
||||
/**
|
||||
* High-level storage presets. `AUTO` picks filesystem when available, memory
|
||||
* otherwise; `MEMORY` and `DISK` are explicit overrides.
|
||||
*/
|
||||
export enum StoragePreset {
|
||||
AUTO = 'auto',
|
||||
MEMORY = 'memory',
|
||||
DISK = 'disk',
|
||||
}
|
||||
|
||||
export type StorageTypeString = 'memory' | 'filesystem'
|
||||
export type StoragePresetString = 'auto' | 'memory' | 'disk'
|
||||
|
||||
export interface StorageConfigResult {
|
||||
type: StorageType | StorageTypeString
|
||||
config: { type: StorageType | StorageTypeString; rootDirectory?: string }
|
||||
reason: string
|
||||
autoSelected: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a storage configuration. Accepts a {@link StorageType},
|
||||
* {@link StoragePreset}, plain string equivalent, or a free-form storage
|
||||
* config object. Returns a normalized result with the picked type, the
|
||||
* underlying config payload, a human-readable reason, and whether the
|
||||
* pick was auto-selected.
|
||||
*/
|
||||
export async function autoDetectStorage(
|
||||
override?: StorageType | StoragePreset | StorageTypeString | StoragePresetString | any
|
||||
): Promise<StorageConfigResult> {
|
||||
if (override && typeof override === 'object') {
|
||||
return {
|
||||
type: (override.type as StorageType) ?? StorageType.MEMORY,
|
||||
config: override,
|
||||
reason: 'Manually configured storage',
|
||||
autoSelected: false,
|
||||
}
|
||||
}
|
||||
|
||||
if (override === StorageType.MEMORY || override === StoragePreset.MEMORY || override === 'memory') {
|
||||
return {
|
||||
type: StorageType.MEMORY,
|
||||
config: { type: StorageType.MEMORY },
|
||||
reason: 'Explicit memory storage',
|
||||
autoSelected: false,
|
||||
}
|
||||
}
|
||||
|
||||
if (override === StorageType.FILESYSTEM || override === StoragePreset.DISK || override === 'filesystem' || override === 'disk') {
|
||||
return {
|
||||
type: StorageType.FILESYSTEM,
|
||||
config: { type: StorageType.FILESYSTEM, rootDirectory: './brainy-data' },
|
||||
reason: 'Explicit filesystem storage',
|
||||
autoSelected: false,
|
||||
}
|
||||
}
|
||||
|
||||
// Default: filesystem on Node-like runtimes, memory in browsers.
|
||||
if (isNode()) {
|
||||
return {
|
||||
type: StorageType.FILESYSTEM,
|
||||
config: { type: StorageType.FILESYSTEM, rootDirectory: './brainy-data' },
|
||||
reason: 'Auto-selected: filesystem (Node-like runtime detected)',
|
||||
autoSelected: true,
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
type: StorageType.MEMORY,
|
||||
config: { type: StorageType.MEMORY },
|
||||
reason: 'Auto-selected: memory (no Node-like runtime detected)',
|
||||
autoSelected: true,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Log a human-readable summary of a resolved storage config. Used by the
|
||||
* zero-config init path for transparency.
|
||||
*/
|
||||
export function logStorageConfig(config: StorageConfigResult, verbose: boolean = false): void {
|
||||
const prefix = config.autoSelected ? '[brainy] auto-selected' : '[brainy] using'
|
||||
console.log(`${prefix} storage: ${config.type} — ${config.reason}`)
|
||||
if (verbose) {
|
||||
console.log('[brainy] storage config:', JSON.stringify(config.config))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,354 +0,0 @@
|
|||
/**
|
||||
* Zero-Configuration System for Brainy
|
||||
* Provides intelligent defaults while preserving full control
|
||||
*/
|
||||
|
||||
import { getModelPrecision, getModelPath, shouldAutoDownloadModels } from './modelAutoConfig.js'
|
||||
import { autoDetectStorage, StorageType, StoragePreset } from './storageAutoConfig.js'
|
||||
import { AutoConfiguration } from '../utils/autoConfiguration.js'
|
||||
|
||||
/**
|
||||
* Simplified configuration interface
|
||||
* Everything is optional - zero config by default!
|
||||
*/
|
||||
export interface BrainyZeroConfig {
|
||||
/**
|
||||
* Configuration preset for common scenarios
|
||||
* - 'production': Optimized for production (disk storage, auto model, default features)
|
||||
* - 'development': Optimized for development (memory storage, fp32, verbose logging)
|
||||
* - 'minimal': Minimal footprint (memory storage, q8, minimal features)
|
||||
* - 'zero': True zero config (all auto-detected)
|
||||
* - 'writer': Writer process (read + write) — the single mutating process in
|
||||
* Brainy's multi-process model
|
||||
* - 'reader': Reader process (read-only, all mutations throw) — any number of
|
||||
* these may run against the same on-disk store as the writer
|
||||
*/
|
||||
mode?: 'production' | 'development' | 'minimal' | 'zero' | 'writer' | 'reader'
|
||||
|
||||
/**
|
||||
* Storage configuration
|
||||
* - 'memory': In-memory only (no persistence)
|
||||
* - 'disk': Local disk (filesystem or OPFS)
|
||||
* - 'cloud': Cloud storage (S3/GCS/R2 if configured)
|
||||
* - 'auto': Auto-detect best option (default)
|
||||
* - Object: Custom storage configuration
|
||||
*/
|
||||
storage?: StorageType | StoragePreset | any
|
||||
|
||||
/**
|
||||
* Feature set configuration
|
||||
* - 'minimal': Core features only (fastest startup)
|
||||
* - 'default': Standard features (balanced)
|
||||
* - 'full': All features enabled (most capable)
|
||||
* - Array: Specific features to enable
|
||||
*/
|
||||
features?: 'minimal' | 'default' | 'full' | string[]
|
||||
|
||||
/**
|
||||
* Logging verbosity
|
||||
* - true: Show configuration decisions and progress
|
||||
* - false: Silent operation (default in production)
|
||||
*/
|
||||
verbose?: boolean
|
||||
|
||||
/**
|
||||
* Advanced configuration (escape hatch for power users)
|
||||
* Any additional configuration can be passed here
|
||||
*/
|
||||
advanced?: any
|
||||
}
|
||||
|
||||
/**
|
||||
* Configuration presets for common scenarios
|
||||
*/
|
||||
const PRESETS = {
|
||||
production: {
|
||||
storage: 'disk' as const,
|
||||
features: 'default' as const,
|
||||
verbose: false
|
||||
},
|
||||
development: {
|
||||
storage: 'memory' as const,
|
||||
features: 'full' as const,
|
||||
verbose: true
|
||||
},
|
||||
minimal: {
|
||||
storage: 'memory' as const,
|
||||
features: 'minimal' as const,
|
||||
verbose: false
|
||||
},
|
||||
zero: {
|
||||
storage: 'auto' as const,
|
||||
features: 'default' as const,
|
||||
verbose: false
|
||||
},
|
||||
writer: {
|
||||
storage: 'auto' as const,
|
||||
features: 'minimal' as const,
|
||||
verbose: false,
|
||||
// The single mutating process: HybridMode (read + write).
|
||||
mode: 'writer' as const
|
||||
},
|
||||
reader: {
|
||||
storage: 'auto' as const,
|
||||
features: 'default' as const,
|
||||
verbose: false,
|
||||
// Read-only process: ReaderMode — every mutation throws.
|
||||
mode: 'reader' as const
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Feature sets configuration
|
||||
*/
|
||||
const FEATURE_SETS = {
|
||||
minimal: [
|
||||
'core',
|
||||
'search',
|
||||
'storage'
|
||||
],
|
||||
default: [
|
||||
'core',
|
||||
'search',
|
||||
'storage',
|
||||
'cache',
|
||||
'metadata-index',
|
||||
'batch-processing',
|
||||
'entity-registry',
|
||||
'request-deduplicator'
|
||||
],
|
||||
full: [
|
||||
'core',
|
||||
'search',
|
||||
'storage',
|
||||
'cache',
|
||||
'metadata-index',
|
||||
'batch-processing',
|
||||
'entity-registry',
|
||||
'request-deduplicator',
|
||||
'connection-pool',
|
||||
'wal',
|
||||
'monitoring',
|
||||
'metrics',
|
||||
'intelligent-verb-scoring',
|
||||
'triple-intelligence',
|
||||
'neural-api'
|
||||
]
|
||||
}
|
||||
|
||||
/**
|
||||
* Process zero-config input into full configuration
|
||||
*/
|
||||
export async function processZeroConfig(input?: string | BrainyZeroConfig): Promise<any> {
|
||||
let config: BrainyZeroConfig = {}
|
||||
|
||||
// Handle string shorthand (preset name)
|
||||
if (typeof input === 'string') {
|
||||
if (input in PRESETS) {
|
||||
// The `in` guard above proved the string is a preset key, which is
|
||||
// exactly the BrainyZeroConfig mode union.
|
||||
config = { mode: input as keyof typeof PRESETS }
|
||||
} else {
|
||||
throw new Error(`Unknown preset: ${input}. Valid presets: ${Object.keys(PRESETS).join(', ')}`)
|
||||
}
|
||||
} else if (input) {
|
||||
config = input
|
||||
}
|
||||
|
||||
// Apply preset if specified
|
||||
if (config.mode && config.mode in PRESETS) {
|
||||
const preset = PRESETS[config.mode]
|
||||
config = {
|
||||
...preset,
|
||||
...config,
|
||||
// Preserve explicit overrides
|
||||
storage: config.storage ?? preset.storage,
|
||||
features: config.features ?? preset.features,
|
||||
verbose: config.verbose ?? preset.verbose
|
||||
}
|
||||
}
|
||||
|
||||
// Auto-detect environment if not in preset mode
|
||||
const environment = detectEnvironmentMode()
|
||||
|
||||
// Get model configuration (always Q8 WASM)
|
||||
const modelConfig = getModelPrecision()
|
||||
|
||||
// Process storage configuration
|
||||
const storageConfig = await autoDetectStorage(config.storage)
|
||||
|
||||
// Process features configuration
|
||||
const features = processFeatures(config.features)
|
||||
|
||||
// Get auto-configuration recommendations. Brainy 8.0 ships filesystem +
|
||||
// memory only; there is no cloud-storage tier.
|
||||
const autoConfig = await AutoConfiguration.getInstance().detectAndConfigure({
|
||||
expectedDataSize: estimateDataSize(environment),
|
||||
s3Available: false,
|
||||
memoryBudget: undefined // Let it auto-detect
|
||||
})
|
||||
|
||||
// Determine verbosity
|
||||
const verbose = config.verbose ?? (process.env.NODE_ENV === 'development')
|
||||
|
||||
// Log configuration decisions if verbose
|
||||
if (verbose) {
|
||||
logConfigurationSummary({
|
||||
mode: config.mode || 'auto',
|
||||
model: modelConfig,
|
||||
storage: storageConfig,
|
||||
features: features,
|
||||
environment: environment,
|
||||
autoConfig: autoConfig
|
||||
})
|
||||
}
|
||||
|
||||
// Build final configuration
|
||||
const finalConfig: any = {
|
||||
// Model configuration
|
||||
embeddingFunction: undefined, // Will be created with correct precision
|
||||
embeddingOptions: {
|
||||
precision: modelConfig.precision,
|
||||
modelPath: getModelPath(),
|
||||
allowRemoteDownload: shouldAutoDownloadModels()
|
||||
},
|
||||
|
||||
// Storage configuration
|
||||
storage: storageConfig.config,
|
||||
storageType: storageConfig.type,
|
||||
|
||||
// HNSW configuration from auto-config
|
||||
hnsw: {
|
||||
M: autoConfig.recommendedConfig.enablePartitioning ? 32 : 16,
|
||||
efConstruction: autoConfig.recommendedConfig.enablePartitioning ? 400 : 200,
|
||||
maxDatasetSize: autoConfig.recommendedConfig.expectedDatasetSize,
|
||||
partitioning: autoConfig.recommendedConfig.enablePartitioning,
|
||||
maxNodesPerPartition: autoConfig.recommendedConfig.maxNodesPerPartition
|
||||
},
|
||||
|
||||
// Cache configuration from auto-config
|
||||
cache: {
|
||||
autoTune: true,
|
||||
hotCacheMaxSize: Math.floor(autoConfig.recommendedConfig.maxMemoryUsage / (1024 * 1024 * 10)), // 10% of memory budget
|
||||
batchSize: autoConfig.recommendedConfig.enablePartitioning ? 100 : 50
|
||||
},
|
||||
|
||||
// Features configuration
|
||||
enabledFeatures: features,
|
||||
|
||||
// Metadata index configuration
|
||||
metadataIndex: features.includes('metadata-index') ? {
|
||||
enabled: true,
|
||||
autoRebuild: true
|
||||
} : undefined,
|
||||
|
||||
// Intelligent verb scoring
|
||||
intelligentVerbScoring: features.includes('intelligent-verb-scoring') ? {
|
||||
enabled: true
|
||||
} : undefined,
|
||||
|
||||
// Logging configuration
|
||||
logging: {
|
||||
verbose: verbose
|
||||
},
|
||||
|
||||
// Performance flags from auto-config
|
||||
optimizations: autoConfig.optimizationFlags,
|
||||
|
||||
// Advanced overrides (if any)
|
||||
...config.advanced
|
||||
}
|
||||
|
||||
// Multi-process role: writer (read + write) or reader (read-only). This is the
|
||||
// only signal Brainy needs — `mode` drives the operational mode (HybridMode vs
|
||||
// ReaderMode) that gates every mutation path.
|
||||
if (config.mode === 'writer' || config.mode === 'reader') {
|
||||
finalConfig.mode = config.mode
|
||||
|
||||
if (verbose) {
|
||||
console.log(`📡 Process role: ${config.mode.toUpperCase()}`)
|
||||
}
|
||||
}
|
||||
|
||||
return finalConfig
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect environment mode if not specified
|
||||
*/
|
||||
function detectEnvironmentMode(): 'production' | 'development' | 'unknown' {
|
||||
if (process.env.NODE_ENV === 'production') return 'production'
|
||||
if (process.env.NODE_ENV === 'development') return 'development'
|
||||
if (process.env.NODE_ENV === 'test') return 'development'
|
||||
|
||||
// Check for CI environments
|
||||
if (process.env.CI || process.env.GITHUB_ACTIONS) return 'production'
|
||||
|
||||
// Check for production indicators
|
||||
if (process.env.VERCEL_ENV === 'production' ||
|
||||
process.env.NETLIFY_ENV === 'production' ||
|
||||
process.env.RAILWAY_ENVIRONMENT === 'production') {
|
||||
return 'production'
|
||||
}
|
||||
|
||||
return 'unknown'
|
||||
}
|
||||
|
||||
/**
|
||||
* Process features configuration
|
||||
*/
|
||||
function processFeatures(features?: 'minimal' | 'default' | 'full' | string[]): string[] {
|
||||
if (Array.isArray(features)) {
|
||||
return features
|
||||
}
|
||||
|
||||
if (features && features in FEATURE_SETS) {
|
||||
return FEATURE_SETS[features]
|
||||
}
|
||||
|
||||
// Default based on environment
|
||||
const env = detectEnvironmentMode()
|
||||
if (env === 'production') return FEATURE_SETS.default
|
||||
if (env === 'development') return FEATURE_SETS.full
|
||||
return FEATURE_SETS.default
|
||||
}
|
||||
|
||||
/**
|
||||
* Estimate dataset size based on environment
|
||||
*/
|
||||
function estimateDataSize(environment: string): number {
|
||||
switch (environment) {
|
||||
case 'production': return 100000
|
||||
case 'development': return 10000
|
||||
default: return 50000
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Log configuration summary
|
||||
*/
|
||||
function logConfigurationSummary(config: any): void {
|
||||
console.log('\n🧠 Brainy Zero-Config Summary')
|
||||
console.log('================================')
|
||||
console.log(`Mode: ${config.mode}`)
|
||||
console.log(`Environment: ${config.environment}`)
|
||||
console.log(`Model: ${config.model.precision.toUpperCase()} (${config.model.reason})`)
|
||||
console.log(`Storage: ${config.storage.type.toUpperCase()} (${config.storage.reason})`)
|
||||
console.log(`Features: ${config.features.length} enabled`)
|
||||
console.log(`Memory Budget: ${Math.floor(config.autoConfig.recommendedConfig.maxMemoryUsage / (1024 * 1024))}MB`)
|
||||
console.log(`Expected Dataset: ${config.autoConfig.recommendedConfig.expectedDatasetSize.toLocaleString()} items`)
|
||||
console.log('================================\n')
|
||||
}
|
||||
|
||||
/**
|
||||
* Create embedding function (always Q8 WASM)
|
||||
*/
|
||||
export async function createEmbeddingFunctionWithPrecision(): Promise<any> {
|
||||
const { createEmbeddingFunction } = await import('../utils/embedding.js')
|
||||
|
||||
// Create embedding function - always Q8 WASM
|
||||
return createEmbeddingFunction({
|
||||
precision: 'q8',
|
||||
verbose: false // Silent by default in zero-config
|
||||
})
|
||||
}
|
||||
|
|
@ -77,11 +77,6 @@ export type {
|
|||
// Export Aggregation Engine
|
||||
export { AggregationIndex, AggregateMaterializer, bucketTimestamp, parseBucketRange } from './aggregation/index.js'
|
||||
|
||||
// Export zero-configuration input type
|
||||
export {
|
||||
BrainyZeroConfig
|
||||
} from './config/index.js'
|
||||
|
||||
// Export Neural Import (AI data understanding)
|
||||
export { NeuralImport } from './neural/neuralImport.js'
|
||||
export type {
|
||||
|
|
|
|||
|
|
@ -18,44 +18,6 @@ import { StorageBatchConfig } from '../baseStorage.js'
|
|||
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
|
||||
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
|
||||
|
||||
// =============================================================================
|
||||
// Progressive Initialization Types
|
||||
// =============================================================================
|
||||
|
||||
/**
|
||||
* Initialization mode for cloud storage adapters.
|
||||
*
|
||||
* Controls how storage adapters initialize during startup, enabling
|
||||
* fast cold starts in serverless environments while maintaining strict
|
||||
* validation for local development.
|
||||
*
|
||||
*
|
||||
* | Mode | Description | Use Case |
|
||||
* |------|-------------|----------|
|
||||
* | `'auto'` | Auto-detect environment (progressive in cloud, strict locally) | Default, zero-config |
|
||||
* | `'progressive'` | Fast init (<200ms), validate lazily on first write | Serverless, Cloud Run |
|
||||
* | `'strict'` | Blocking validation during init (traditional behavior) | Local dev, testing |
|
||||
*
|
||||
* @example
|
||||
* ```typescript
|
||||
* // Zero-config: auto-detects Cloud Run, Lambda, etc.
|
||||
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
|
||||
*
|
||||
* // Force progressive mode for all environments
|
||||
* const storage = new FileSystemStorage({
|
||||
* bucket: 'my-bucket',
|
||||
* initMode: 'progressive'
|
||||
* })
|
||||
*
|
||||
* // Force strict validation (useful for testing)
|
||||
* const storage = new FileSystemStorage({
|
||||
* bucket: 'my-bucket',
|
||||
* initMode: 'strict'
|
||||
* })
|
||||
* ```
|
||||
*/
|
||||
export type InitMode = 'progressive' | 'strict' | 'auto'
|
||||
|
||||
/**
|
||||
* Base class for storage adapters that implements statistics tracking
|
||||
*/
|
||||
|
|
@ -372,71 +334,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
status: 'normal' | 'throttled' | 'recovering'
|
||||
}> = new Map()
|
||||
|
||||
// =============================================
|
||||
// Progressive Initialization State
|
||||
// =============================================
|
||||
// These properties enable fast cold starts in cloud environments
|
||||
// by deferring validation and count loading to background tasks.
|
||||
|
||||
/**
|
||||
* Initialization mode for this storage adapter.
|
||||
*
|
||||
* - `'auto'` (default): Progressive in cloud environments, strict locally
|
||||
* - `'progressive'`: Always use fast init with lazy validation
|
||||
* - `'strict'`: Always validate during init (traditional behavior)
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected initMode: InitMode = 'auto'
|
||||
|
||||
/**
|
||||
* Whether the bucket/container has been validated to exist.
|
||||
*
|
||||
* In progressive mode, this starts as `false` and is set to `true`
|
||||
* after the first successful write operation or background validation.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected bucketValidated = false
|
||||
|
||||
/**
|
||||
* Error from bucket validation, if any.
|
||||
*
|
||||
* Stored here so subsequent operations can fail fast with the same error.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected bucketValidationError: Error | null = null
|
||||
|
||||
/**
|
||||
* Whether counts have been loaded from storage.
|
||||
*
|
||||
* In progressive mode, counts start at 0 and are loaded in the background.
|
||||
* Operations work immediately; counts become accurate after background load.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected countsLoaded = false
|
||||
|
||||
/**
|
||||
* Whether all background initialization tasks have completed.
|
||||
*
|
||||
* Useful for tests and diagnostics to ensure full initialization.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected backgroundTasksComplete = false
|
||||
|
||||
/**
|
||||
* Promise that resolves when background initialization completes.
|
||||
*
|
||||
* Can be awaited by callers who need to ensure full initialization
|
||||
* before proceeding (e.g., in tests).
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected backgroundInitPromise: Promise<void> | null = null
|
||||
|
||||
// Statistics-specific methods that must be implemented by subclasses
|
||||
protected abstract saveStatisticsData(
|
||||
statistics: StatisticsData
|
||||
|
|
@ -1129,18 +1026,11 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL
|
||||
|
||||
// =============================================
|
||||
// Smart Count Batching
|
||||
// Count Persistence
|
||||
// =============================================
|
||||
|
||||
// Count batching state - mirrors statistics batching pattern
|
||||
protected pendingCountPersist = false // Counts changed since last persist?
|
||||
protected lastCountPersistTime = 0 // Timestamp of last persist
|
||||
protected scheduledCountPersistTimeout: NodeJS.Timeout | null = null // Scheduled persist timer
|
||||
protected pendingCountOperations = 0 // Operations since last persist
|
||||
|
||||
// Batching configuration (overridable by subclasses for custom strategies)
|
||||
protected countPersistBatchSize = 10 // Operations before forcing persist (cloud storage)
|
||||
protected countPersistInterval = 5000 // Milliseconds before forcing persist (cloud storage)
|
||||
// Counts changed since the last persist? Drives the write-through flush.
|
||||
protected pendingCountPersist = false
|
||||
|
||||
/**
|
||||
* Get total noun count - O(1) operation
|
||||
|
|
@ -1295,230 +1185,23 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
// =============================================
|
||||
|
||||
/**
|
||||
* Detect if this storage adapter uses cloud storage (network I/O)
|
||||
* Cloud storage benefits from batching; local storage does not.
|
||||
* Persist counts immediately.
|
||||
*
|
||||
* Override this method in subclasses for accurate detection.
|
||||
* Default implementation checks storage type from getStorageStatus().
|
||||
*
|
||||
* @returns true if cloud storage (GCS, S3, R2), false if local (File, Memory)
|
||||
*/
|
||||
protected isCloudStorage(): boolean {
|
||||
// Default: assume local storage (conservative, prefers reliability over performance)
|
||||
// Subclasses should override this for accurate detection
|
||||
return false
|
||||
}
|
||||
|
||||
// =============================================
|
||||
// Progressive Initialization Methods
|
||||
// =============================================
|
||||
|
||||
/**
|
||||
* Detect if running in a cloud/serverless environment.
|
||||
*
|
||||
* Cloud environments benefit from progressive initialization to minimize
|
||||
* cold start times. This method checks for common environment variables
|
||||
* set by cloud providers.
|
||||
*
|
||||
* | Provider | Environment Variable |
|
||||
* |----------|---------------------|
|
||||
* | Cloud Run | `K_SERVICE`, `K_REVISION` |
|
||||
* | Lambda | `AWS_LAMBDA_FUNCTION_NAME` |
|
||||
* | Cloud Functions | `FUNCTIONS_TARGET` |
|
||||
* | Azure Functions | `AZURE_FUNCTIONS_ENVIRONMENT` |
|
||||
*
|
||||
* @returns `true` if running in a detected cloud environment
|
||||
* @protected
|
||||
*/
|
||||
protected detectCloudEnvironment(): boolean {
|
||||
return !!(
|
||||
process.env.K_SERVICE || // Google Cloud Run
|
||||
process.env.K_REVISION || // Google Cloud Run (revision)
|
||||
process.env.AWS_LAMBDA_FUNCTION_NAME || // AWS Lambda
|
||||
process.env.FUNCTIONS_TARGET || // Google Cloud Functions
|
||||
process.env.AZURE_FUNCTIONS_ENVIRONMENT // Azure Functions
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Determine the effective initialization mode.
|
||||
*
|
||||
* Resolves `'auto'` to either `'progressive'` or `'strict'` based on
|
||||
* the detected environment.
|
||||
*
|
||||
* @returns The resolved init mode (`'progressive'` or `'strict'`)
|
||||
* @protected
|
||||
*/
|
||||
protected resolveInitMode(): 'progressive' | 'strict' {
|
||||
if (this.initMode === 'auto') {
|
||||
return this.detectCloudEnvironment() ? 'progressive' : 'strict'
|
||||
}
|
||||
return this.initMode
|
||||
}
|
||||
|
||||
/**
|
||||
* Schedule background initialization tasks.
|
||||
*
|
||||
* This method is called in progressive mode after the adapter is marked
|
||||
* as initialized. It schedules non-blocking tasks to:
|
||||
* 1. Validate bucket/container existence
|
||||
* 2. Load counts from storage
|
||||
*
|
||||
* Override in subclasses to add storage-specific background tasks.
|
||||
* Always call `super.scheduleBackgroundInit()` first.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected scheduleBackgroundInit(): void {
|
||||
// Create a promise that tracks all background work
|
||||
this.backgroundInitPromise = this.runBackgroundInit()
|
||||
|
||||
// Don't await - let it run in background
|
||||
this.backgroundInitPromise
|
||||
.then(() => {
|
||||
this.backgroundTasksComplete = true
|
||||
})
|
||||
.catch((error) => {
|
||||
// Log but don't throw - progressive mode is best-effort
|
||||
console.error('[Progressive Init] Background initialization failed:', error)
|
||||
this.backgroundTasksComplete = true // Mark complete even on error
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Run background initialization tasks.
|
||||
*
|
||||
* Override in subclasses to add storage-specific tasks.
|
||||
* The default implementation does nothing (for local storage adapters).
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected async runBackgroundInit(): Promise<void> {
|
||||
// Default implementation: nothing to do for local storage
|
||||
// Cloud storage adapters override this to:
|
||||
// 1. Validate bucket/container
|
||||
// 2. Load counts from storage
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for background initialization to complete.
|
||||
*
|
||||
* Useful in tests or when you need to ensure all background tasks
|
||||
* have finished before proceeding.
|
||||
*
|
||||
* @example
|
||||
* ```typescript
|
||||
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
|
||||
* await storage.init()
|
||||
*
|
||||
* // Wait for background validation and count loading
|
||||
* await storage.awaitBackgroundInit()
|
||||
*
|
||||
* // Now counts are accurate and bucket is validated
|
||||
* const count = await storage.getNounCount()
|
||||
* ```
|
||||
*
|
||||
* @public
|
||||
*/
|
||||
public async awaitBackgroundInit(): Promise<void> {
|
||||
if (this.backgroundInitPromise) {
|
||||
await this.backgroundInitPromise
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if background initialization has completed.
|
||||
*
|
||||
* @returns `true` if all background tasks are complete
|
||||
* @public
|
||||
*/
|
||||
public isBackgroundInitComplete(): boolean {
|
||||
return this.backgroundTasksComplete
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure bucket/container is validated before write operations.
|
||||
*
|
||||
* In progressive mode, bucket validation is deferred until the first
|
||||
* write operation. This method performs lazy validation and caches
|
||||
* the result (or error) for subsequent calls.
|
||||
*
|
||||
* Override in subclasses to implement storage-specific validation.
|
||||
* The base implementation does nothing (for local storage adapters).
|
||||
*
|
||||
* @throws Error if bucket validation fails
|
||||
* @protected
|
||||
*/
|
||||
protected async ensureValidatedForWrite(): Promise<void> {
|
||||
// If already validated, nothing to do
|
||||
if (this.bucketValidated) {
|
||||
return
|
||||
}
|
||||
|
||||
// If we have a cached validation error, throw it
|
||||
if (this.bucketValidationError) {
|
||||
throw this.bucketValidationError
|
||||
}
|
||||
|
||||
// Default implementation: no validation needed (local storage)
|
||||
// Cloud storage adapters override this to validate bucket/container
|
||||
this.bucketValidated = true
|
||||
}
|
||||
|
||||
/**
|
||||
* Schedule a smart batched persist operation.
|
||||
*
|
||||
* Strategy:
|
||||
* - Local Storage: Persist immediately (fast, no network latency)
|
||||
* - Cloud Storage: Batch persist (10 ops OR 5 seconds, whichever first)
|
||||
*
|
||||
* This mirrors the statistics batching pattern for consistency.
|
||||
* Filesystem and memory storage have no network latency, so counts are
|
||||
* written through on every change rather than batched.
|
||||
*/
|
||||
protected async scheduleCountPersist(): Promise<void> {
|
||||
// Mark counts as pending persist
|
||||
this.pendingCountPersist = true
|
||||
this.pendingCountOperations++
|
||||
|
||||
// Local storage: persist immediately (fast enough, no benefit from batching)
|
||||
if (!this.isCloudStorage()) {
|
||||
await this.flushCounts()
|
||||
return
|
||||
}
|
||||
|
||||
// Cloud storage: use smart batching
|
||||
// Persist if we've hit the batch size threshold
|
||||
if (this.pendingCountOperations >= this.countPersistBatchSize) {
|
||||
await this.flushCounts()
|
||||
return
|
||||
}
|
||||
|
||||
// Otherwise, schedule a time-based persist if not already scheduled
|
||||
if (!this.scheduledCountPersistTimeout) {
|
||||
this.scheduledCountPersistTimeout = setTimeout(() => {
|
||||
this.flushCounts().catch(error => {
|
||||
console.error('Failed to flush counts on timer:', error)
|
||||
})
|
||||
}, this.countPersistInterval)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Flush counts immediately to storage.
|
||||
* Flush pending counts to storage.
|
||||
*
|
||||
* Used for:
|
||||
* - Graceful shutdown (SIGTERM handler)
|
||||
* - Forced persist (batch threshold reached)
|
||||
* - Local storage immediate persist
|
||||
*
|
||||
* This is the public API that shutdown hooks can call.
|
||||
* Used for graceful shutdown (SIGTERM handler) and the immediate
|
||||
* write-through path. This is the public API that shutdown hooks can call.
|
||||
*/
|
||||
async flushCounts(): Promise<void> {
|
||||
// Clear any scheduled persist
|
||||
if (this.scheduledCountPersistTimeout) {
|
||||
clearTimeout(this.scheduledCountPersistTimeout)
|
||||
this.scheduledCountPersistTimeout = null
|
||||
}
|
||||
|
||||
// Nothing to flush?
|
||||
if (!this.pendingCountPersist) {
|
||||
return
|
||||
|
|
@ -1527,13 +1210,9 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
try {
|
||||
// Persist to storage (implemented by subclass)
|
||||
await this.persistCounts()
|
||||
|
||||
// Update state
|
||||
this.lastCountPersistTime = Date.now()
|
||||
this.pendingCountPersist = false
|
||||
this.pendingCountOperations = 0
|
||||
} catch (error) {
|
||||
console.error('❌ CRITICAL: Failed to flush counts to storage:', error)
|
||||
console.error('CRITICAL: Failed to flush counts to storage:', error)
|
||||
// Keep pending flag set so we retry on next operation
|
||||
throw error
|
||||
}
|
||||
|
|
|
|||
|
|
@ -29,17 +29,6 @@ export interface BrainyInterface<T = unknown> {
|
|||
*/
|
||||
readonly isInitialized: boolean
|
||||
|
||||
/**
|
||||
* Check if all initialization including background tasks is complete
|
||||
*/
|
||||
isFullyInitialized(): boolean
|
||||
|
||||
/**
|
||||
* Wait for all background initialization tasks to complete
|
||||
* For cloud storage adapters, this waits for bucket validation and count sync.
|
||||
*/
|
||||
awaitBackgroundInit(): Promise<void>
|
||||
|
||||
/**
|
||||
* Modern add method - unified entity creation
|
||||
* @param params Parameters for adding entities
|
||||
|
|
|
|||
|
|
@ -1,428 +0,0 @@
|
|||
/**
|
||||
* Automatic Configuration System for Brainy Vector Database
|
||||
* Detects environment, resources, and data patterns to provide optimal settings
|
||||
*/
|
||||
|
||||
import { isNode } from './environment.js'
|
||||
|
||||
export interface AutoConfigResult {
|
||||
// Environment details
|
||||
environment: 'nodejs' | 'serverless' | 'unknown'
|
||||
|
||||
// Resource detection
|
||||
availableMemory: number // bytes
|
||||
cpuCores: number
|
||||
|
||||
|
||||
// Storage capabilities
|
||||
persistentStorageAvailable: boolean
|
||||
s3StorageDetected: boolean
|
||||
|
||||
// Recommended configuration
|
||||
recommendedConfig: {
|
||||
expectedDatasetSize: number
|
||||
maxMemoryUsage: number
|
||||
targetSearchLatency: number
|
||||
enablePartitioning: boolean
|
||||
enableCompression: boolean
|
||||
enablePredictiveCaching: boolean
|
||||
partitionStrategy: 'semantic' | 'hash'
|
||||
maxNodesPerPartition: number
|
||||
semanticClusters: number
|
||||
}
|
||||
|
||||
// Performance optimization flags
|
||||
optimizationFlags: {
|
||||
useMemoryMapping: boolean
|
||||
aggressiveCaching: boolean
|
||||
backgroundOptimization: boolean
|
||||
compressionLevel: 'none' | 'light' | 'aggressive'
|
||||
}
|
||||
}
|
||||
|
||||
export interface DatasetAnalysis {
|
||||
estimatedSize: number
|
||||
vectorDimension?: number
|
||||
growthRate?: number // vectors per second
|
||||
accessPatterns?: 'read-heavy' | 'write-heavy' | 'balanced'
|
||||
}
|
||||
|
||||
/**
|
||||
* Automatic configuration system that detects environment and optimizes settings
|
||||
*/
|
||||
export class AutoConfiguration {
|
||||
private static instance: AutoConfiguration
|
||||
private cachedConfig: AutoConfigResult | null = null
|
||||
private datasetStats: DatasetAnalysis = { estimatedSize: 0 }
|
||||
|
||||
private constructor() {}
|
||||
|
||||
public static getInstance(): AutoConfiguration {
|
||||
if (!AutoConfiguration.instance) {
|
||||
AutoConfiguration.instance = new AutoConfiguration()
|
||||
}
|
||||
return AutoConfiguration.instance
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect environment and generate optimal configuration
|
||||
*/
|
||||
public async detectAndConfigure(hints?: {
|
||||
expectedDataSize?: number
|
||||
s3Available?: boolean
|
||||
memoryBudget?: number
|
||||
}): Promise<AutoConfigResult> {
|
||||
if (this.cachedConfig && !hints) {
|
||||
return this.cachedConfig
|
||||
}
|
||||
|
||||
const environment = this.detectEnvironment()
|
||||
const resources = await this.detectResources()
|
||||
const storage = await this.detectStorageCapabilities(hints?.s3Available)
|
||||
|
||||
const config: AutoConfigResult = {
|
||||
environment,
|
||||
...resources,
|
||||
...storage,
|
||||
recommendedConfig: this.generateRecommendedConfig(environment, resources, hints),
|
||||
optimizationFlags: this.generateOptimizationFlags(environment, resources)
|
||||
}
|
||||
|
||||
this.cachedConfig = config
|
||||
return config
|
||||
}
|
||||
|
||||
/**
|
||||
* Update configuration based on runtime dataset analysis
|
||||
*/
|
||||
public async adaptToDataset(analysis: DatasetAnalysis): Promise<AutoConfigResult> {
|
||||
this.datasetStats = analysis
|
||||
|
||||
// Regenerate configuration with dataset insights
|
||||
const currentConfig = await this.detectAndConfigure()
|
||||
const adaptedConfig = this.adaptConfigurationToData(currentConfig, analysis)
|
||||
|
||||
this.cachedConfig = adaptedConfig
|
||||
return adaptedConfig
|
||||
}
|
||||
|
||||
/**
|
||||
* Learn from performance metrics and adjust configuration
|
||||
*/
|
||||
public async learnFromPerformance(metrics: {
|
||||
averageSearchTime: number
|
||||
memoryUsage: number
|
||||
cacheHitRate: number
|
||||
errorRate: number
|
||||
}): Promise<Partial<AutoConfigResult['recommendedConfig']>> {
|
||||
const adjustments: Partial<AutoConfigResult['recommendedConfig']> = {}
|
||||
|
||||
// Learn from search performance
|
||||
if (metrics.averageSearchTime > 200) {
|
||||
// Too slow - optimize for speed
|
||||
adjustments.maxNodesPerPartition = Math.max(10000, (this.cachedConfig?.recommendedConfig.maxNodesPerPartition || 50000) * 0.8)
|
||||
} else if (metrics.averageSearchTime < 50) {
|
||||
// Very fast - can optimize for quality
|
||||
adjustments.maxNodesPerPartition = Math.min(100000, (this.cachedConfig?.recommendedConfig.maxNodesPerPartition || 50000) * 1.2)
|
||||
}
|
||||
|
||||
// Learn from memory usage
|
||||
if (metrics.memoryUsage > (this.cachedConfig?.recommendedConfig.maxMemoryUsage || 0) * 0.9) {
|
||||
// High memory usage - enable compression
|
||||
adjustments.enableCompression = true
|
||||
}
|
||||
|
||||
// Learn from cache performance
|
||||
if (metrics.cacheHitRate < 0.7) {
|
||||
// Poor cache performance - enable predictive caching
|
||||
adjustments.enablePredictiveCaching = true
|
||||
}
|
||||
|
||||
// Update cached config with learned adjustments
|
||||
if (this.cachedConfig) {
|
||||
this.cachedConfig.recommendedConfig = {
|
||||
...this.cachedConfig.recommendedConfig,
|
||||
...adjustments
|
||||
}
|
||||
}
|
||||
|
||||
return adjustments
|
||||
}
|
||||
|
||||
/**
|
||||
* Get minimal configuration for quick setup
|
||||
*/
|
||||
public async getQuickSetupConfig(scenario: 'small' | 'medium' | 'large' | 'enterprise'): Promise<{
|
||||
expectedDatasetSize: number
|
||||
maxMemoryUsage: number
|
||||
targetSearchLatency: number
|
||||
s3Required: boolean
|
||||
}> {
|
||||
const environment = this.detectEnvironment()
|
||||
const resources = await this.detectResources()
|
||||
|
||||
switch (scenario) {
|
||||
case 'small':
|
||||
return {
|
||||
expectedDatasetSize: 10000,
|
||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.3, 1024 * 1024 * 1024), // 1GB max
|
||||
targetSearchLatency: 100,
|
||||
s3Required: false
|
||||
}
|
||||
|
||||
case 'medium':
|
||||
return {
|
||||
expectedDatasetSize: 100000,
|
||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.5, 4 * 1024 * 1024 * 1024), // 4GB max
|
||||
targetSearchLatency: 150,
|
||||
s3Required: environment === 'serverless'
|
||||
}
|
||||
|
||||
case 'large':
|
||||
return {
|
||||
expectedDatasetSize: 1000000,
|
||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.7, 8 * 1024 * 1024 * 1024), // 8GB max
|
||||
targetSearchLatency: 200,
|
||||
s3Required: true
|
||||
}
|
||||
|
||||
case 'enterprise':
|
||||
return {
|
||||
expectedDatasetSize: 10000000,
|
||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.8, 32 * 1024 * 1024 * 1024), // 32GB max
|
||||
targetSearchLatency: 300,
|
||||
s3Required: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect the current runtime environment
|
||||
*/
|
||||
private detectEnvironment(): 'nodejs' | 'serverless' | 'unknown' {
|
||||
if (isNode()) {
|
||||
// Check for serverless environment indicators
|
||||
if (process.env.AWS_LAMBDA_FUNCTION_NAME ||
|
||||
process.env.VERCEL ||
|
||||
process.env.NETLIFY ||
|
||||
process.env.CLOUDFLARE_WORKERS) {
|
||||
return 'serverless'
|
||||
}
|
||||
return 'nodejs'
|
||||
}
|
||||
|
||||
return 'unknown'
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect available system resources
|
||||
*/
|
||||
private async detectResources(): Promise<{
|
||||
availableMemory: number
|
||||
cpuCores: number
|
||||
}> {
|
||||
let availableMemory = 2 * 1024 * 1024 * 1024 // Default 2GB
|
||||
let cpuCores = 4 // Default 4 cores
|
||||
|
||||
if (isNode()) {
|
||||
try {
|
||||
const os = await import('node:os')
|
||||
availableMemory = os.totalmem() * 0.7 // Use 70% of total memory
|
||||
cpuCores = os.cpus().length
|
||||
} catch {
|
||||
// Fallback to defaults
|
||||
}
|
||||
}
|
||||
|
||||
return { availableMemory, cpuCores }
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect available storage capabilities
|
||||
*/
|
||||
private async detectStorageCapabilities(s3Hint?: boolean): Promise<{
|
||||
persistentStorageAvailable: boolean
|
||||
s3StorageDetected: boolean
|
||||
}> {
|
||||
let persistentStorageAvailable = false
|
||||
let s3StorageDetected = s3Hint || false
|
||||
|
||||
if (isNode()) {
|
||||
persistentStorageAvailable = true
|
||||
// 8.0 dropped cloud storage adapters; the s3StorageDetected flag is
|
||||
// preserved as a hint for operator backup tooling, not adapter wiring.
|
||||
s3StorageDetected = s3Hint ||
|
||||
!!(process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY) ||
|
||||
!!(process.env.S3_BUCKET_NAME)
|
||||
}
|
||||
|
||||
return {
|
||||
persistentStorageAvailable,
|
||||
s3StorageDetected
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate recommended configuration based on detected environment and resources
|
||||
*/
|
||||
private generateRecommendedConfig(
|
||||
environment: string,
|
||||
resources: { availableMemory: number; cpuCores: number },
|
||||
hints?: { expectedDataSize?: number; memoryBudget?: number }
|
||||
): AutoConfigResult['recommendedConfig'] {
|
||||
const datasetSize = hints?.expectedDataSize || this.estimateDatasetSize()
|
||||
const memoryBudget = hints?.memoryBudget || Math.floor(resources.availableMemory * 0.6)
|
||||
|
||||
// Base configuration
|
||||
let config = {
|
||||
expectedDatasetSize: datasetSize,
|
||||
maxMemoryUsage: memoryBudget,
|
||||
targetSearchLatency: 150,
|
||||
enablePartitioning: datasetSize > 25000,
|
||||
enableCompression: memoryBudget < 2 * 1024 * 1024 * 1024,
|
||||
enablePredictiveCaching: true,
|
||||
partitionStrategy: 'semantic' as const,
|
||||
maxNodesPerPartition: 50000,
|
||||
semanticClusters: 8
|
||||
}
|
||||
|
||||
// Environment-specific adjustments
|
||||
switch (environment) {
|
||||
case 'serverless':
|
||||
config = {
|
||||
...config,
|
||||
targetSearchLatency: 500, // Account for cold starts
|
||||
enablePredictiveCaching: false, // Avoid background processes
|
||||
maxNodesPerPartition: 30000 // Moderate partition size
|
||||
}
|
||||
break
|
||||
|
||||
case 'nodejs':
|
||||
config = {
|
||||
...config,
|
||||
targetSearchLatency: 100, // Aggressive for Node.js
|
||||
maxNodesPerPartition: Math.min(100000, Math.floor(datasetSize / 10)), // Larger partitions
|
||||
semanticClusters: Math.min(16, Math.max(4, Math.floor(datasetSize / 50000))) // Scale clusters with data
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
// Dataset size adjustments
|
||||
if (datasetSize > 1000000) {
|
||||
config.semanticClusters = Math.min(32, Math.floor(datasetSize / 100000))
|
||||
config.maxNodesPerPartition = 100000
|
||||
} else if (datasetSize < 10000) {
|
||||
config.enablePartitioning = false
|
||||
config.partitionStrategy = 'semantic' // Keep semantic but disable partitioning
|
||||
}
|
||||
|
||||
return config
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate optimization flags based on environment and resources
|
||||
*/
|
||||
private generateOptimizationFlags(
|
||||
environment: string,
|
||||
resources: { availableMemory: number; cpuCores: number }
|
||||
): AutoConfigResult['optimizationFlags'] {
|
||||
return {
|
||||
useMemoryMapping: environment === 'nodejs' && resources.availableMemory > 4 * 1024 * 1024 * 1024,
|
||||
aggressiveCaching: resources.availableMemory > 2 * 1024 * 1024 * 1024,
|
||||
backgroundOptimization: environment !== 'serverless' && resources.cpuCores > 2,
|
||||
compressionLevel: resources.availableMemory < 1024 * 1024 * 1024 ? 'aggressive' :
|
||||
resources.availableMemory < 4 * 1024 * 1024 * 1024 ? 'light' : 'none'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Adapt configuration based on actual dataset analysis
|
||||
*/
|
||||
private adaptConfigurationToData(
|
||||
baseConfig: AutoConfigResult,
|
||||
analysis: DatasetAnalysis
|
||||
): AutoConfigResult {
|
||||
const updatedConfig = { ...baseConfig }
|
||||
|
||||
// Adjust based on actual dataset size
|
||||
if (analysis.estimatedSize !== baseConfig.recommendedConfig.expectedDatasetSize) {
|
||||
const sizeRatio = analysis.estimatedSize / baseConfig.recommendedConfig.expectedDatasetSize
|
||||
|
||||
updatedConfig.recommendedConfig.expectedDatasetSize = analysis.estimatedSize
|
||||
|
||||
// Scale partition size with dataset
|
||||
if (sizeRatio > 2) {
|
||||
updatedConfig.recommendedConfig.maxNodesPerPartition = Math.min(
|
||||
100000,
|
||||
Math.floor(updatedConfig.recommendedConfig.maxNodesPerPartition * 1.5)
|
||||
)
|
||||
updatedConfig.recommendedConfig.semanticClusters = Math.min(
|
||||
32,
|
||||
Math.floor(updatedConfig.recommendedConfig.semanticClusters * 1.5)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// Adjust based on vector dimension
|
||||
if (analysis.vectorDimension) {
|
||||
if (analysis.vectorDimension > 1024) {
|
||||
// High-dimensional vectors - optimize for compression
|
||||
updatedConfig.recommendedConfig.enableCompression = true
|
||||
updatedConfig.optimizationFlags.compressionLevel = 'aggressive'
|
||||
}
|
||||
}
|
||||
|
||||
// Adjust based on access patterns
|
||||
if (analysis.accessPatterns === 'read-heavy') {
|
||||
updatedConfig.recommendedConfig.enablePredictiveCaching = true
|
||||
updatedConfig.optimizationFlags.aggressiveCaching = true
|
||||
} else if (analysis.accessPatterns === 'write-heavy') {
|
||||
updatedConfig.recommendedConfig.enablePredictiveCaching = false
|
||||
updatedConfig.optimizationFlags.backgroundOptimization = false
|
||||
}
|
||||
|
||||
return updatedConfig
|
||||
}
|
||||
|
||||
/**
|
||||
* Estimate dataset size if not provided
|
||||
*/
|
||||
private estimateDatasetSize(): number {
|
||||
// Start with conservative estimate
|
||||
const environment = this.detectEnvironment()
|
||||
|
||||
switch (environment) {
|
||||
case 'serverless': return 50000
|
||||
case 'nodejs': return 100000
|
||||
default: return 25000
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset cached configuration (for testing or manual refresh)
|
||||
*/
|
||||
public resetCache(): void {
|
||||
this.cachedConfig = null
|
||||
this.datasetStats = { estimatedSize: 0 }
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience function for quick auto-configuration
|
||||
*/
|
||||
export async function autoConfigureBrainy(hints?: {
|
||||
expectedDataSize?: number
|
||||
s3Available?: boolean
|
||||
memoryBudget?: number
|
||||
}): Promise<AutoConfigResult> {
|
||||
const autoConfig = AutoConfiguration.getInstance()
|
||||
return autoConfig.detectAndConfigure(hints)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get quick setup configuration for common scenarios
|
||||
*/
|
||||
export async function getQuickSetup(scenario: 'small' | 'medium' | 'large' | 'enterprise') {
|
||||
const autoConfig = AutoConfiguration.getInstance()
|
||||
return autoConfig.getQuickSetupConfig(scenario)
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue