refactor(8.0)!: remove orphaned zero-config subsystem + dead cloud/progressive-init storage vestige

The old config-generation subsystem (src/config/ + autoConfiguration.ts) was
superseded during the 8.0 rework and never wired into init(): it emitted settings
for a partitioning subsystem that no longer exists and probed deleted cloud env
vars. The live zero-config path is inline — recall preset → HNSW knobs, storage
auto-detect, auto persistMode, container-memory-aware cache sizing.

The storage progressive-init / cloud-detection cluster was equally dead after the
cloud adapters were dropped: isCloudStorage() is permanently false (no overriders),
scheduleBackgroundInit/runBackgroundInit were never called (the latter an empty
body), initMode was never assigned, and Brainy.isFullyInitialized()/
awaitBackgroundInit() were always-trivial with zero callers. scheduleCountPersist()
collapses to its only-ever-taken immediate write-through path.

Removed:
- src/config/{index,zeroConfig,storageAutoConfig,modelAutoConfig,sharedConfigManager}.ts
- src/utils/autoConfiguration.ts + the inert BrainyZeroConfig export
- Brainy.isFullyInitialized()/awaitBackgroundInit() (+ BrainyInterface decls)
- InitMode type, isCloudStorage/detectCloudEnvironment/resolveInitMode,
  scheduleBackgroundInit/runBackgroundInit/ensureValidatedForWrite and their state
- Dead cloud env-var probes (K_SERVICE/K_REVISION/AWS_LAMBDA_FUNCTION_NAME/
  FUNCTIONS_TARGET/AZURE_FUNCTIONS_ENVIRONMENT)

Kept (verified live): production-detection logging (environment.ts), container-
memory cache sizing (memoryDetection/paramValidation), on-disk hash bucketing
(sharding.ts).

Docs: scrubbed deleted-subsystem references (JS quantization knobs, cloud/OPFS
adapters, partitioning, old zero-config API) across 14 files; deleted two wholly-
obsolete feature docs (complete-feature-list, v3-features); rewrote
architecture/zero-config for 8.0.

~3,700 LOC removed. Build clean; 1392 unit + 24 db-mvcc green.
This commit is contained in:
David Snelling 2026-06-15 11:11:21 -07:00
parent 00d3203d68
commit 35b9d7ef43
28 changed files with 596 additions and 3752 deletions

View file

@ -428,8 +428,7 @@ brainy inspect health /data/brain
See [the multi-process model](docs/concepts/multi-process.md) and the See [the multi-process model](docs/concepts/multi-process.md) and the
[inspection guide](docs/guides/inspection.md) for the full story, including [inspection guide](docs/guides/inspection.md) for the full story, including
stale-lock detection, the cross-process flush RPC, and what's not yet stale-lock detection and the cross-process flush RPC.
enforced on cloud storage backends.
## Contributing ## Contributing

View file

@ -391,13 +391,12 @@ await brain.init() // Instant (0-10ms)
### Vector Index Tuning Knobs ### Vector Index Tuning Knobs
Brainy 8.0 exposes exactly three knobs on `config.vector`: Brainy 8.0 exposes two knobs on `config.vector`:
```javascript ```javascript
const brain = new Brainy({ const brain = new Brainy({
vector: { vector: {
recall: 'fast', // 'fast' | 'balanced' | 'accurate' recall: 'fast', // 'fast' | 'balanced' | 'accurate'
quantization: { bits: 8 }, // 4 | 8 (SQ4 / SQ8)
persistMode: 'deferred' // 'immediate' | 'deferred' persistMode: 'deferred' // 'immediate' | 'deferred'
} }
}) })

View file

@ -35,10 +35,9 @@ Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery,
The three knobs that matter most: The three knobs that matter most:
1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`) 1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
2. **`config.vector.quantization`** — `{ bits: 4 | 8 }` for memory savings on the open-core JS vector index 2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
3. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
The native vector provider (via the optional `@soulcraft/cortex` package) extends this with a higher-performing index when installed. The native vector provider (via the optional `@soulcraft/cortex` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
## Storage Configurations ## Storage Configurations
@ -95,7 +94,6 @@ const brain = new Brainy({
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' }, storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
vector: { vector: {
recall: 'fast', // Trade recall for latency recall: 'fast', // Trade recall for latency
quantization: { bits: 8 }, // SQ8 quantization
persistMode: 'deferred' // Batch persistence persistMode: 'deferred' // Batch persistence
} }
}) })
@ -142,8 +140,7 @@ Your service layer handles routing and isolation; Brainy stays simple.
const brain = new Brainy({ const brain = new Brainy({
storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' }, storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
vector: { vector: {
recall: 'accurate', recall: 'accurate'
quantization: { bits: 8 }
} }
}) })
``` ```
@ -153,7 +150,6 @@ const brain = new Brainy({
| Setting | Values | When to change | | Setting | Values | When to change |
|---------|--------|----------------| |---------|--------|----------------|
| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency | | `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency |
| `vector.quantization.bits` | `4` / `8` | Smaller index, lower memory |
| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability | | `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability |
| `storage.cache.maxSize` | integer | Hot-path read cache size | | `storage.cache.maxSize` | integer | Hot-path read cache size |
| `storage.cache.ttl` | ms | Cache freshness | | `storage.cache.ttl` | ms | Cache freshness |
@ -174,14 +170,13 @@ const stats = await brain.stats()
### Issue: Slow queries ### Issue: Slow queries
1. Switch to `vector.recall: 'fast'` 1. Switch to `vector.recall: 'fast'`
2. Enable SQ8 quantization 2. Increase the read cache (`storage.cache.maxSize`)
3. Increase the read cache (`storage.cache.maxSize`) 3. Consider the optional native vector provider via `@soulcraft/cortex`
4. Consider the optional native vector provider via `@soulcraft/cortex`
### Issue: Memory pressure ### Issue: Memory pressure
1. Enable `vector.quantization: { bits: 4 }` or `{ bits: 8 }` 1. Reduce `storage.cache.maxSize`
2. Reduce `storage.cache.maxSize` 2. Move to `vector.persistMode: 'deferred'` to batch writes
3. Move to `vector.persistMode: 'deferred'` to batch writes 3. Consider the optional native vector provider via `@soulcraft/cortex` for at-scale index acceleration
### Issue: Slow startup after a crash ### Issue: Slow startup after a crash
1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage 1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage
@ -198,5 +193,5 @@ const stats = await brain.stats()
- Brainy 8.0 is a **library**, not a cluster - Brainy 8.0 is a **library**, not a cluster
- Storage adapters: `filesystem`, `memory`, `auto` - Storage adapters: `filesystem`, `memory`, `auto`
- Vector tuning: `recall`, `quantization`, `persistMode` - Vector tuning: `recall`, `persistMode`
- Backup is an operator-layer concern — snapshot `rootDirectory` - Backup is an operator-layer concern — snapshot `rootDirectory`

View file

@ -1441,10 +1441,9 @@ const brain = new Brainy({
rootDirectory: './brainy-data' rootDirectory: './brainy-data'
}, },
// Vector index configuration (3 knobs) // Vector index configuration (2 knobs)
vector: { vector: {
recall: 'balanced', // 'fast' | 'balanced' | 'accurate' recall: 'balanced', // 'fast' | 'balanced' | 'accurate'
quantization: { bits: 8 }, // 4 | 8 (SQ4 / SQ8)
persistMode: 'immediate' // 'immediate' | 'deferred' persistMode: 'immediate' // 'immediate' | 'deferred'
}, },

View file

@ -47,16 +47,13 @@ await visualizationAugmentation.graphRelationships(authors)
#### 2. **Data Portability** #### 2. **Data Portability**
```typescript ```typescript
// Export from one Brainy instance // Snapshot from one Brainy instance, restore into another —
const data = await brain1.export() // types are universally understood
const pin = brain1.now()
await pin.persist('/snapshots/brain1')
await pin.release()
// Import to another—types are universally understood const brain2 = await Brainy.load('/snapshots/brain1') // Types match perfectly
await brain2.import(data)
// Or sync between different storage backends
const cloudBrain = new Brainy({ storage: 's3' })
const localBrain = new Brainy({ storage: 'filesystem' })
await cloudBrain.sync(localBrain) // Types match perfectly
``` ```
#### 3. **AI Model Compatibility** #### 3. **AI Model Compatibility**

View file

@ -45,7 +45,7 @@ brainy-data/
### Vector Index ### Vector Index
Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph: Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph:
- **Performance**: O(log n) search complexity - **Performance**: O(log n) search complexity
- **Memory Efficient**: SQ4/SQ8 scalar quantization support - **Configurable recall**: `fast` / `balanced` / `accurate` presets trade recall for latency
- **Scalable**: Handles millions of vectors per process - **Scalable**: Handles millions of vectors per process
- **Persistent**: Serializable to storage - **Persistent**: Serializable to storage
- **Swappable**: Replace with a native implementation (such as `@soulcraft/cortex`) via the plugin system without changing application code - **Swappable**: Replace with a native implementation (such as `@soulcraft/cortex`) via the plugin system without changing application code

View file

@ -13,776 +13,145 @@ next:
# Zero Configuration & Auto-Adaptation # Zero Configuration & Auto-Adaptation
> **Status (8.0):** This document predates Brainy 8.0 and needs a rewrite before > **"Zero config by default, fully tunable when you need it."** Construct a
> republication. Large parts describe storage backends and environments that 8.0 > `Brainy()` with no options and it picks sensible, environment-aware defaults.
> removed (browser/OPFS/IndexedDB, edge KV, S3) or features that were never built > Every default below is overridable through the constructor — see the
> (model auto-selection, workload detection). Brainy 8.0 is server-only (Node.js/Bun) > [API Reference](../api/README.md#configuration).
> with two storage adapters: `memory` and `filesystem` — see
> [Storage Adapters](../concepts/storage-adapters.md) for the accurate story.
> Basic zero-config (`new Brainy()` with auto-selected storage) works as described.
## Overview ## Overview
Brainy is designed with **"Zero Config by Default, Infinite Tunability"** philosophy. It automatically detects your environment, adapts to available resources, learns from usage patterns, and optimizes itself for your specific workload—all without any configuration. Brainy 8.0 is server-only (Node.js 22+ / Bun). With no configuration it:
## Zero Configuration Magic - selects a storage adapter from the runtime,
- initializes the embedding model (all-MiniLM-L6-v2, 384 dimensions),
- builds and maintains the metadata, graph, and vector indexes,
- sizes its caches and write buffers to the detected memory budget,
- chooses a persistence mode that matches the storage backend, and
- quiets its own logging when it detects a production environment.
### Instant Start There is no public config-generation function — adaptation happens inside the
constructor and `init()`.
## Instant Start
```typescript ```typescript
import { Brainy } from 'brainy' import { Brainy } from '@soulcraft/brainy'
// That's it. No config needed. // That's it. No config needed.
const brain = new Brainy() const brain = new Brainy()
await brain.init() await brain.init()
// Brainy automatically: await brain.add({ data: 'First entity', type: 'concept' })
// ✓ Detects environment (Node.js, Browser, Edge, Deno) const results = await brain.find('first')
// ✓ Chooses optimal storage (FileSystem, OPFS, Memory)
// ✓ Downloads required models (if needed)
// ✓ Configures vector dimensions (384 optimal)
// ✓ Sets up indexing strategies
// ✓ Enables appropriate augmentations
// ✓ Configures caching layers
// ✓ Optimizes for your hardware
``` ```
### Environment Detection ✅ Available ## What Auto-Adaptation Covers
Brainy automatically detects and adapts to your runtime: ### 1. Storage auto-detection
With no `storage` option, Brainy uses `type: 'auto'`:
- **Filesystem** when running on a runtime with a writable Node filesystem and a
resolvable root directory. This is the default for typical Node/Bun servers and
persists across restarts.
- **In-memory** otherwise (no filesystem access, or an explicit memory request).
Fast, zero I/O, discarded on process exit — ideal for tests and ephemeral
caches.
8.0 ships exactly two storage adapters — `memory` and `filesystem` — plus the
`auto` selector that resolves to one of them. See
[Storage Adapters](../concepts/storage-adapters.md) for the full contract.
```typescript ```typescript
// Brainy's environment detection // Explicit override when you want a specific root
const environment = { const brain = new Brainy({
// Runtime detection storage: { type: 'filesystem', rootDirectory: './brainy-data' }
isNode: typeof process !== 'undefined',
isBrowser: typeof window !== 'undefined',
isDeno: typeof Deno !== 'undefined',
isEdge: typeof EdgeRuntime !== 'undefined',
isWebWorker: typeof WorkerGlobalScope !== 'undefined',
// Capability detection
hasFileSystem: /* auto-detected */,
hasIndexedDB: /* auto-detected */,
hasOPFS: /* auto-detected */,
hasWebGPU: /* auto-detected */,
hasWASM: /* auto-detected */,
// Resource detection
cpuCores: /* auto-detected */,
memory: /* auto-detected */,
storage: /* auto-detected */
}
```
## Auto-Adaptive Storage ✅ Available
> **Current**: Brainy automatically selects the best storage adapter for your environment.
### Storage Selection Logic
```typescript
// Brainy's intelligent storage selection
async function autoSelectStorage() {
// Server environments
if (environment.isNode) {
if (await hasWritePermission('./data')) {
return 'filesystem' // Best for servers
} else if (process.env.S3_BUCKET) {
return 's3' // Cloud deployment
} else {
return 'memory' // Fallback for restricted environments
}
}
// Browser environments
if (environment.isBrowser) {
if (await navigator.storage.estimate() > 1GB) {
return 'opfs' // Best for modern browsers
} else if (indexedDB) {
return 'indexeddb' // Fallback for older browsers
} else {
return 'memory' // In-memory for restricted contexts
}
}
// Edge environments
if (environment.isEdge) {
return 'kv' // Use edge KV stores (Cloudflare, Vercel)
}
}
```
### Storage Migration
Brainy seamlessly migrates between storage types:
```typescript
// Start with memory storage (development)
const brain = new Brainy() // Auto-selects memory
// Later, migrate to production storage
await brain.migrate({
to: 'filesystem',
path: './production-data'
}) })
// All data seamlessly transferred
``` ```
## Learning & Optimization 🚧 Coming Soon ### 2. HNSW quality from the `recall` preset
> **Note**: These features are planned for Q2 2025. Currently, Brainy uses static optimizations. Vector-index quality comes from a single preset rather than hand-tuned graph
parameters. `config.vector.recall` accepts `'fast'`, `'balanced'`, or
### Query Pattern Learning 🚧 Planned `'accurate'` and defaults to `'balanced'`. The preset maps internally to the
HNSW construction and search parameters (`M` / `efConstruction` / `efSearch`),
Brainy learns from your query patterns and optimizes accordingly: so you trade recall against latency with one knob instead of three.
```typescript ```typescript
// Brainy observes query patterns const brain = new Brainy({
class QueryPatternLearner { vector: { recall: 'fast' } // favor latency over recall
analyze(queries: Query[]) { })
return {
// Frequency analysis
mostCommonFields: this.getTopFields(queries),
avgResultSize: this.getAvgSize(queries),
temporalPatterns: this.getTimePatterns(queries),
// Relationship analysis
commonTraversals: this.getGraphPatterns(queries),
typicalDepth: this.getAvgDepth(queries),
// Performance analysis
slowQueries: this.getSlowQueries(queries),
cacheability: this.getCacheability(queries)
}
}
}
// Automatic optimizations based on learning:
// - Creates indexes for frequently queried fields
// - Pre-computes common graph traversals
// - Adjusts cache sizes based on working set
// - Optimizes vector search parameters
``` ```
### Auto-Indexing 🚧 Planned The default JS index is `JsHnswVectorIndex`. An optional native acceleration
provider (the `@soulcraft/cortex` package) can replace it with a
higher-performing implementation; the public knobs stay the same. Quantization
and other index-internal acceleration are the native provider's concern, not a
Brainy configuration option.
Brainy automatically creates indexes based on usage: ### 3. Persistence mode follows the backend
`config.vector.persistMode` accepts `'immediate'` or `'deferred'`. Left unset,
Brainy chooses for you:
- **Immediate** on filesystem storage, so the index file stays in lock-step with
the data and survives a crash.
- **Deferred** on in-memory storage, where there is nothing durable to sync to,
so writes are batched for throughput.
```typescript ```typescript
// No manual index configuration needed const brain = new Brainy({
await brain.find({ where: { category: "tech" } }) // First query vector: { persistMode: 'deferred' } // batch persistence for write-heavy loads
// Brainy notices 'category' field usage })
await brain.find({ where: { category: "science" } }) // Second query
// Pattern detected - auto-creates category index
await brain.find({ where: { category: "tech" } }) // Third query
// Now using index - 100x faster!
``` ```
### Adaptive Caching 🚧 Planned ### 4. Memory-aware cache and buffer sizing
Cache strategies adapt to your access patterns: Brainy reads the container's memory budget — `CLOUD_RUN_MEMORY`, `MEMORY_LIMIT`,
or the cgroup memory limit when running in a container — and sizes its read
caches and write buffers to fit. On a small instance it stays conservative; on a
large one it uses more of the available headroom. Query-result limits are capped
against the same budget (roughly 25 KB per result) to keep a single oversized
query from exhausting memory.
You can pin the cache explicitly:
```typescript ```typescript
class AdaptiveCache { const brain = new Brainy({
async adapt(metrics: AccessMetrics) { cache: { maxSize: 10000, ttl: 3_600_000 }
if (metrics.hitRate < 0.3) { })
// Low hit rate - switch strategy
this.strategy = 'lfu' // Least Frequently Used
} else if (metrics.workingSet > this.size) {
// Working set too large - increase size
this.size = Math.min(metrics.workingSet * 1.5, maxMemory)
} else if (metrics.temporalLocality > 0.8) {
// High temporal locality - use time-based eviction
this.strategy = 'ttl'
this.ttl = metrics.avgAccessInterval * 2
}
}
}
``` ```
## Performance Auto-Scaling 🚧 Coming Soon ### 5. Logging quiets in production
### Dynamic Batch Sizing Brainy detects production-style environments (for example `NODE_ENV` set to a
non-development value) and reduces its own log verbosity automatically. This is
Brainy adjusts batch sizes based on system load: logging-only behavior — it does not change indexing, storage, or query results.
```typescript
class DynamicBatcher {
calculateOptimalBatch() {
const cpuUsage = process.cpuUsage()
const memoryUsage = process.memoryUsage()
if (cpuUsage < 30 && memoryUsage < 50) {
return 1000 // System idle - large batches
} else if (cpuUsage < 60 && memoryUsage < 70) {
return 100 // Moderate load - medium batches
} else {
return 10 // High load - small batches
}
}
}
// Automatically applied during bulk operations
for (const item of millionItems) {
await brain.add(item) // Internally batched optimally
}
```
### Memory Management
Automatic memory pressure handling:
```typescript
class MemoryManager {
async handlePressure() {
const usage = process.memoryUsage()
const available = os.freemem()
if (available < 100 * 1024 * 1024) { // Less than 100MB free
// Emergency mode
await this.flushCaches()
await this.compactIndexes()
await this.offloadToDisk()
} else if (usage.heapUsed / usage.heapTotal > 0.9) {
// Preventive mode
await this.reduceCacheSizes()
await this.pauseBackgroundTasks()
}
}
}
```
### Connection Pooling
Automatic connection management for storage backends:
```typescript
class ConnectionPool {
async getOptimalPoolSize() {
// Adapts based on workload
const metrics = await this.getMetrics()
if (metrics.waitTime > 100) {
// Queries waiting - increase pool
this.size = Math.min(this.size * 1.5, this.maxSize)
} else if (metrics.idleConnections > this.size * 0.5) {
// Too many idle - decrease pool
this.size = Math.max(this.size * 0.7, this.minSize)
}
return this.size
}
}
```
## Model Auto-Selection
### Embedding Model Selection
Brainy chooses the best embedding model for your use case:
```typescript
async function autoSelectModel(data: Sample[]) {
const analysis = {
languages: detectLanguages(data),
domainSpecific: detectDomain(data),
averageLength: getAvgLength(data),
requiresMultilingual: languages.length > 1
}
if (analysis.requiresMultilingual) {
return 'multilingual-e5-base' // Handles 100+ languages
} else if (analysis.domainSpecific === 'code') {
return 'codebert-base' // Optimized for code
} else if (analysis.averageLength > 512) {
return 'all-mpnet-base-v2' // Better for long text
} else {
return 'all-MiniLM-L6-v2' // Fast and efficient default
}
}
```
### Model Downloading
Models are automatically downloaded when needed:
```typescript
// First use - model auto-downloads
const brain = new Brainy()
await brain.init() // Downloads model if not cached
// Intelligent model caching
const modelCache = {
location: process.env.MODEL_CACHE || '~/.brainy/models',
maxSize: 5 * 1024 * 1024 * 1024, // 5GB max
strategy: 'lru', // Least recently used eviction
// CDN selection based on location
cdn: await selectFastestCDN([
'https://cdn.brainy.io',
'https://brainy.b-cdn.net',
'https://models.huggingface.co'
])
}
```
## Workload Detection
### Pattern Recognition
Brainy identifies your workload type and optimizes:
```typescript
enum WorkloadType {
OLTP = 'oltp', // Many small transactions
OLAP = 'olap', // Analytical queries
STREAMING = 'streaming', // Real-time ingestion
BATCH = 'batch', // Bulk processing
HYBRID = 'hybrid' // Mixed workload
}
class WorkloadDetector {
detect(metrics: OperationMetrics): WorkloadType {
if (metrics.writesPerSecond > 1000 && metrics.avgWriteSize < 1024) {
return WorkloadType.STREAMING
} else if (metrics.avgQueryComplexity > 0.8 && metrics.avgResultSize > 10000) {
return WorkloadType.OLAP
} else if (metrics.batchOperations > metrics.singleOperations) {
return WorkloadType.BATCH
} else if (metrics.writeReadRatio > 0.3 && metrics.writeReadRatio < 0.7) {
return WorkloadType.HYBRID
} else {
return WorkloadType.OLTP
}
}
}
```
### Optimization Strategies
Different optimizations for different workloads:
```typescript
class WorkloadOptimizer {
optimize(workload: WorkloadType) {
switch (workload) {
case WorkloadType.STREAMING:
return {
entityRegistry: true, // Deduplication
batchSize: 1000,
walEnabled: true,
cacheSize: 'small',
indexStrategy: 'lazy'
}
case WorkloadType.OLAP:
return {
entityRegistry: false,
batchSize: 10000,
walEnabled: false,
cacheSize: 'large',
indexStrategy: 'eager',
parallelQueries: true
}
case WorkloadType.BATCH:
return {
entityRegistry: false,
batchSize: 50000,
walEnabled: false,
cacheSize: 'minimal',
indexStrategy: 'deferred'
}
default:
return this.defaultConfig
}
}
}
```
## Hardware Adaptation 🚧 Coming Soon
> **Note**: GPU acceleration and hardware optimization planned for Q3 2025.
### CPU Optimization
Adapts to available CPU resources:
```typescript
class CPUAdapter {
async optimize() {
const cores = os.cpus().length
const type = os.cpus()[0].model
// Parallel processing based on cores
this.parallelism = Math.max(1, cores - 1) // Leave one core free
// SIMD detection for vector operations
if (type.includes('Intel') || type.includes('AMD')) {
this.enableSIMD = await checkSIMDSupport()
}
// Thread pool sizing
this.threadPoolSize = cores * 2 // Optimal for I/O bound
// Vector search optimization
if (cores >= 8) {
this.hnswConstruction = 200 // Higher quality index
this.hnswSearch = 100 // More accurate search
} else {
this.hnswConstruction = 100 // Balanced
this.hnswSearch = 50 // Faster search
}
}
}
```
### Memory Adaptation
Intelligent memory allocation:
```typescript
class MemoryAdapter {
async configure() {
const totalMemory = os.totalmem()
const availableMemory = os.freemem()
// Allocate based on available memory
const allocation = {
cache: Math.min(availableMemory * 0.25, 2 * GB),
vectors: Math.min(availableMemory * 0.30, 4 * GB),
indexes: Math.min(availableMemory * 0.20, 2 * GB),
working: Math.min(availableMemory * 0.25, 2 * GB)
}
// Adjust for low memory systems
if (totalMemory < 4 * GB) {
allocation.cache *= 0.5
allocation.vectors *= 0.7
this.enableSwapping = true
}
return allocation
}
}
```
### GPU Acceleration
Automatic GPU detection and utilization:
```typescript
class GPUAdapter {
async detect() {
// WebGPU in browsers
if (navigator?.gpu) {
const adapter = await navigator.gpu.requestAdapter()
return {
available: true,
type: 'webgpu',
memory: adapter.limits.maxBufferSize,
compute: adapter.limits.maxComputeWorkgroupsPerDimension
}
}
// CUDA in Node.js
if (process.platform === 'linux' || process.platform === 'win32') {
const hasCuda = await checkCudaSupport()
if (hasCuda) {
return {
available: true,
type: 'cuda',
memory: await getCudaMemory(),
compute: await getCudaCores()
}
}
}
return { available: false }
}
async optimize(gpu: GPUInfo) {
if (gpu.available) {
// Offload vector operations to GPU
this.vectorOps = 'gpu'
this.embeddingGeneration = 'gpu'
this.matrixMultiplication = 'gpu'
// Larger batch sizes for GPU
this.batchSize = gpu.memory > 8 * GB ? 10000 : 1000
}
}
}
```
## Network Adaptation
### Bandwidth Detection
Optimizes for available network bandwidth:
```typescript
class NetworkAdapter {
async measureBandwidth() {
const testSize = 1 * MB
const start = Date.now()
await this.transfer(testSize)
const duration = Date.now() - start
const bandwidth = (testSize / duration) * 1000 // bytes/sec
if (bandwidth < 1 * MB) {
// Low bandwidth - optimize
this.compression = 'aggressive'
this.batchTransfers = true
this.cacheRemote = true
} else if (bandwidth > 100 * MB) {
// High bandwidth
this.compression = 'minimal'
this.parallelTransfers = true
}
}
}
```
### Latency Optimization
Adapts to network latency:
```typescript
class LatencyOptimizer {
async optimize() {
const latency = await this.measureLatency()
if (latency > 100) { // High latency
// Batch operations
this.minBatchSize = 100
// Aggressive prefetching
this.prefetchDepth = 3
// Local caching
this.cacheStrategy = 'aggressive'
// Connection pooling
this.connectionPool = Math.min(latency / 10, 50)
}
}
}
```
## Cloud Provider Detection 🚧 Coming Soon
> **Note**: Cloud provider auto-detection planned for Q3 2025.
### Automatic Cloud Optimization
Detects and optimizes for cloud providers:
```typescript
class CloudDetector {
async detect() {
// AWS Detection
if (process.env.AWS_REGION || await canReachMetadata('169.254.169.254')) {
return {
provider: 'aws',
instance: await getEC2InstanceType(),
region: process.env.AWS_REGION,
services: {
storage: 's3',
cache: 'elasticache',
compute: 'lambda'
}
}
}
// Google Cloud Detection
if (process.env.GOOGLE_CLOUD_PROJECT || await canReachMetadata('metadata.google.internal')) {
return {
provider: 'gcp',
instance: await getGCEInstanceType(),
region: process.env.GOOGLE_CLOUD_REGION,
services: {
storage: 'gcs',
cache: 'memorystore',
compute: 'cloud-run'
}
}
}
// Vercel Edge Detection
if (process.env.VERCEL) {
return {
provider: 'vercel',
region: process.env.VERCEL_REGION,
services: {
storage: 'vercel-kv',
cache: 'edge-config',
compute: 'edge-runtime'
}
}
}
}
}
```
## Development vs Production
### Automatic Environment Detection
```typescript
class EnvironmentDetector {
detect() {
const indicators = {
// Development indicators
isDevelopment:
process.env.NODE_ENV === 'development' ||
process.env.DEBUG ||
process.argv.includes('--dev') ||
isLocalhost() ||
hasDevTools(),
// Test indicators
isTest:
process.env.NODE_ENV === 'test' ||
process.env.CI ||
isTestRunner(),
// Production indicators
isProduction:
process.env.NODE_ENV === 'production' ||
process.env.VERCEL ||
process.env.NETLIFY ||
!isLocalhost()
}
return indicators
}
}
// Different defaults for different environments
const config = environment.isProduction ? {
storage: 'filesystem',
wal: true,
monitoring: true,
compression: true,
caching: 'aggressive'
} : {
storage: 'memory',
wal: false,
monitoring: false,
compression: false,
caching: 'minimal'
}
```
## Error Recovery
### Automatic Fallbacks
Brainy automatically recovers from errors:
```typescript
class AutoRecovery {
async handleStorageFailure() {
try {
await this.primaryStorage.write(data)
} catch (error) {
console.warn('Primary storage failed, trying fallback')
// Try secondary storage
if (this.secondaryStorage) {
await this.secondaryStorage.write(data)
} else {
// Fall back to memory
await this.memoryStorage.write(data)
// Schedule retry
this.scheduleRetry(data)
}
}
}
async handleModelFailure() {
try {
return await this.primaryModel.embed(text)
} catch (error) {
// Fall back to simpler model
return await this.fallbackModel.embed(text)
}
}
}
```
## Configuration Override ## Configuration Override
While zero-config is default, you can override when needed: Zero-config is the default, not a ceiling. Every adaptive decision above has an
explicit constructor option:
```typescript ```typescript
// Explicit configuration when needed
const brain = new Brainy({ const brain = new Brainy({
// Override auto-detection storage: { type: 'filesystem', rootDirectory: '/var/lib/brainy' },
storage: { vector: {
type: 'filesystem', recall: 'accurate',
path: '/custom/path' persistMode: 'immediate'
}, },
cache: { maxSize: 50000, ttl: 600_000 }
// Override auto-optimization
optimization: {
autoIndex: false,
autoCache: false,
autoBatch: false
},
// Override auto-scaling
scaling: {
maxMemory: 2 * GB,
maxConnections: 100,
maxBatchSize: 1000
}
})
```
## Monitoring Auto-Adaptation
Brainy provides visibility into its auto-adaptation:
```typescript
brain.on('adaptation', (event) => {
console.log(`Brainy adapted: ${event.type}`)
console.log(`Reason: ${event.reason}`)
console.log(`Before: ${JSON.stringify(event.before)}`)
console.log(`After: ${JSON.stringify(event.after)}`)
}) })
// Example events: await brain.init()
// - Index created for frequently queried field
// - Cache strategy changed due to low hit rate
// - Batch size increased due to high throughput
// - Storage migrated due to space constraints
// - Model switched due to multilingual content
``` ```
## Conclusion See the [API Reference](../api/README.md#configuration) for the complete option
list.
Brainy's zero-configuration and auto-adaptation capabilities mean you can focus on your application logic while Brainy handles:
- Environment detection and optimization
- Storage selection and migration
- Performance tuning and scaling
- Resource management
- Error recovery
- Workload optimization
Just create a Brainy instance and start using it. Brainy will learn, adapt, and optimize itself for your specific use case—no configuration required.
## See Also ## See Also
- [Architecture Overview](./overview.md) - [Architecture Overview](./overview.md)
- [Storage Architecture](./storage.md) - [Storage Adapters](../concepts/storage-adapters.md)
- [Performance Guide](../guides/performance.md) - [Scaling Guide](../SCALING.md)
- [Augmentations System](./augmentations.md) - [API Reference](../api/README.md)

View file

@ -135,11 +135,12 @@ The CLI `brainy inspect` subcommands all do this for you by default
## What's not enforced (yet) ## What's not enforced (yet)
- **Cloud storage backends** (S3, GCS, R2, Azure) do not currently enforce - **Non-filesystem backends** are out of scope in 8.0, which ships only the
multi-process locking. Two processes pointing at the same bucket can both filesystem and memory adapters. A custom `BaseStorage` subclass that is not
succeed at `init()` in writer mode and clobber each other's writes. A filesystem-backed does not enforce multi-process locking by default: two
best-effort warning is logged in writer mode against a non-filesystem processes can both succeed at `init()` in writer mode and clobber each
backend. Lock semantics for cloud backends will land in a future release. other's writes. A best-effort warning is logged in writer mode against a
non-filesystem backend.
- **Long-running readers** do not automatically pick up new Cortex segments - **Long-running readers** do not automatically pick up new Cortex segments
the writer publishes. One-shot inspector calls re-open the store and see the writer publishes. One-shot inspector calls re-open the store and see
fresh segments; a reader that stays open for hours sees its column store fresh segments; a reader that stays open for hours sees its column store

View file

@ -75,7 +75,7 @@ That's the full ceremony for inheriting multi-process safety.
## When NOT to extend FileSystemStorage ## When NOT to extend FileSystemStorage
If your storage is **not filesystem-backed** (S3, GCS, R2, Azure, a custom If your storage is **not filesystem-backed** (a custom
network backend), extend `BaseStorage` directly: network backend), extend `BaseStorage` directly:
```typescript ```typescript

View file

@ -1,414 +0,0 @@
# 🚀 Brainy 2.0 - Complete Feature List
> **The Truth**: Brainy is MORE powerful than previously documented! This is the complete list of ALL implemented features.
## 🧠 Core Intelligence Engine
### Triple Intelligence System ✅
Unified query system that automatically combines:
- **Vector Search**: HNSW-indexed semantic similarity (O(log n) performance)
- **Graph Traversal**: Relationship-based discovery
- **Field Filtering**: Metadata and attribute queries
- **Auto-optimization**: Queries are automatically optimized based on data patterns
```typescript
// All three intelligences work together automatically
const results = await brain.find({
like: 'AI research', // Vector search
where: { year: 2024 }, // Metadata filtering
connected: { to: authorId } // Graph traversal
})
```
### Neural Query Understanding ✅
- **220+ embedded patterns** for query intent detection
- Natural language query processing
- Automatic query type detection
- Query rewriting and optimization
## 🔧 12+ Production Augmentations
```typescript
// Full crash recovery, checkpointing, replay
```
### 2. Entity Registry ✅
```typescript
import { EntityRegistryAugmentation } from 'brainy'
// Bloom filter-based deduplication for streaming data
// Handles millions of entities with minimal memory
```
### 3. Auto-Register Entities ✅
```typescript
import { AutoRegisterEntitiesAugmentation } from 'brainy'
// Automatically extracts and registers entities from text
```
### 4. Intelligent Verb Scoring ✅
```typescript
import { IntelligentVerbScoringAugmentation } from 'brainy'
// Multi-factor relationship strength:
// - Semantic similarity
// - Temporal decay
// - Frequency amplification
// - Context awareness
```
### 5. Batch Processing ✅
```typescript
import { BatchProcessingAugmentation } from 'brainy'
// Adaptive batching with backpressure
// Dynamically adjusts batch size based on load
```
### 6. Connection Pool ✅
```typescript
import { ConnectionPoolAugmentation } from 'brainy'
// Auto-scaling connection management
// Optimized for high-concurrency workloads
```
### 7. Request Deduplicator ✅
```typescript
import { RequestDeduplicatorAugmentation } from 'brainy'
// In-flight request deduplication
// 3x performance boost for concurrent operations
```
### 8. WebSocket Conduit ✅
```typescript
import { WebSocketConduitAugmentation } from 'brainy'
// Real-time bidirectional streaming
// Auto-reconnection and heartbeat
```
### 9. WebRTC Conduit ✅
```typescript
import { WebRTCConduitAugmentation } from 'brainy'
// Peer-to-peer data channels
// Direct browser-to-browser communication
```
### 10. Memory Storage Optimization ✅
```typescript
import { MemoryStorageAugmentation } from 'brainy'
// Memory-specific optimizations
// Circular buffers, compression
```
### 11. Server Search Conduit ✅
```typescript
import { ServerSearchConduitAugmentation } from 'brainy'
// Forwards queries to a remote Brainy server
```
### 12. Neural Import ✅
```typescript
import { NeuralImportAugmentation } from 'brainy'
// AI-powered data understanding
// Automatic entity detection and classification
// Relationship discovery
```
## 🤖 Neural Import Capabilities (FULLY IMPLEMENTED!)
```typescript
const neuralImport = new NeuralImport(brain)
// ALL of these work TODAY:
await neuralImport.neuralImport('data.csv')
await neuralImport.detectEntitiesWithNeuralAnalysis(data)
await neuralImport.detectNounType(entity)
await neuralImport.detectRelationships(entities)
await neuralImport.generateInsights(data)
```
### Features:
- **Auto-detects file format** (CSV, JSON, XML, etc.)
- **Identifies entity types** using AI
- **Discovers relationships** between entities
- **Generates insights** about the data
- **Creates optimal graph structure** automatically
## 🎯 Zero-Config Model Loading Cascade
Brainy automatically loads models with ZERO configuration required:
```typescript
const brain = new Brainy() // That's it!
await brain.init()
// Models load automatically from best available source
```
### Loading Priority:
1. **Local Cache** (./models) - Instant, no network
2. **CDN** (models.soulcraft.com) - Fast, global [Coming Soon]
3. **GitHub Releases** - Reliable backup
4. **HuggingFace** - Ultimate fallback
### Key Features:
- **Automatic fallback** if sources fail
- **Model verification** with checksums
- **Offline support** with bundled models
- **No environment variables needed**
- **Works in all environments** (Node, Browser, Workers)
## 🏢 Operation Modes
Run many reader processes against one shared on-disk store with a single writer.
### Reader Mode ✅
```typescript
const brain = new Brainy({ mode: 'reader' })
// Optimized for read-heavy workloads
// 80% cache ratio, aggressive prefetch
// 1 hour TTL, minimal writes
```
### Writer Mode ✅
```typescript
const brain = new Brainy({ mode: 'writer' })
// Optimized for write-heavy workloads
// Large write buffers, batch writes
// Minimal caching, fast ingestion
```
### Hybrid Mode ✅
```typescript
const brain = new Brainy({ mode: 'hybrid' })
// Balanced for mixed workloads
// Adaptive caching and batching
```
## 💾 Advanced Caching System
### 3-Level Cache Architecture ✅
```typescript
const cacheConfig = {
hotCache: {
size: 1000, // L1 - RAM
ttl: 60000 // 1 minute
},
warmCache: {
size: 10000, // L2 - Fast storage
ttl: 300000 // 5 minutes
},
coldCache: {
size: 100000, // L3 - Persistent
ttl: null // No expiry
}
}
```
### Cache Features:
- **Automatic promotion/demotion** between levels
- **LRU eviction** within each level
- **Compression** for cold cache
- **Statistics tracking** for optimization
## 📊 Comprehensive Statistics
```typescript
const stats = await brain.getStats()
// Returns detailed metrics:
{
nouns: {
count, created, updated, deleted,
size, avgSize
},
verbs: {
count, created, types,
weights: { min, max, avg }
},
vectors: {
dimensions: 384,
indexSize, partitions,
avgSearchTime
},
cache: {
hits, misses, evictions,
hitRate, sizes
},
performance: {
operations, avgTimes,
p95Latency, p99Latency
},
storage: {
used, available,
compression, files
},
throttling: {
delays, rateLimited,
backoffMs, retries
}
}
```
## 🚀 GPU Acceleration Support
```typescript
// Automatic GPU detection
const device = await detectBestDevice()
// Returns: 'cpu' | 'webgpu' | 'cuda'
// WebGPU in browser (when available)
if (device === 'webgpu') {
// Transformer models use WebGPU automatically
}
// CUDA in Node.js (future GPU support)
if (device === 'cuda') {
// Future: GPU acceleration for embeddings
}
```
## 🔄 Adaptive Systems
### Adaptive Backpressure ✅
```typescript
// Automatically adjusts flow based on system load
// Prevents OOM and maintains throughput
```
### Adaptive Socket Manager ✅
```typescript
// Dynamic connection pooling
// Scales connections based on traffic patterns
```
### Cache Auto-Configuration ✅
```typescript
// Sizes cache based on available memory
// Adjusts strategies based on usage patterns
```
### S3 Throttling Protection ✅
```typescript
// Built-in exponential backoff
// Rate limit detection and adaptation
// Automatic retry with jitter
```
## 🛠️ Storage Adapters
All included, auto-selected based on environment:
### FileSystem Storage ✅
- Default for Node.js
- Efficient file-based storage
- Automatic directory management
### Memory Storage ✅
- Ultra-fast in-memory operations
- Perfect for testing and temporary data
- Circular buffer support
### OPFS Storage ✅
- Browser persistent storage
- Survives page refreshes
- Quota management
### S3 Storage ✅
- AWS S3 compatible
- Automatic multipart uploads
- Throttling protection
- Batch operations
## 🎨 Natural Language Processing
### Built-in Patterns (220+)
- Question types (what, why, how, when, where)
- Temporal queries (yesterday, last week, 2024)
- Comparative queries (better than, similar to)
- Aggregations (count, sum, average)
- Filters (only, except, without)
- Relationships (related to, connected with)
### Coverage: 94-98% of typical queries!
## 🔐 Security Features
### Built-in Security ✅
- Automatic input sanitization
- SQL injection prevention
- XSS protection for web contexts
- Rate limiting support
### Encryption Ready ✅
```typescript
import { crypto } from 'brainy/utils'
// AES-256-GCM encryption utilities
// Key derivation functions
// Secure random generation
```
## 🎯 Key Design Principles
### 1. Zero Configuration
```typescript
const brain = new Brainy()
await brain.init()
// Everything else is automatic!
```
### 2. Fixed Dimensions (384)
- **ALWAYS** uses all-MiniLM-L6-v2 model
- **ALWAYS** 384 dimensions
- **NOT** configurable (by design)
- Ensures everything works together
### 3. Progressive Enhancement
- Starts simple, scales automatically
- Adapts to workload patterns
- Optimizes based on usage
### 4. Universal Compatibility
- Works in Node.js 18+
- Works in modern browsers
- Works in Web Workers
- Works in Edge environments
## 📦 What Ships in Core (MIT Licensed)
**EVERYTHING** is included in the core package:
- ✅ All engines (vector, graph, field, neural)
- ✅ All augmentations (12+)
- ✅ All storage adapters
- ✅ Reader / writer / hybrid operation modes
- ✅ Complete statistics
- ✅ GPU support
- ✅ No feature limitations
- ✅ No premium tiers
- ✅ 100% MIT licensed
## 🚀 Quick Start
```typescript
import { Brainy } from 'brainy'
// Zero config required!
const brain = new Brainy()
await brain.init()
// Add data (auto-detects type)
await brain.add('Content here')
// Search with natural language
const results = await brain.find('related content from last week')
// Everything else is automatic!
```
## 📈 Performance Characteristics
- **Vector Search**: O(log n) with HNSW indexing
- **Graph Traversal**: O(k) for k-hop queries
- **Field Filtering**: O(1) with metadata index
- **Memory Usage**: ~100MB base + data
- **Embedding Speed**: ~100ms for batch of 10
- **Query Speed**: <10ms for most queries
## 🎉 Summary
Brainy 2.0 is a **complete**, **production-ready** AI database that requires **ZERO configuration**. Every feature listed here is **implemented and working** today. No configuration, no setup, no complexity - just powerful AI capabilities that work out of the box!

View file

@ -1,242 +0,0 @@
# 🚀 Brainy - Production-Ready Features
> **Status**: All features listed here are IMPLEMENTED and TESTED
## 📊 Performance Metrics
- **Search Latency**: <10ms for 10,000+ items
- **Write Throughput**: 10,000+ ops/sec
- **Memory Efficiency**: <500MB for 10K items
- **Concurrent Operations**: 100+ simultaneous operations
## 🧠 Core Intelligence Features
### Triple Intelligence System ✅
Unified query system combining three types of intelligence:
```typescript
const results = await brain.find({
like: 'AI research', // Vector similarity search
where: { year: 2024 }, // Metadata filtering
connected: { to: authorId } // Graph relationships
})
```
### Intelligent Type Mapping ✅
Prevents semantic degradation by intelligently inferring types:
```typescript
// Automatically infers 'person' from email field
brain.add({ name: "John", email: "john@example.com" }, 'entity')
// → Stored as type: 'person', not generic 'entity'
```
### Neural Query Understanding ✅
- 220+ embedded patterns for intent detection
- Natural language query processing
- Automatic query optimization
- Pattern-based query rewriting
## 🔐 Security & Compliance
### Rate Limiting ✅
Per-operation configurable limits:
```typescript
const rateLimiter = createRateLimitAugmentation({
limits: {
searches: 1000, // per minute
writes: 100,
reads: 5000,
deletes: 50
}
})
```
### Audit Logging ✅
Comprehensive operation tracking:
```typescript
const auditLogger = createAuditLogAugmentation({
logLevel: 'detailed',
retention: 90, // days
includeMetadata: true
})
// Query audit logs
const logs = auditLogger.queryLogs({
operation: 'add',
startTime: Date.now() - 3600000
})
```
## 📦 Storage & Persistence
Full crash recovery and replay:
```typescript
enabled: true,
checkpointInterval: 1000,
maxLogSize: 100 * 1024 * 1024 // 100MB
}))
```
### Multi-Tenancy ✅
Service-based data isolation:
```typescript
// Isolated data per service
await brain.add(data, 'document', { service: 'tenant-1' })
await brain.find('query', { service: 'tenant-1' })
```
### Write-Only Mode ✅
For dedicated write nodes:
```typescript
const brain = new Brainy({
mode: 'write-only',
storage: 's3://bucket/path'
})
```
## 🚀 Performance Features
### Batch Operations ✅
Optimized bulk processing:
```typescript
// Parallel processing with automatic batching
await brain.addMany(items) // <10ms per item
await brain.updateMany(updates)
await brain.removeMany(filters)
```
### Request Deduplication ✅
Automatic duplicate request handling:
```typescript
brain.use(new RequestDeduplicatorAugmentation())
// Identical concurrent requests return same result
```
### Smart Caching ✅
Intelligent search result caching:
```typescript
brain.use(new CacheAugmentation({
maxSize: 10000,
ttl: 300000, // 5 minutes
invalidateOnWrite: true
}))
```
## 🔄 Data Processing
### Entity Registry ✅
Bloom filter-based deduplication:
```typescript
brain.use(new EntityRegistryAugmentation())
// Handles millions of entities with minimal memory
```
### Neural Import ✅
Intelligent data import with type inference:
```typescript
await brain.import({
source: 'data.json',
autoDetectTypes: true,
batchSize: 1000
})
```
### Streaming Pipeline ✅
Real-time data processing:
```typescript
brain.stream()
.pipe(transform)
.pipe(enrich)
.pipe(brain.writer())
```
## 📊 Analytics & Monitoring
### Metrics Collection ✅
Built-in performance metrics:
```typescript
const metrics = brain.getMetrics()
// {
// operations: { add: 1000, find: 5000 },
// performance: { p95: 8, p99: 12 },
// cache: { hits: 4500, misses: 500 }
// }
```
### Health Monitoring ✅
Automatic health checks:
```typescript
const health = brain.getHealth()
// {
// status: 'healthy',
// storage: 'connected',
// memory: { used: 245, limit: 512 }
// }
```
## 🛠️ Developer Experience
### Zero Configuration ✅
Works out of the box:
```typescript
import Brainy from '@soulcraft/brainy'
const brain = new Brainy() // Auto-configures everything
```
### TypeScript First ✅
Full type safety and inference:
```typescript
// Types are automatically inferred
const results = await brain.find<MyType>('query')
```
### Augmentation System ✅
Extensible plugin architecture:
```typescript
class CustomAugmentation extends BaseAugmentation {
execute(operation, params, next) {
// Your custom logic
return next()
}
}
```
## 🔧 Operational Features
### Graceful Shutdown ✅
Clean shutdown with data persistence:
```typescript
process.on('SIGTERM', async () => {
await brain.shutdown() // Saves all pending data
})
```
### Hot Reload ✅
Configuration updates without restart:
```typescript
brain.updateConfig({
cache: { enabled: false }
})
```
### Backup & Restore ✅
Full data backup capabilities:
```typescript
await brain.backup('backup.bin')
await brain.restore('backup.bin')
```
## 📈 Proven at Scale
- **10,000+ items**: Sub-10ms search
- **1M+ operations**: Stable memory usage
- **100+ concurrent users**: No performance degradation
## 🚫 NOT Implemented (Planned)
These features are documented but NOT yet implemented:
- GraphQL API (use REST API instead)
- Kubernetes operators (use Docker)
---
*Last Updated: Latest Version*
*All features listed above are production-ready and tested*

View file

@ -1,15 +1,17 @@
# Framework Integration Guide # Framework Integration Guide
Brainy 3.0 is **framework-first** - designed from the ground up to work seamlessly with modern JavaScript frameworks. This guide shows you how to integrate Brainy into any framework. Brainy is **framework-friendly** - designed to drop into the server side of any modern JavaScript framework. This guide shows you how to integrate Brainy into framework-based apps.
## 🎯 Why Framework-First? > **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun (server-side only). It is not a browser library. In framework apps, run Brainy in API routes, server components, server actions, loaders, or a dedicated backend service - never in client-side bundles.
Traditional AI databases require complex browser polyfills and bundler configurations. Brainy 3.0 trusts your framework to handle this: ## 🎯 Why Server-Side?
Brainy embeds an HNSW vector index, a graph engine, and a filesystem-backed persistence layer. These belong on the server:
- **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'` - **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'`
- **Framework responsibility**: Let Next.js, Vite, Webpack handle Node.js polyfills - **Auto storage detection**: `new Brainy()` auto-selects filesystem persistence on Node
- **Cleaner code**: No browser-specific entry points or conditional imports - **Cleaner code**: No browser polyfills, no conditional client/server imports
- **Better DX**: Same API everywhere - browser, server, edge - **Better DX**: One instance shared across your server routes
## 🚀 Quick Start ## 🚀 Quick Start
@ -24,7 +26,8 @@ npm install @soulcraft/brainy
```javascript ```javascript
import { Brainy } from '@soulcraft/brainy' import { Brainy } from '@soulcraft/brainy'
// Works in any framework! // Run on the server (API route, server component, backend service)
// new Brainy() auto-detects filesystem persistence on Node
const brain = new Brainy() const brain = new Brainy()
await brain.init() await brain.init()
@ -41,52 +44,48 @@ const results = await brain.find("framework integration")
## ⚛️ React Integration ## ⚛️ React Integration
Brainy runs on the server, so a React client component talks to it through an API endpoint (see the Next.js API route below). The hook fetches results; it never instantiates Brainy in the browser.
### Basic Hook Pattern ### Basic Hook Pattern
```jsx ```jsx
import { useState, useEffect, useCallback } from 'react' import { useState, useCallback } from 'react'
import { Brainy } from '@soulcraft/brainy'
function useBrainy() { function useBrainySearch(endpoint = '/api/search') {
const [brain, setBrain] = useState(null) const [results, setResults] = useState([])
const [isReady, setIsReady] = useState(false) const [loading, setLoading] = useState(false)
useEffect(() => { const search = useCallback(async (query) => {
const initBrain = async () => { if (!query) return
const newBrain = new Brainy({ setLoading(true)
storage: { type: 'opfs' } // Browser storage try {
const res = await fetch(endpoint, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query })
}) })
await newBrain.init() const { results } = await res.json()
setBrain(newBrain) setResults(results)
setIsReady(true) } finally {
setLoading(false)
} }
}, [endpoint])
initBrain() return { results, loading, search }
}, [])
return { brain, isReady }
} }
// Usage in component // Usage in component
function SearchComponent() { function SearchComponent() {
const { brain, isReady } = useBrainy() const { results, loading, search } = useBrainySearch()
const [results, setResults] = useState([])
const handleSearch = useCallback(async (query) => {
if (!isReady) return
const searchResults = await brain.find(query)
setResults(searchResults)
}, [brain, isReady])
if (!isReady) return <div>Loading AI...</div>
return ( return (
<div> <div>
<input <input
type="text" type="text"
placeholder="Search..." placeholder="Search..."
onChange={(e) => handleSearch(e.target.value)} onChange={(e) => search(e.target.value)}
/> />
{loading && <div>Searching...</div>}
<div> <div>
{results.map(result => ( {results.map(result => (
<div key={result.id}> <div key={result.id}>
@ -100,48 +99,34 @@ function SearchComponent() {
} }
``` ```
### React Context Pattern ### Shared Server Instance
```jsx On the server, create one Brainy instance and reuse it across requests. This module is imported only by server code (API routes, server components), never by client components:
import React, { createContext, useContext, useEffect, useState } from 'react'
```javascript
// lib/brain.server.js
import { Brainy } from '@soulcraft/brainy' import { Brainy } from '@soulcraft/brainy'
const BrainyContext = createContext() let brainPromise
export function BrainyProvider({ children }) { export function getBrain() {
const [brain, setBrain] = useState(null) if (!brainPromise) {
const [isReady, setIsReady] = useState(false) brainPromise = (async () => {
// new Brainy() auto-detects filesystem persistence on Node
useEffect(() => { const brain = new Brainy()
const initBrain = async () => { await brain.init()
const newBrain = new Brainy() return brain
await newBrain.init() })()
setBrain(newBrain)
setIsReady(true)
}
initBrain()
}, [])
return (
<BrainyContext.Provider value={{ brain, isReady }}>
{children}
</BrainyContext.Provider>
)
}
export function useBrainContext() {
const context = useContext(BrainyContext)
if (!context) {
throw new Error('useBrainContext must be used within BrainyProvider')
} }
return context return brainPromise
} }
``` ```
## 🟢 Vue.js Integration ## 🟢 Vue.js Integration
### Composition API Vue components call a server endpoint (see the Nuxt server route in the [Vue.js Integration Guide](vue-integration.md)); Brainy itself runs on the server.
### Composition API (client component)
```vue ```vue
<template> <template>
@ -155,100 +140,70 @@ export function useBrainContext() {
</template> </template>
<script setup> <script setup>
import { ref, onMounted } from 'vue' import { ref } from 'vue'
import { Brainy } from '@soulcraft/brainy'
const brain = ref(null)
const isReady = ref(false)
const query = ref('') const query = ref('')
const results = ref([]) const results = ref([])
onMounted(async () => {
brain.value = new Brainy({
storage: { type: 'opfs' }
})
await brain.value.init()
isReady.value = true
})
const search = async () => { const search = async () => {
if (!isReady.value || !query.value) return if (!query.value) return
results.value = await brain.value.find(query.value) const res = await fetch('/api/search', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: query.value })
})
results.value = (await res.json()).results
} }
</script> </script>
``` ```
### Vue 3 Plugin ### Shared Server Instance
On the server, create one Brainy instance and reuse it across requests:
```javascript ```javascript
// plugins/brainy.js // server/brain.js (server-only module)
import { Brainy } from '@soulcraft/brainy' import { Brainy } from '@soulcraft/brainy'
export default { let brainPromise
install(app, options) {
const brain = new Brainy(options)
app.config.globalProperties.$brain = brain export function getBrain() {
app.provide('brain', brain) if (!brainPromise) {
brainPromise = (async () => {
// Initialize on app mount // new Brainy() auto-detects filesystem persistence on Node
brain.init() const brain = new Brainy()
await brain.init()
return brain
})()
} }
return brainPromise
} }
// main.js
import { createApp } from 'vue'
import BrainyPlugin from './plugins/brainy'
const app = createApp(App)
app.use(BrainyPlugin, {
storage: { type: 'opfs' }
})
``` ```
## 🅰️ Angular Integration ## 🅰️ Angular Integration
### Service Pattern The Angular service calls your backend over HTTP; Brainy lives in that backend, not in the browser.
### Service Pattern (calls the backend)
```typescript ```typescript
// brainy.service.ts // brainy.service.ts
import { Injectable } from '@angular/core' import { Injectable } from '@angular/core'
import { BehaviorSubject, Observable } from 'rxjs' import { HttpClient } from '@angular/common/http'
import { Brainy } from '@soulcraft/brainy' import { Observable } from 'rxjs'
@Injectable({ @Injectable({
providedIn: 'root' providedIn: 'root'
}) })
export class BrainyService { export class BrainyService {
private brain: Brainy constructor(private http: HttpClient) {}
private readySubject = new BehaviorSubject<boolean>(false)
ready$: Observable<boolean> = this.readySubject.asObservable() search(query: string): Observable<{ results: any[] }> {
return this.http.post<{ results: any[] }>('/api/search', { query })
constructor() {
this.initBrain()
} }
private async initBrain() { add(data: any, type: string, metadata?: any): Observable<{ id: string }> {
this.brain = new Brainy({ return this.http.post<{ id: string }>('/api/add', { data, type, metadata })
storage: { type: 'opfs' }
})
await this.brain.init()
this.readySubject.next(true)
}
async search(query: string): Promise<any[]> {
if (!this.readySubject.value) {
throw new Error('Brain not ready')
}
return await this.brain.find(query)
}
async add(data: any, type: string, metadata?: any): Promise<string> {
if (!this.readySubject.value) {
throw new Error('Brain not ready')
}
return await this.brain.add({ data, type, metadata })
} }
} }
``` ```
@ -280,64 +235,52 @@ export class SearchComponent {
constructor(private brainyService: BrainyService) {} constructor(private brainyService: BrainyService) {}
async search() { search() {
if (!this.query) return if (!this.query) return
this.results = await this.brainyService.search(this.query) this.brainyService.search(this.query).subscribe(({ results }) => {
this.results = results
})
} }
} }
``` ```
The matching backend endpoint uses Brainy directly (Node/Bun):
```typescript
// server: api/search
import { Brainy } from '@soulcraft/brainy'
const brain = new Brainy() // auto-detects filesystem persistence on Node
await brain.init()
export async function handleSearch(query: string) {
return await brain.find(query)
}
```
## 🚀 Next.js Integration ## 🚀 Next.js Integration
### App Router (Next.js 13+) In Next.js, Brainy lives in server code only: API routes, server components, or server actions. Create one shared instance in a server-only module.
```jsx ### Shared Server Instance
// app/providers.jsx
'use client' ```javascript
import { createContext, useContext, useEffect, useState } from 'react' // lib/brain.server.js (imported only by server code)
import { Brainy } from '@soulcraft/brainy' import { Brainy } from '@soulcraft/brainy'
const BrainyContext = createContext() let brainPromise
export function BrainyProvider({ children }) { export function getBrain() {
const [brain, setBrain] = useState(null) if (!brainPromise) {
const [isReady, setIsReady] = useState(false) brainPromise = (async () => {
const brain = new Brainy({
useEffect(() => { storage: { type: 'filesystem', rootDirectory: './data' }
const initBrain = async () => { })
const newBrain = new Brainy() await brain.init()
await newBrain.init() return brain
setBrain(newBrain) })()
setIsReady(true) }
} return brainPromise
initBrain()
}, [])
return (
<BrainyContext.Provider value={{ brain, isReady }}>
{children}
</BrainyContext.Provider>
)
}
export const useBrainy = () => useContext(BrainyContext)
```
```jsx
// app/layout.jsx
import { BrainyProvider } from './providers'
export default function RootLayout({ children }) {
return (
<html>
<body>
<BrainyProvider>
{children}
</BrainyProvider>
</body>
</html>
)
} }
``` ```
@ -345,45 +288,78 @@ export default function RootLayout({ children }) {
```javascript ```javascript
// app/api/search/route.js // app/api/search/route.js
import { Brainy } from '@soulcraft/brainy' import { getBrain } from '@/lib/brain.server'
const brain = new Brainy({
storage: { type: 'filesystem', rootDirectory: './data' }
})
await brain.init()
export async function POST(request) { export async function POST(request) {
const { query } = await request.json() const { query } = await request.json()
const brain = await getBrain()
const results = await brain.find(query) const results = await brain.find(query)
return Response.json({ results }) return Response.json({ results })
} }
``` ```
## 🔷 Svelte Integration ### Server Action
```javascript
// app/actions.js
'use server'
import { getBrain } from '@/lib/brain.server'
export async function search(query) {
const brain = await getBrain()
return await brain.find(query)
}
```
## 🔷 SvelteKit Integration
Brainy runs in a server-only module (`*.server.js`); the component fetches results from an endpoint.
```javascript
// src/lib/server/brain.js (server-only — note the .server suffix)
import { Brainy } from '@soulcraft/brainy'
let brainPromise
export function getBrain() {
if (!brainPromise) {
brainPromise = (async () => {
const brain = new Brainy() // auto-detects filesystem persistence
await brain.init()
return brain
})()
}
return brainPromise
}
```
```javascript
// src/routes/api/search/+server.js
import { json } from '@sveltejs/kit'
import { getBrain } from '$lib/server/brain'
export async function POST({ request }) {
const { query } = await request.json()
const brain = await getBrain()
return json({ results: await brain.find(query) })
}
```
```svelte ```svelte
<!-- SearchComponent.svelte --> <!-- SearchComponent.svelte -->
<script> <script>
import { onMount } from 'svelte'
import { Brainy } from '@soulcraft/brainy'
let brain = null
let isReady = false
let query = '' let query = ''
let results = [] let results = []
onMount(async () => {
brain = new Brainy({
storage: { type: 'opfs' }
})
await brain.init()
isReady = true
})
async function search() { async function search() {
if (!isReady || !query) return if (!query) return
results = await brain.find(query) const res = await fetch('/api/search', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query })
})
results = (await res.json()).results
} }
</script> </script>
@ -401,27 +377,23 @@ export async function POST(request) {
## 🌟 Solid.js Integration ## 🌟 Solid.js Integration
The component calls a server endpoint (use SolidStart server routes, or any backend, to host Brainy):
```jsx ```jsx
import { createSignal, onMount } from 'solid-js' import { createSignal } from 'solid-js'
import { Brainy } from '@soulcraft/brainy'
function SearchComponent() { function SearchComponent() {
const [brain, setBrain] = createSignal(null)
const [isReady, setIsReady] = createSignal(false)
const [query, setQuery] = createSignal('') const [query, setQuery] = createSignal('')
const [results, setResults] = createSignal([]) const [results, setResults] = createSignal([])
onMount(async () => {
const newBrain = new Brainy()
await newBrain.init()
setBrain(newBrain)
setIsReady(true)
})
const search = async () => { const search = async () => {
if (!isReady() || !query()) return if (!query()) return
const searchResults = await brain().find(query()) const res = await fetch('/api/search', {
setResults(searchResults) method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: query() })
})
setResults((await res.json()).results)
} }
return ( return (
@ -450,57 +422,25 @@ function SearchComponent() {
## 📦 Bundler Configuration ## 📦 Bundler Configuration
### Vite (Recommended) Brainy is a server-side dependency, so keep it out of client bundles. Import it only from server-only modules (`*.server.js`, API routes, server components, server actions). If your bundler ever tries to pull Brainy into a client bundle, that's a sign it's being imported from a client component — move the import to a server module.
For server builds, mark Brainy as external so the bundler doesn't inline it:
```javascript ```javascript
// vite.config.js // vite.config.js (SSR build)
import { defineConfig } from 'vite' import { defineConfig } from 'vite'
export default defineConfig({ export default defineConfig({
define: { ssr: {
global: 'globalThis' external: ['@soulcraft/brainy']
},
optimizeDeps: {
include: ['@soulcraft/brainy']
} }
}) })
``` ```
### Webpack
```javascript ```javascript
// webpack.config.js // rollup.config.js (server bundle)
module.exports = {
resolve: {
fallback: {
"fs": false,
"path": require.resolve("path-browserify"),
"crypto": require.resolve("crypto-browserify")
}
},
plugins: [
new webpack.ProvidePlugin({
global: 'global'
})
]
}
```
### Rollup
```javascript
// rollup.config.js
import { nodeResolve } from '@rollup/plugin-node-resolve'
import commonjs from '@rollup/plugin-commonjs'
export default { export default {
plugins: [ external: ['@soulcraft/brainy', 'node:fs', 'node:path', 'node:crypto']
nodeResolve({
browser: true,
preferBuiltins: false
}),
commonjs()
]
} }
``` ```
@ -508,30 +448,24 @@ export default {
### Server-Side Rendering ### Server-Side Rendering
Instantiate Brainy on the server and feed its results into the rendered page. Never construct it in client code.
```javascript ```javascript
// Check if running in browser // Server-side data loading (framework loader / getServerSideProps / load fn)
if (typeof window !== 'undefined') { import { getBrain } from './brain.server'
// Browser-only code
const brain = new Brainy({
storage: { type: 'opfs' }
})
}
// Or use dynamic imports export async function load({ url }) {
const initBrainForBrowser = async () => { const brain = await getBrain()
if (typeof window === 'undefined') return null const query = url.searchParams.get('q') ?? ''
const results = query ? await brain.find(query) : []
const { Brainy } = await import('@soulcraft/brainy') return { results }
const brain = new Brainy()
await brain.init()
return brain
} }
``` ```
### Static Site Generation ### Static Site Generation
```javascript ```javascript
// For build-time usage // For build-time usage (runs in Node during the build)
import { Brainy } from '@soulcraft/brainy' import { Brainy } from '@soulcraft/brainy'
export async function generateStaticProps() { export async function generateStaticProps() {
@ -552,67 +486,46 @@ export async function generateStaticProps() {
## 🔧 Framework-Specific Tips ## 🔧 Framework-Specific Tips
### React ### React
- Use `useCallback` for search functions to prevent re-renders - Keep components client-side and call a Brainy-backed API route
- Consider `useMemo` for expensive brain operations - Use `useCallback` for fetch handlers to prevent re-renders
- Implement cleanup in `useEffect` for proper memory management - Debounce keystroke-driven searches before hitting the endpoint
### Vue ### Vue
- Use `shallowRef` for the brain instance (it's not reactive data) - Components call an endpoint; the shared instance lives in a server module
- Consider Pinia for global brain state management - Consider Pinia for caching results client-side
- Use `watchEffect` for reactive search queries - Debounce reactive search queries
### Angular ### Angular
- Implement proper dependency injection with services - Use `HttpClient` and RxJS to call the backend
- Use RxJS observables for reactive search - Hold the shared Brainy instance in your Node backend, not the app
- Consider lazy loading brain in feature modules - Consider lazy loading search features in feature modules
### Next.js ### Next.js
- Use dynamic imports for client-side only features - Put Brainy in server-only modules (`*.server.js`), API routes, or server actions
- Consider API routes for server-side brain operations - Reuse one shared instance across requests
- Implement proper error boundaries - Implement proper error boundaries for failed fetches
## 🚨 Common Issues & Solutions ## 🚨 Common Issues & Solutions
### Issue: "crypto is not defined" ### Issue: "fs module not found" / "crypto is not defined" in the browser
**Solution**: Your framework should handle this automatically. If not: **Cause**: Brainy was imported into a client bundle. Brainy 8.0 is a server-side library (Node 22+/Bun) and uses Node built-ins like `fs` and `crypto`.
```javascript **Solution**: Import Brainy only from server code — server-only modules (`*.server.js`), API routes, server components, or server actions. From client components, call those endpoints instead.
// Add to your bundle config
define: {
global: 'globalThis'
}
```
### Issue: "fs module not found" ### Issue: Large client bundle size
**Solution**: This is expected in browsers. Use browser-compatible storage: **Cause**: A client module is pulling in Brainy.
```javascript **Solution**: Move the `import { Brainy } from '@soulcraft/brainy'` into a server-only module so it never reaches the browser bundle.
const brain = new Brainy({
storage: { type: 'opfs' } // Or 'memory' for development
})
```
### Issue: Large bundle size
**Solution**: Use dynamic imports for optional features:
```javascript
const brain = await import('@soulcraft/brainy').then(m => new m.Brainy())
```
### Issue: SSR hydration mismatch ### Issue: SSR hydration mismatch
**Solution**: Initialize brain only on client: **Solution**: Run the search on the server (loader / server action / API route) and pass the results down as props, so server and client render the same markup.
```javascript
useEffect(() => {
// Browser-only initialization
initBrain()
}, [])
```
## 🎯 Best Practices ## 🎯 Best Practices
1. **Initialize Once**: Create brain instance at app level, not component level 1. **Initialize Once**: Create one shared Brainy instance per server process, not per request
2. **Use Context**: Share brain instance across components with context/providers 2. **Server-Only**: Import Brainy only from server modules — never from client components
3. **Handle Loading**: Always show loading states during brain initialization 3. **Endpoint Boundary**: Expose search/add through API routes or server actions
4. **Error Boundaries**: Implement proper error handling for brain operations 4. **Handle Loading**: Show loading states in the client while the fetch is in flight
5. **Memory Management**: Clean up brain instances on unmount 5. **Error Handling**: Catch and surface failed endpoint calls gracefully
6. **Storage Strategy**: Choose appropriate storage for your deployment target 6. **Storage**: Use `filesystem` for persistence (the default on Node) or `memory` for ephemeral/tests
## 📚 Next Steps ## 📚 Next Steps

View file

@ -1743,7 +1743,7 @@ For off-site backup, snapshot `rootDirectory` from your scheduler with `gsutil r
- HNSW Index: O(log n) search (1B entities = ~30 hops) - HNSW Index: O(log n) search (1B entities = ~30 hops)
- Metadata Index: O(1) filtering - Metadata Index: O(1) filtering
- Graph Adjacency: O(1) relationship lookups - Graph Adjacency: O(1) relationship lookups
- Storage: Unlimited (cloud buckets) - Storage: Bounded by the filesystem volume backing `rootDirectory`
### Optimization Tips ### Optimization Tips
@ -1887,7 +1887,7 @@ groupBy: 'type'
- ✅ O(1) metadata filtering - ✅ O(1) metadata filtering
- ✅ O(1) relationship traversal - ✅ O(1) relationship traversal
- ✅ Human-readable VFS structure - ✅ Human-readable VFS structure
- ✅ Cloud storage support (GCS/S3/R2) - ✅ Filesystem-backed persistence (snapshot/sync the directory for off-site backup)
- ✅ Billion-scale performance - ✅ Billion-scale performance
- ✅ Zero mocks, production-ready! - ✅ Zero mocks, production-ready!

View file

@ -45,7 +45,7 @@ console.log('Brainy ready.')
## Native Acceleration (Optional) ## Native Acceleration (Optional)
For production workloads, add Cortex for Rust-accelerated SIMD distance calculations, vector quantization, and native embeddings: For production workloads, add Cortex for Rust-accelerated SIMD distance calculations and native embeddings:
```bash ```bash
npm install @soulcraft/cortex npm install @soulcraft/cortex

View file

@ -108,4 +108,4 @@ await brain.find({ query: 'people who work at Anthropic' })
- [Triple Intelligence](/docs/concepts/triple-intelligence) — understand how the query engine works - [Triple Intelligence](/docs/concepts/triple-intelligence) — understand how the query engine works
- [The Find System](/docs/guides/find-system) — advanced queries, operators, and graph traversal - [The Find System](/docs/guides/find-system) — advanced queries, operators, and graph traversal
- [API Reference](/docs/api/reference) — complete method documentation - [API Reference](/docs/api/reference) — complete method documentation
- [Storage Adapters](/docs/guides/storage-adapters) — S3, GCS, Azure, filesystem, OPFS - [Storage Adapters](/docs/guides/storage-adapters) — filesystem, memory

View file

@ -271,7 +271,7 @@ Progress is reported after each batch (batch size is determined by the storage a
## Storage Backend Compatibility ## Storage Backend Compatibility
Migrations work identically across all storage backends (Memory, FileSystem, S3, R2, GCS, OPFS). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter. Migrations work identically across all storage backends (Memory, FileSystem). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter.
Batch size and rate limiting are automatically configured per adapter — no tuning required. Batch size and rate limiting are automatically configured per adapter — no tuning required.

View file

@ -2,6 +2,8 @@
Complete guide to integrating Brainy with Vue.js applications, covering Vue 3, Nuxt.js, Composition API, Options API, and advanced patterns. Complete guide to integrating Brainy with Vue.js applications, covering Vue 3, Nuxt.js, Composition API, Options API, and advanced patterns.
> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun, server-side only — it is not a browser library. In Vue apps, Brainy lives behind an HTTP endpoint (a Nuxt server route, or any Node/Bun backend), and your components call that endpoint. The composables, plugin, and store below are wrappers around `fetch`; the single Brainy instance is created once on the server (see [Nuxt Server Route](#nuxt-server-route)).
## 🚀 Quick Start ## 🚀 Quick Start
### Installation ### Installation
@ -15,34 +17,47 @@ npm install @soulcraft/brainy
### Basic Setup ### Basic Setup
The composable talks to a Brainy-backed endpoint (see [Nuxt Server Route](#nuxt-server-route)). It is always "ready" because there is no client-side initialization — the server owns the Brainy instance.
```javascript ```javascript
// src/composables/useBrainy.js // src/composables/useBrainy.js
import { ref, onMounted } from 'vue' import { ref } from 'vue'
import { Brainy } from '@soulcraft/brainy'
const brain = ref(null) export function useBrainy(endpoint = '/api/brain') {
const isReady = ref(false) const error = ref(null)
const error = ref(null)
export function useBrainy() { const search = async (query, options = {}) => {
onMounted(async () => { error.value = null
try { try {
brain.value = new Brainy({ const res = await fetch(`${endpoint}/search`, {
storage: { type: 'opfs' } // Browser storage method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query, options })
}) })
await brain.value.init() if (!res.ok) throw new Error(`Search failed: ${res.status}`)
isReady.value = true return (await res.json()).results
} catch (err) { } catch (err) {
error.value = err.message error.value = err.message
console.error('Brainy initialization failed:', err) console.error('Brainy search failed:', err)
return []
} }
})
return {
brain: readonly(brain),
isReady: readonly(isReady),
error: readonly(error)
} }
const add = async (data, type, metadata) => {
const res = await fetch(`${endpoint}/add`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ data, type, metadata })
})
return (await res.json()).id
}
const stats = async () => {
const res = await fetch(`${endpoint}/stats`)
return await res.json()
}
return { search, add, stats, error: readonly(error) }
} }
``` ```
@ -54,43 +69,36 @@ export function useBrainy() {
<!-- src/components/Search.vue --> <!-- src/components/Search.vue -->
<template> <template>
<div class="search-container"> <div class="search-container">
<div v-if="!isReady" class="loading"> <div class="search-input">
<div class="spinner"></div> <input
<span>Initializing AI...</span> v-model="query"
@input="handleSearch"
placeholder="Search..."
class="input"
/>
</div> </div>
<div v-else> <div v-if="loading" class="loading">
<div class="search-input"> Searching...
<input </div>
v-model="query"
@input="handleSearch" <div v-else class="results">
placeholder="Search with AI..." <div
class="input" v-for="result in results"
/> :key="result.id"
class="result-item"
>
<h3>{{ result.data }}</h3>
<div class="result-meta">
<span>Score: {{ (result.score * 100).toFixed(1) }}%</span>
<span v-if="result.metadata?.type">
Type: {{ result.metadata.type }}
</span>
</div>
</div> </div>
<div v-if="loading" class="loading"> <div v-if="query && !loading && results.length === 0" class="no-results">
Searching... No results found for "{{ query }}"
</div>
<div v-else class="results">
<div
v-for="result in results"
:key="result.id"
class="result-item"
>
<h3>{{ result.data }}</h3>
<div class="result-meta">
<span>Score: {{ (result.score * 100).toFixed(1) }}%</span>
<span v-if="result.metadata?.type">
Type: {{ result.metadata.type }}
</span>
</div>
</div>
<div v-if="query && !loading && results.length === 0" class="no-results">
No results found for "{{ query }}"
</div>
</div> </div>
</div> </div>
</div> </div>
@ -101,22 +109,21 @@ import { ref, watch } from 'vue'
import { useBrainy } from '../composables/useBrainy' import { useBrainy } from '../composables/useBrainy'
import { debounce } from '../utils/debounce' import { debounce } from '../utils/debounce'
const { brain, isReady } = useBrainy() const { search: searchBrain } = useBrainy()
const query = ref('') const query = ref('')
const results = ref([]) const results = ref([])
const loading = ref(false) const loading = ref(false)
const search = async (searchQuery) => { const search = async (searchQuery) => {
if (!isReady.value || !searchQuery.trim()) { if (!searchQuery.trim()) {
results.value = [] results.value = []
return return
} }
loading.value = true loading.value = true
try { try {
const searchResults = await brain.value.find(searchQuery) results.value = await searchBrain(searchQuery)
results.value = searchResults
} catch (error) { } catch (error) {
console.error('Search error:', error) console.error('Search error:', error)
results.value = [] results.value = []
@ -228,11 +235,7 @@ watch(query, (newQuery) => {
<div class="data-manager"> <div class="data-manager">
<h2>Data Management</h2> <h2>Data Management</h2>
<div v-if="!isReady" class="loading"> <div>
Initializing...
</div>
<div v-else>
<!-- Add Data Form --> <!-- Add Data Form -->
<form @submit.prevent="addData" class="add-form"> <form @submit.prevent="addData" class="add-form">
<h3>Add New Data</h3> <h3>Add New Data</h3>
@ -289,7 +292,7 @@ watch(query, (newQuery) => {
import { ref, reactive, onMounted } from 'vue' import { ref, reactive, onMounted } from 'vue'
import { useBrainy } from '../composables/useBrainy' import { useBrainy } from '../composables/useBrainy'
const { brain, isReady } = useBrainy() const { add, stats: fetchStats } = useBrainy()
const newItem = reactive({ const newItem = reactive({
data: '', data: '',
@ -299,18 +302,7 @@ const newItem = reactive({
const stats = ref(null) const stats = ref(null)
onMounted(async () => { onMounted(loadStats)
if (isReady.value) {
await loadStats()
}
})
// Watch for brain readiness
watch(isReady, async (ready) => {
if (ready) {
await loadStats()
}
})
const addData = async () => { const addData = async () => {
try { try {
@ -319,11 +311,7 @@ const addData = async () => {
tags: newItem.tags.split(',').map(tag => tag.trim()).filter(Boolean) tags: newItem.tags.split(',').map(tag => tag.trim()).filter(Boolean)
} }
await brain.value.add({ await add(newItem.data, newItem.type, metadata)
data: newItem.data,
type: newItem.type,
metadata
})
// Reset form // Reset form
newItem.data = '' newItem.data = ''
@ -339,7 +327,7 @@ const addData = async () => {
const loadStats = async () => { const loadStats = async () => {
try { try {
stats.value = await brain.value.stats() stats.value = await fetchStats()
} catch (error) { } catch (error) {
console.error('Failed to load stats:', error) console.error('Failed to load stats:', error)
} }
@ -435,73 +423,55 @@ button:disabled {
<!-- src/components/SearchOptions.vue --> <!-- src/components/SearchOptions.vue -->
<template> <template>
<div class="search-container"> <div class="search-container">
<div v-if="!isReady" class="loading"> <input
Initializing AI... v-model="query"
</div> @input="handleSearch"
placeholder="Search..."
class="search-input"
/>
<div v-else> <div v-if="loading" class="loading">Searching...</div>
<input
v-model="query"
@input="handleSearch"
placeholder="Search..."
class="search-input"
/>
<div v-if="loading" class="loading">Searching...</div> <div class="results">
<div
<div class="results"> v-for="result in results"
<div :key="result.id"
v-for="result in results" class="result-item"
:key="result.id" >
class="result-item" <h3>{{ result.data }}</h3>
> <p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
<h3>{{ result.data }}</h3>
<p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
</div>
</div> </div>
</div> </div>
</div> </div>
</template> </template>
<script> <script>
import { Brainy } from '@soulcraft/brainy'
import { debounce } from '../utils/debounce' import { debounce } from '../utils/debounce'
export default { export default {
name: 'SearchOptions', name: 'SearchOptions',
data() { data() {
return { return {
brain: null,
isReady: false,
query: '', query: '',
results: [], results: [],
loading: false loading: false
} }
}, },
async mounted() {
await this.initBrain()
},
methods: { methods: {
async initBrain() {
try {
this.brain = new Brainy({
storage: { type: 'opfs' }
})
await this.brain.init()
this.isReady = true
} catch (error) {
console.error('Brain initialization failed:', error)
}
},
async search(query) { async search(query) {
if (!this.isReady || !query.trim()) { if (!query.trim()) {
this.results = [] this.results = []
return return
} }
this.loading = true this.loading = true
try { try {
this.results = await this.brain.find(query) const res = await fetch('/api/brain/search', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query })
})
this.results = (await res.json()).results
} catch (error) { } catch (error) {
console.error('Search error:', error) console.error('Search error:', error)
this.results = [] this.results = []
@ -520,10 +490,6 @@ export default {
this.loading = false this.loading = false
} }
} }
},
beforeUnmount() {
// Cleanup if needed
this.brain = null
} }
} }
</script> </script>
@ -533,31 +499,22 @@ export default {
### Global Brainy Plugin ### Global Brainy Plugin
The plugin exposes a global `$searchBrain` helper that calls your Brainy-backed endpoint. Brainy itself runs on the server (see [Nuxt Server Route](#nuxt-server-route)).
```javascript ```javascript
// src/plugins/brainy.js // src/plugins/brainy.js
import { Brainy } from '@soulcraft/brainy'
export default { export default {
install(app, options = {}) { install(app, options = {}) {
const defaultOptions = { const endpoint = options.endpoint ?? '/api/brain'
storage: { type: 'opfs' },
...options
}
const brain = new Brainy(defaultOptions) // Add global search method (calls the server endpoint)
app.config.globalProperties.$searchBrain = async (query, searchOptions) => {
// Make brain available globally const res = await fetch(`${endpoint}/search`, {
app.config.globalProperties.$brain = brain method: 'POST',
app.provide('brain', brain) headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query, options: searchOptions })
// Auto-initialize })
brain.init().catch(error => { return (await res.json()).results
console.error('Brainy plugin initialization failed:', error)
})
// Add global method
app.config.globalProperties.$searchBrain = async (query, options) => {
return await brain.find(query, options)
} }
} }
} }
@ -574,7 +531,7 @@ import BrainyPlugin from './plugins/brainy'
const app = createApp(App) const app = createApp(App)
app.use(BrainyPlugin, { app.use(BrainyPlugin, {
storage: { type: 'opfs' } endpoint: '/api/brain'
}) })
app.mount('#app') app.mount('#app')
@ -611,24 +568,58 @@ export default {
## 🏰 Nuxt.js Integration ## 🏰 Nuxt.js Integration
### Nuxt Plugin Nuxt's server engine (Nitro) is the natural home for Brainy: it runs on Node/Bun, so the Brainy instance lives in `server/`, and pages/components call its routes.
### Nuxt Server Route
```javascript ```javascript
// plugins/brainy.client.js // server/utils/brain.js (server-only — Nitro never bundles this into the client)
import { Brainy } from '@soulcraft/brainy' import { Brainy } from '@soulcraft/brainy'
export default defineNuxtPlugin(async () => { let brainPromise
const brain = new Brainy({
storage: { type: 'opfs' }
})
await brain.init() export function getBrain() {
if (!brainPromise) {
return { brainPromise = (async () => {
provide: { // new Brainy() auto-detects filesystem persistence on Node
brain const brain = new Brainy()
} await brain.init()
return brain
})()
} }
return brainPromise
}
```
```javascript
// server/api/brain/search.post.js
import { getBrain } from '../../utils/brain'
export default defineEventHandler(async (event) => {
const { query, options } = await readBody(event)
const brain = await getBrain()
return { results: await brain.find(query, options) }
})
```
```javascript
// server/api/brain/add.post.js
import { getBrain } from '../../utils/brain'
export default defineEventHandler(async (event) => {
const { data, type, metadata } = await readBody(event)
const brain = await getBrain()
return { id: await brain.add({ data, type, metadata }) }
})
```
```javascript
// server/api/brain/stats.get.js
import { getBrain } from '../../utils/brain'
export default defineEventHandler(async () => {
const brain = await getBrain()
return await brain.stats()
}) })
``` ```
@ -637,26 +628,25 @@ export default defineNuxtPlugin(async () => {
```javascript ```javascript
// composables/useBrainy.js // composables/useBrainy.js
export const useBrainy = () => { export const useBrainy = () => {
const { $brain } = useNuxtApp()
const search = async (query, options = {}) => { const search = async (query, options = {}) => {
return await $brain.find(query, options) return (await $fetch('/api/brain/search', {
method: 'POST',
body: { query, options }
})).results
} }
const add = async (data, type, metadata) => { const add = async (data, type, metadata) => {
return await $brain.add({ data, type, metadata }) return (await $fetch('/api/brain/add', {
method: 'POST',
body: { data, type, metadata }
})).id
} }
const stats = async () => { const stats = async () => {
return await $brain.stats() return await $fetch('/api/brain/stats')
} }
return { return { search, add, stats }
brain: $brain,
search,
add,
stats
}
} }
``` ```
@ -666,26 +656,20 @@ export const useBrainy = () => {
<!-- pages/search.vue --> <!-- pages/search.vue -->
<template> <template>
<div> <div>
<h1>AI Search</h1> <h1>Search</h1>
<div v-if="pending" class="loading"> <input
Initializing AI... v-model="query"
</div> @input="handleSearch"
placeholder="Search..."
/>
<div v-else> <div v-if="searching" class="loading">Searching...</div>
<input
v-model="query"
@input="handleSearch"
placeholder="Search..."
/>
<div v-if="searching" class="loading">Searching...</div> <div class="results">
<div v-for="result in results" :key="result.id" class="result">
<div class="results"> <h3>{{ result.data }}</h3>
<div v-for="result in results" :key="result.id" class="result"> <p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
<h3>{{ result.data }}</h3>
<p>Score: {{ (result.score * 100).toFixed(1) }}%</p>
</div>
</div> </div>
</div> </div>
</div> </div>
@ -697,12 +681,6 @@ const { search } = useBrainy()
const query = ref('') const query = ref('')
const results = ref([]) const results = ref([])
const searching = ref(false) const searching = ref(false)
const pending = ref(true)
// Initialize
onMounted(() => {
pending.value = false
})
const handleSearch = debounce(async () => { const handleSearch = debounce(async () => {
if (!query.value.trim()) { if (!query.value.trim()) {
@ -722,82 +700,58 @@ const handleSearch = debounce(async () => {
</script> </script>
``` ```
### Nuxt Configuration
```javascript
// nuxt.config.ts
export default defineNuxtConfig({
ssr: false, // Disable SSR for browser-only features
// Or use client-side hydration
nitro: {
experimental: {
wasm: true
}
},
vite: {
define: {
global: 'globalThis'
}
}
})
```
## 🛠️ Advanced Patterns ## 🛠️ Advanced Patterns
### Global State Management with Pinia ### Global State Management with Pinia
The store caches results and stats client-side; all Brainy work happens behind the server endpoints.
```javascript ```javascript
// stores/brainy.js // stores/brainy.js
import { defineStore } from 'pinia' import { defineStore } from 'pinia'
import { Brainy } from '@soulcraft/brainy'
export const useBrainyStore = defineStore('brainy', () => { export const useBrainyStore = defineStore('brainy', () => {
const brain = ref(null)
const isReady = ref(false)
const error = ref(null) const error = ref(null)
const stats = ref(null) const stats = ref(null)
const init = async (options = {}) => { const search = async (query, options = {}) => {
error.value = null
try { try {
brain.value = new Brainy(options) const res = await fetch('/api/brain/search', {
await brain.value.init() method: 'POST',
isReady.value = true headers: { 'Content-Type': 'application/json' },
await loadStats() body: JSON.stringify({ query, options })
})
return (await res.json()).results
} catch (err) { } catch (err) {
error.value = err.message error.value = err.message
console.error('Brainy initialization failed:', err) throw err
} }
} }
const search = async (query, options = {}) => {
if (!isReady.value) throw new Error('Brain not ready')
return await brain.value.find(query, options)
}
const add = async (data, type, metadata) => { const add = async (data, type, metadata) => {
if (!isReady.value) throw new Error('Brain not ready') const res = await fetch('/api/brain/add', {
const id = await brain.value.add({ data, type, metadata }) method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ data, type, metadata })
})
const { id } = await res.json()
await loadStats() // Refresh stats await loadStats() // Refresh stats
return id return id
} }
const loadStats = async () => { const loadStats = async () => {
if (!isReady.value) return
try { try {
stats.value = await brain.value.stats() const res = await fetch('/api/brain/stats')
stats.value = await res.json()
} catch (err) { } catch (err) {
console.error('Failed to load stats:', err) console.error('Failed to load stats:', err)
} }
} }
return { return {
brain: readonly(brain),
isReady: readonly(isReady),
error: readonly(error), error: readonly(error),
stats: readonly(stats), stats: readonly(stats),
init,
search, search,
add, add,
loadStats loadStats
@ -867,7 +821,7 @@ const showResults = ref(false)
const selectedIndex = ref(-1) const selectedIndex = ref(-1)
const search = async (searchQuery) => { const search = async (searchQuery) => {
if (!brainyStore.isReady || !searchQuery.trim()) { if (!searchQuery.trim()) {
results.value = [] results.value = []
return return
} }
@ -1166,21 +1120,23 @@ onMounted(() => {
### Component Testing with Vitest ### Component Testing with Vitest
The component talks to the server over `fetch`, so mock the endpoint, not Brainy itself.
```javascript ```javascript
// tests/components/Search.test.js // tests/components/Search.test.js
import { mount } from '@vue/test-utils' import { mount } from '@vue/test-utils'
import { describe, it, expect, vi, beforeEach } from 'vitest' import { describe, it, expect, vi, beforeEach } from 'vitest'
import Search from '../src/components/Search.vue' import Search from '../src/components/Search.vue'
// Mock Brainy // Mock the server endpoint
vi.mock('@soulcraft/brainy', () => ({ beforeEach(() => {
Brainy: vi.fn().mockImplementation(() => ({ global.fetch = vi.fn().mockResolvedValue({
init: vi.fn().mockResolvedValue(undefined), ok: true,
find: vi.fn().mockResolvedValue([ json: async () => ({
{ id: '1', data: 'Test result', score: 0.9 } results: [{ id: '1', data: 'Test result', score: 0.9 }]
]) })
})) })
})) })
describe('Search Component', () => { describe('Search Component', () => {
let wrapper let wrapper
@ -1193,14 +1149,7 @@ describe('Search Component', () => {
expect(wrapper.find('input').exists()).toBe(true) expect(wrapper.find('input').exists()).toBe(true)
}) })
it('shows loading state initially', () => {
expect(wrapper.text()).toContain('Initializing AI')
})
it('performs search when input changes', async () => { it('performs search when input changes', async () => {
// Wait for brain to initialize
await wrapper.vm.$nextTick()
const input = wrapper.find('input') const input = wrapper.find('input')
await input.setValue('test query') await input.setValue('test query')
await input.trigger('input') await input.trigger('input')
@ -1208,6 +1157,7 @@ describe('Search Component', () => {
// Wait for debounced search // Wait for debounced search
await new Promise(resolve => setTimeout(resolve, 350)) await new Promise(resolve => setTimeout(resolve, 350))
expect(global.fetch).toHaveBeenCalled()
expect(wrapper.text()).toContain('Test result') expect(wrapper.text()).toContain('Test result')
}) })
}) })
@ -1241,6 +1191,8 @@ test('search functionality works', async ({ page }) => {
### Bundle Optimization ### Bundle Optimization
Keep Brainy out of the client bundle — it belongs to the server build only. Mark it external for SSR so the bundler resolves it at runtime instead of inlining it.
```javascript ```javascript
// vite.config.js // vite.config.js
import { defineConfig } from 'vite' import { defineConfig } from 'vite'
@ -1248,63 +1200,46 @@ import vue from '@vitejs/plugin-vue'
export default defineConfig({ export default defineConfig({
plugins: [vue()], plugins: [vue()],
define: { ssr: {
global: 'globalThis' external: ['@soulcraft/brainy']
},
optimizeDeps: {
include: ['@soulcraft/brainy']
},
build: {
rollupOptions: {
output: {
manualChunks: {
'brainy': ['@soulcraft/brainy']
}
}
}
} }
}) })
``` ```
### Error Handling ### Error Handling
Wrap the endpoint call with retry/backoff so transient network failures don't surface to the user.
```javascript ```javascript
// src/composables/useBrainyWithErrorHandling.js // src/composables/useBrainyWithErrorHandling.js
import { ref, onMounted } from 'vue' import { ref } from 'vue'
import { Brainy } from '@soulcraft/brainy'
export function useBrainyWithErrorHandling() { export function useBrainyWithErrorHandling(endpoint = '/api/brain') {
const brain = ref(null)
const isReady = ref(false)
const error = ref(null) const error = ref(null)
const retryCount = ref(0)
const init = async () => { const search = async (query, options = {}, maxRetries = 3) => {
try { error.value = null
brain.value = new Brainy() for (let attempt = 0; attempt <= maxRetries; attempt++) {
await brain.value.init() try {
isReady.value = true const res = await fetch(`${endpoint}/search`, {
error.value = null method: 'POST',
retryCount.value = 0 headers: { 'Content-Type': 'application/json' },
} catch (err) { body: JSON.stringify({ query, options })
error.value = err.message })
console.error('Brainy initialization failed:', err) if (!res.ok) throw new Error(`Search failed: ${res.status}`)
return (await res.json()).results
// Retry logic } catch (err) {
if (retryCount.value < 3) { error.value = err.message
retryCount.value++ if (attempt === maxRetries) throw err
setTimeout(init, 2000 * retryCount.value) // Exponential backoff before retrying
await new Promise(resolve => setTimeout(resolve, 2000 * (attempt + 1)))
} }
} }
} }
onMounted(init)
return { return {
brain: readonly(brain),
isReady: readonly(isReady),
error: readonly(error), error: readonly(error),
retry: init search
} }
} }
``` ```
@ -1315,6 +1250,13 @@ Here's a complete Vue 3 application structure:
``` ```
my-brainy-vue-app/ my-brainy-vue-app/
├── server/ # Server-side (Node/Bun) — hosts Brainy
│ ├── utils/
│ │ └── brain.js # Shared Brainy instance (getBrain)
│ └── api/brain/
│ ├── search.post.js
│ ├── add.post.js
│ └── stats.get.js
├── src/ ├── src/
│ ├── components/ │ ├── components/
│ │ ├── Search.vue │ │ ├── Search.vue
@ -1336,7 +1278,7 @@ my-brainy-vue-app/
└── package.json └── package.json
``` ```
This provides a complete, production-ready Vue.js application with comprehensive Brainy integration. This provides a complete, production-ready Vue.js application: Brainy runs on the server, and the client talks to it over HTTP.
## 📚 Next Steps ## 📚 Next Steps

View file

@ -386,8 +386,7 @@ npm install @soulcraft/brainy
## Requirements ## Requirements
- Node.js 18+ (for server/desktop) - Node.js 22+ or Bun (server-only)
- Modern browser (for web apps)
- Brainy 3.0+ - Brainy 3.0+
## API Reference ## API Reference

View file

@ -172,19 +172,6 @@ interface MetadataIndexOptionalHooks {
close?: () => Promise<void> | void close?: () => Promise<void> | void
} }
/**
* Optional progressive-initialization surface implemented by cloud storage
* adapters (plugin-provided in 8.0 the built-in filesystem/memory adapters
* complete their full init synchronously inside `init()` and don't expose
* these). Feature-detected with `typeof x === 'function'` before invoking.
*/
interface StorageBackgroundInitHooks {
/** Whether all background init tasks (bucket validation, count sync) finished. */
isBackgroundInitComplete?: () => boolean
/** Resolves when all background init tasks complete. */
awaitBackgroundInit?: () => Promise<void>
}
/** /**
* Brainy's internal resolved configuration. `normalizeConfig()` defaults every * Brainy's internal resolved configuration. `normalizeConfig()` defaults every
* field at construction except the genuinely-optional ones listed in the * field at construction except the genuinely-optional ones listed in the
@ -1075,13 +1062,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* This Promise is created in the constructor and resolves when init() completes. * This Promise is created in the constructor and resolves when init() completes.
* It can be awaited multiple times safely - the result is cached. * It can be awaited multiple times safely - the result is cached.
* *
* This enables reliable readiness detection for consumers, * This enables reliable readiness detection for consumers await it
* especially in cloud environments where progressive initialization means * before issuing queries when init() was fired without awaiting.
* init() returns quickly but background tasks may still be running.
* *
* @example Waiting for readiness before API calls * @example Waiting for readiness before API calls
* ```typescript * ```typescript
* const brain = new Brainy({ storage: { type: 'gcs', ... } }) * const brain = new Brainy()
* brain.init() // Fire and forget * brain.init() // Fire and forget
* *
* // Elsewhere in your code (e.g., API handler) * // Elsewhere in your code (e.g., API handler)
@ -1115,82 +1101,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return this._readyPromise return this._readyPromise
} }
/**
* Check if Brainy is fully initialized including all background tasks
*
* This checks both:
* 1. Basic initialization complete (init() returned)
* 2. Storage background tasks complete (bucket validation, count sync)
*
* Useful for determining if all lazy/progressive initialization is done.
*
* @returns true if all initialization including background tasks is complete
*
* @example Health check with background status
* ```typescript
* app.get('/health', (req, res) => {
* res.json({
* ready: brain.isInitialized,
* fullyInitialized: brain.isFullyInitialized(),
* status: brain.isFullyInitialized() ? 'ready' : 'warming'
* })
* })
* ```
*/
isFullyInitialized(): boolean {
if (!this.initialized) return false
// Check if storage has background init methods (cloud storage adapters)
const storage = this.storage as BaseStorage & StorageBackgroundInitHooks
if (typeof storage?.isBackgroundInitComplete === 'function') {
return storage.isBackgroundInitComplete()
}
// Non-cloud storage adapters are fully initialized after init()
return true
}
/**
* Wait for all background initialization tasks to complete
*
* For cloud storage adapters with progressive initialization,
* this waits for:
* - Bucket/container validation
* - Count synchronization
* - Any other background tasks
*
* For non-cloud storage, this resolves immediately.
*
* **Use Case**: Call this when you need guaranteed consistency, such as:
* - Before running batch operations
* - Before reporting full system health
* - When transitioning from "initializing" to "ready" status
*
* @returns Promise that resolves when all background tasks complete
*
* @example Ensuring full initialization
* ```typescript
* const brain = new Brainy({ storage: { type: 'gcs', ... } })
* await brain.init() // Fast return in cloud (<200ms)
*
* // Optional: wait for background tasks if needed
* await brain.awaitBackgroundInit()
* console.log('All background tasks complete')
* ```
*/
async awaitBackgroundInit(): Promise<void> {
// Must be initialized first
await this.ready
// Check if storage has background init methods (cloud storage adapters)
const storage = this.storage as BaseStorage & StorageBackgroundInitHooks
if (typeof storage?.awaitBackgroundInit === 'function') {
await storage.awaitBackgroundInit()
}
// Non-cloud storage: no background init to wait for
}
// ============= CORE CRUD OPERATIONS ============= // ============= CORE CRUD OPERATIONS =============
/** /**

View file

@ -1,54 +0,0 @@
/**
* Zero-Configuration System
* Main entry point for all auto-configuration features
*/
// Model configuration (simplified - always Q8 WASM)
export {
getModelPrecision,
shouldAutoDownloadModels,
getModelPath
} from './modelAutoConfig.js'
// Storage configuration
export {
autoDetectStorage,
StorageType,
StoragePreset,
StorageConfigResult,
logStorageConfig,
// Backward compatibility types
type StorageTypeString,
type StoragePresetString
} from './storageAutoConfig.js'
// Shared configuration for multi-instance
export {
SharedConfig,
SharedConfigManager
} from './sharedConfigManager.js'
// Main zero-config processor
export {
BrainyZeroConfig,
processZeroConfig,
createEmbeddingFunctionWithPrecision
} from './zeroConfig.js'
/**
* Main zero-config processor
* This is what Brainy will call
*/
export async function applyZeroConfig(input?: string | any): Promise<any> {
// Handle legacy config (full object) by detecting known legacy properties
if (input && typeof input === 'object' &&
(input.storage?.forceMemoryStorage || input.storage?.forceFileSystemStorage || input.storage?.s3Storage)) {
// This is a legacy config object - pass through unchanged
console.log('📦 Using legacy configuration format')
return input
}
// Process as zero-config (includes new object format)
const { processZeroConfig } = await import('./zeroConfig.js')
return processZeroConfig(input)
}

View file

@ -1,71 +0,0 @@
/**
* Model Configuration
* Brainy uses Q8 WASM embeddings - no configuration needed (zero-config)
*/
import { isNode } from '../utils/environment.js'
interface ModelConfigResult {
precision: 'q8'
reason: string
autoSelected: boolean
}
/**
* Get model precision configuration
* Always returns Q8 - the optimal balance of size and accuracy
*/
export function getModelPrecision(): ModelConfigResult {
return {
precision: 'q8',
reason: 'Q8 WASM (23MB bundled, no downloads)',
autoSelected: true
}
}
/**
* Check if running in a serverless environment
*/
function isServerlessEnvironment(): boolean {
if (!isNode()) return false
return !!(
process.env.AWS_LAMBDA_FUNCTION_NAME ||
process.env.VERCEL ||
process.env.NETLIFY ||
process.env.CLOUDFLARE_WORKERS ||
process.env.FUNCTIONS_WORKER_RUNTIME ||
process.env.K_SERVICE // Google Cloud Run
)
}
/**
* Check if models need to be downloaded
* With bundled WASM model, this is rarely needed
*/
export function shouldAutoDownloadModels(): boolean {
// Model is bundled - no downloads needed in normal operation
// This flag exists for edge cases only
const explicitlyDisabled = process.env.BRAINY_ALLOW_REMOTE_MODELS === 'false'
return !explicitlyDisabled
}
/**
* Get the model path
* With bundled WASM model, this points to the package assets
*/
export function getModelPath(): string {
// Check if user explicitly set a path (for advanced users)
if (process.env.BRAINY_MODELS_PATH) {
return process.env.BRAINY_MODELS_PATH
}
// Serverless - use /tmp for ephemeral storage
if (isServerlessEnvironment()) {
return '/tmp/.brainy/models'
}
// Node-like runtime - use home directory for persistent storage
const homeDir = process.env.HOME || process.env.USERPROFILE || '~'
return `${homeDir}/.brainy/models`
}

View file

@ -1,266 +0,0 @@
/**
* Shared Configuration Manager
* Ensures configuration consistency across multiple instances using shared storage
*/
import { StorageType } from './storageAutoConfig.js'
import { getBrainyVersion } from '../utils/version.js'
export interface SharedConfig {
// Critical parameters that MUST match across instances
version: string
precision: 'q8'
dimensions: number
hnswM: number
hnswEfConstruction: number
distanceFunction: string
createdAt: string
lastUpdated: string
// Informational (can differ between instances)
instanceCount?: number
lastAccessedBy?: string
}
/**
* Manages configuration consistency for shared storage
*/
export class SharedConfigManager {
private static CONFIG_FILE = '.brainy/config.json'
/**
* Load or create shared configuration
* When connecting to existing data, this OVERRIDES auto-configuration!
*/
static async loadOrCreateSharedConfig(
storage: any,
localConfig: any
): Promise<{ config: any; warnings: string[] }> {
const warnings: string[] = []
try {
// Check if we're using shared storage
if (!this.isSharedStorage(localConfig.storageType)) {
// Local storage - use local config
return { config: localConfig, warnings: [] }
}
// Try to load existing configuration from shared storage
const existingConfig = await this.loadConfigFromStorage(storage)
if (existingConfig) {
// EXISTING SHARED DATA - Must use its configuration!
console.log('📁 Found existing shared data configuration')
// Check for critical mismatches
const mismatches = this.checkCriticalMismatches(localConfig, existingConfig)
if (mismatches.length > 0) {
console.warn('⚠️ Configuration override required for shared storage:')
mismatches.forEach(m => {
console.warn(` - ${m.param}: ${m.local}${m.shared} (using shared)`)
warnings.push(`${m.param} overridden: ${m.local}${m.shared}`)
})
}
// Override critical parameters with shared values
const mergedConfig = this.mergeWithSharedConfig(localConfig, existingConfig)
// Update last accessed
await this.updateAccessInfo(storage, existingConfig)
return { config: mergedConfig, warnings }
} else {
// NEW SHARED STORAGE - Save our configuration
console.log('📝 Initializing new shared storage with configuration')
const sharedConfig = this.createSharedConfig(localConfig)
await this.saveConfigToStorage(storage, sharedConfig)
return { config: localConfig, warnings: [] }
}
} catch (error) {
console.error('Failed to manage shared configuration:', error)
warnings.push('Could not verify shared configuration - proceeding with caution')
return { config: localConfig, warnings }
}
}
/**
* Check if storage type is shared (multi-instance)
*/
private static isSharedStorage(storageType: StorageType): boolean {
return ['s3', 'gcs', 'r2'].includes(storageType)
}
/**
* Load configuration from shared storage
*/
private static async loadConfigFromStorage(storage: any): Promise<SharedConfig | null> {
try {
const configData = await storage.get(this.CONFIG_FILE)
if (!configData) return null
const config = JSON.parse(configData)
return config as SharedConfig
} catch (error) {
// Config doesn't exist yet
return null
}
}
/**
* Save configuration to shared storage
*/
private static async saveConfigToStorage(storage: any, config: SharedConfig): Promise<void> {
try {
await storage.set(this.CONFIG_FILE, JSON.stringify(config, null, 2))
} catch (error) {
console.error('Failed to save shared configuration:', error)
throw new Error('Cannot initialize shared storage without saving configuration')
}
}
/**
* Create shared configuration from local config
*/
private static createSharedConfig(localConfig: any): SharedConfig {
return {
version: getBrainyVersion(), // Brainy version
precision: localConfig.embeddingOptions?.precision || 'fp32',
dimensions: 384, // Fixed for all-MiniLM-L6-v2
hnswM: localConfig.hnsw?.M || 16,
hnswEfConstruction: localConfig.hnsw?.efConstruction || 200,
distanceFunction: localConfig.distanceFunction || 'cosine',
createdAt: new Date().toISOString(),
lastUpdated: new Date().toISOString(),
instanceCount: 1,
lastAccessedBy: this.getInstanceIdentifier()
}
}
/**
* Check for critical configuration mismatches
*/
private static checkCriticalMismatches(
localConfig: any,
sharedConfig: SharedConfig
): Array<{ param: string; local: any; shared: any }> {
const mismatches: Array<{ param: string; local: any; shared: any }> = []
// Model precision - CRITICAL!
const localPrecision = localConfig.embeddingOptions?.precision || 'fp32'
if (localPrecision !== sharedConfig.precision) {
mismatches.push({
param: 'Model Precision',
local: localPrecision,
shared: sharedConfig.precision
})
}
// HNSW parameters - Important for index consistency
const localM = localConfig.hnsw?.M || 16
if (localM !== sharedConfig.hnswM) {
mismatches.push({
param: 'HNSW M',
local: localM,
shared: sharedConfig.hnswM
})
}
const localEf = localConfig.hnsw?.efConstruction || 200
if (localEf !== sharedConfig.hnswEfConstruction) {
mismatches.push({
param: 'HNSW efConstruction',
local: localEf,
shared: sharedConfig.hnswEfConstruction
})
}
// Distance function
const localDistance = localConfig.distanceFunction || 'cosine'
if (localDistance !== sharedConfig.distanceFunction) {
mismatches.push({
param: 'Distance Function',
local: localDistance,
shared: sharedConfig.distanceFunction
})
}
return mismatches
}
/**
* Merge local config with shared config (shared takes precedence for critical params)
*/
private static mergeWithSharedConfig(localConfig: any, sharedConfig: SharedConfig): any {
return {
...localConfig,
// Override critical parameters with shared values
embeddingOptions: {
...localConfig.embeddingOptions,
precision: sharedConfig.precision // MUST use shared precision!
},
hnsw: {
...localConfig.hnsw,
M: sharedConfig.hnswM,
efConstruction: sharedConfig.hnswEfConstruction
},
distanceFunction: sharedConfig.distanceFunction,
// Add metadata about shared configuration
_sharedConfig: {
loaded: true,
version: sharedConfig.version,
createdAt: sharedConfig.createdAt,
precision: sharedConfig.precision
}
}
}
/**
* Update access information in shared config
*/
private static async updateAccessInfo(storage: any, config: SharedConfig): Promise<void> {
try {
config.lastUpdated = new Date().toISOString()
config.instanceCount = (config.instanceCount || 0) + 1
config.lastAccessedBy = this.getInstanceIdentifier()
await this.saveConfigToStorage(storage, config)
} catch {
// Non-critical - don't fail if we can't update access info
}
}
/**
* Get unique identifier for this instance
*/
private static getInstanceIdentifier(): string {
if (process.env.HOSTNAME) return process.env.HOSTNAME
if (process.env.CONTAINER_ID) return process.env.CONTAINER_ID
if (process.env.K_SERVICE) return process.env.K_SERVICE
if (process.env.AWS_LAMBDA_FUNCTION_NAME) return process.env.AWS_LAMBDA_FUNCTION_NAME
// Generate a random identifier
return `instance-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`
}
/**
* Validate that a configuration is compatible with shared data
*/
static validateCompatibility(config1: SharedConfig, config2: SharedConfig): boolean {
return (
config1.precision === config2.precision &&
config1.dimensions === config2.dimensions &&
config1.hnswM === config2.hnswM &&
config1.hnswEfConstruction === config2.hnswEfConstruction &&
config1.distanceFunction === config2.distanceFunction
)
}
}

View file

@ -1,113 +0,0 @@
/**
* @module config/storageAutoConfig
* @description Storage configuration auto-detection for Brainy 8.0.
*
* Brainy 8.0 ships **two storage adapters**:
* - `FILESYSTEM` persistent on-disk storage (Node, Bun, Deno).
* - `MEMORY` in-memory, ephemeral.
*
* Cloud storage adapters (GCS / S3 / R2 / Azure) and the browser-only OPFS
* adapter were removed. Cloud backup is now an operator concern handled by
* `gsutil` / `aws s3 cp` / `rclone` / `azcopy` against the on-disk artefact.
*
* This module is kept thin: a two-value enum, a tiny preset enum that mirrors
* it, and an `autoDetectStorage()` that picks filesystem when available and
* memory otherwise.
*/
import { isNode } from '../utils/environment.js'
/**
* Storage backend types shipped in Brainy 8.0.
*/
export enum StorageType {
MEMORY = 'memory',
FILESYSTEM = 'filesystem',
}
/**
* High-level storage presets. `AUTO` picks filesystem when available, memory
* otherwise; `MEMORY` and `DISK` are explicit overrides.
*/
export enum StoragePreset {
AUTO = 'auto',
MEMORY = 'memory',
DISK = 'disk',
}
export type StorageTypeString = 'memory' | 'filesystem'
export type StoragePresetString = 'auto' | 'memory' | 'disk'
export interface StorageConfigResult {
type: StorageType | StorageTypeString
config: { type: StorageType | StorageTypeString; rootDirectory?: string }
reason: string
autoSelected: boolean
}
/**
* Resolve a storage configuration. Accepts a {@link StorageType},
* {@link StoragePreset}, plain string equivalent, or a free-form storage
* config object. Returns a normalized result with the picked type, the
* underlying config payload, a human-readable reason, and whether the
* pick was auto-selected.
*/
export async function autoDetectStorage(
override?: StorageType | StoragePreset | StorageTypeString | StoragePresetString | any
): Promise<StorageConfigResult> {
if (override && typeof override === 'object') {
return {
type: (override.type as StorageType) ?? StorageType.MEMORY,
config: override,
reason: 'Manually configured storage',
autoSelected: false,
}
}
if (override === StorageType.MEMORY || override === StoragePreset.MEMORY || override === 'memory') {
return {
type: StorageType.MEMORY,
config: { type: StorageType.MEMORY },
reason: 'Explicit memory storage',
autoSelected: false,
}
}
if (override === StorageType.FILESYSTEM || override === StoragePreset.DISK || override === 'filesystem' || override === 'disk') {
return {
type: StorageType.FILESYSTEM,
config: { type: StorageType.FILESYSTEM, rootDirectory: './brainy-data' },
reason: 'Explicit filesystem storage',
autoSelected: false,
}
}
// Default: filesystem on Node-like runtimes, memory in browsers.
if (isNode()) {
return {
type: StorageType.FILESYSTEM,
config: { type: StorageType.FILESYSTEM, rootDirectory: './brainy-data' },
reason: 'Auto-selected: filesystem (Node-like runtime detected)',
autoSelected: true,
}
}
return {
type: StorageType.MEMORY,
config: { type: StorageType.MEMORY },
reason: 'Auto-selected: memory (no Node-like runtime detected)',
autoSelected: true,
}
}
/**
* Log a human-readable summary of a resolved storage config. Used by the
* zero-config init path for transparency.
*/
export function logStorageConfig(config: StorageConfigResult, verbose: boolean = false): void {
const prefix = config.autoSelected ? '[brainy] auto-selected' : '[brainy] using'
console.log(`${prefix} storage: ${config.type}${config.reason}`)
if (verbose) {
console.log('[brainy] storage config:', JSON.stringify(config.config))
}
}

View file

@ -1,354 +0,0 @@
/**
* Zero-Configuration System for Brainy
* Provides intelligent defaults while preserving full control
*/
import { getModelPrecision, getModelPath, shouldAutoDownloadModels } from './modelAutoConfig.js'
import { autoDetectStorage, StorageType, StoragePreset } from './storageAutoConfig.js'
import { AutoConfiguration } from '../utils/autoConfiguration.js'
/**
* Simplified configuration interface
* Everything is optional - zero config by default!
*/
export interface BrainyZeroConfig {
/**
* Configuration preset for common scenarios
* - 'production': Optimized for production (disk storage, auto model, default features)
* - 'development': Optimized for development (memory storage, fp32, verbose logging)
* - 'minimal': Minimal footprint (memory storage, q8, minimal features)
* - 'zero': True zero config (all auto-detected)
* - 'writer': Writer process (read + write) the single mutating process in
* Brainy's multi-process model
* - 'reader': Reader process (read-only, all mutations throw) any number of
* these may run against the same on-disk store as the writer
*/
mode?: 'production' | 'development' | 'minimal' | 'zero' | 'writer' | 'reader'
/**
* Storage configuration
* - 'memory': In-memory only (no persistence)
* - 'disk': Local disk (filesystem or OPFS)
* - 'cloud': Cloud storage (S3/GCS/R2 if configured)
* - 'auto': Auto-detect best option (default)
* - Object: Custom storage configuration
*/
storage?: StorageType | StoragePreset | any
/**
* Feature set configuration
* - 'minimal': Core features only (fastest startup)
* - 'default': Standard features (balanced)
* - 'full': All features enabled (most capable)
* - Array: Specific features to enable
*/
features?: 'minimal' | 'default' | 'full' | string[]
/**
* Logging verbosity
* - true: Show configuration decisions and progress
* - false: Silent operation (default in production)
*/
verbose?: boolean
/**
* Advanced configuration (escape hatch for power users)
* Any additional configuration can be passed here
*/
advanced?: any
}
/**
* Configuration presets for common scenarios
*/
const PRESETS = {
production: {
storage: 'disk' as const,
features: 'default' as const,
verbose: false
},
development: {
storage: 'memory' as const,
features: 'full' as const,
verbose: true
},
minimal: {
storage: 'memory' as const,
features: 'minimal' as const,
verbose: false
},
zero: {
storage: 'auto' as const,
features: 'default' as const,
verbose: false
},
writer: {
storage: 'auto' as const,
features: 'minimal' as const,
verbose: false,
// The single mutating process: HybridMode (read + write).
mode: 'writer' as const
},
reader: {
storage: 'auto' as const,
features: 'default' as const,
verbose: false,
// Read-only process: ReaderMode — every mutation throws.
mode: 'reader' as const
}
}
/**
* Feature sets configuration
*/
const FEATURE_SETS = {
minimal: [
'core',
'search',
'storage'
],
default: [
'core',
'search',
'storage',
'cache',
'metadata-index',
'batch-processing',
'entity-registry',
'request-deduplicator'
],
full: [
'core',
'search',
'storage',
'cache',
'metadata-index',
'batch-processing',
'entity-registry',
'request-deduplicator',
'connection-pool',
'wal',
'monitoring',
'metrics',
'intelligent-verb-scoring',
'triple-intelligence',
'neural-api'
]
}
/**
* Process zero-config input into full configuration
*/
export async function processZeroConfig(input?: string | BrainyZeroConfig): Promise<any> {
let config: BrainyZeroConfig = {}
// Handle string shorthand (preset name)
if (typeof input === 'string') {
if (input in PRESETS) {
// The `in` guard above proved the string is a preset key, which is
// exactly the BrainyZeroConfig mode union.
config = { mode: input as keyof typeof PRESETS }
} else {
throw new Error(`Unknown preset: ${input}. Valid presets: ${Object.keys(PRESETS).join(', ')}`)
}
} else if (input) {
config = input
}
// Apply preset if specified
if (config.mode && config.mode in PRESETS) {
const preset = PRESETS[config.mode]
config = {
...preset,
...config,
// Preserve explicit overrides
storage: config.storage ?? preset.storage,
features: config.features ?? preset.features,
verbose: config.verbose ?? preset.verbose
}
}
// Auto-detect environment if not in preset mode
const environment = detectEnvironmentMode()
// Get model configuration (always Q8 WASM)
const modelConfig = getModelPrecision()
// Process storage configuration
const storageConfig = await autoDetectStorage(config.storage)
// Process features configuration
const features = processFeatures(config.features)
// Get auto-configuration recommendations. Brainy 8.0 ships filesystem +
// memory only; there is no cloud-storage tier.
const autoConfig = await AutoConfiguration.getInstance().detectAndConfigure({
expectedDataSize: estimateDataSize(environment),
s3Available: false,
memoryBudget: undefined // Let it auto-detect
})
// Determine verbosity
const verbose = config.verbose ?? (process.env.NODE_ENV === 'development')
// Log configuration decisions if verbose
if (verbose) {
logConfigurationSummary({
mode: config.mode || 'auto',
model: modelConfig,
storage: storageConfig,
features: features,
environment: environment,
autoConfig: autoConfig
})
}
// Build final configuration
const finalConfig: any = {
// Model configuration
embeddingFunction: undefined, // Will be created with correct precision
embeddingOptions: {
precision: modelConfig.precision,
modelPath: getModelPath(),
allowRemoteDownload: shouldAutoDownloadModels()
},
// Storage configuration
storage: storageConfig.config,
storageType: storageConfig.type,
// HNSW configuration from auto-config
hnsw: {
M: autoConfig.recommendedConfig.enablePartitioning ? 32 : 16,
efConstruction: autoConfig.recommendedConfig.enablePartitioning ? 400 : 200,
maxDatasetSize: autoConfig.recommendedConfig.expectedDatasetSize,
partitioning: autoConfig.recommendedConfig.enablePartitioning,
maxNodesPerPartition: autoConfig.recommendedConfig.maxNodesPerPartition
},
// Cache configuration from auto-config
cache: {
autoTune: true,
hotCacheMaxSize: Math.floor(autoConfig.recommendedConfig.maxMemoryUsage / (1024 * 1024 * 10)), // 10% of memory budget
batchSize: autoConfig.recommendedConfig.enablePartitioning ? 100 : 50
},
// Features configuration
enabledFeatures: features,
// Metadata index configuration
metadataIndex: features.includes('metadata-index') ? {
enabled: true,
autoRebuild: true
} : undefined,
// Intelligent verb scoring
intelligentVerbScoring: features.includes('intelligent-verb-scoring') ? {
enabled: true
} : undefined,
// Logging configuration
logging: {
verbose: verbose
},
// Performance flags from auto-config
optimizations: autoConfig.optimizationFlags,
// Advanced overrides (if any)
...config.advanced
}
// Multi-process role: writer (read + write) or reader (read-only). This is the
// only signal Brainy needs — `mode` drives the operational mode (HybridMode vs
// ReaderMode) that gates every mutation path.
if (config.mode === 'writer' || config.mode === 'reader') {
finalConfig.mode = config.mode
if (verbose) {
console.log(`📡 Process role: ${config.mode.toUpperCase()}`)
}
}
return finalConfig
}
/**
* Detect environment mode if not specified
*/
function detectEnvironmentMode(): 'production' | 'development' | 'unknown' {
if (process.env.NODE_ENV === 'production') return 'production'
if (process.env.NODE_ENV === 'development') return 'development'
if (process.env.NODE_ENV === 'test') return 'development'
// Check for CI environments
if (process.env.CI || process.env.GITHUB_ACTIONS) return 'production'
// Check for production indicators
if (process.env.VERCEL_ENV === 'production' ||
process.env.NETLIFY_ENV === 'production' ||
process.env.RAILWAY_ENVIRONMENT === 'production') {
return 'production'
}
return 'unknown'
}
/**
* Process features configuration
*/
function processFeatures(features?: 'minimal' | 'default' | 'full' | string[]): string[] {
if (Array.isArray(features)) {
return features
}
if (features && features in FEATURE_SETS) {
return FEATURE_SETS[features]
}
// Default based on environment
const env = detectEnvironmentMode()
if (env === 'production') return FEATURE_SETS.default
if (env === 'development') return FEATURE_SETS.full
return FEATURE_SETS.default
}
/**
* Estimate dataset size based on environment
*/
function estimateDataSize(environment: string): number {
switch (environment) {
case 'production': return 100000
case 'development': return 10000
default: return 50000
}
}
/**
* Log configuration summary
*/
function logConfigurationSummary(config: any): void {
console.log('\n🧠 Brainy Zero-Config Summary')
console.log('================================')
console.log(`Mode: ${config.mode}`)
console.log(`Environment: ${config.environment}`)
console.log(`Model: ${config.model.precision.toUpperCase()} (${config.model.reason})`)
console.log(`Storage: ${config.storage.type.toUpperCase()} (${config.storage.reason})`)
console.log(`Features: ${config.features.length} enabled`)
console.log(`Memory Budget: ${Math.floor(config.autoConfig.recommendedConfig.maxMemoryUsage / (1024 * 1024))}MB`)
console.log(`Expected Dataset: ${config.autoConfig.recommendedConfig.expectedDatasetSize.toLocaleString()} items`)
console.log('================================\n')
}
/**
* Create embedding function (always Q8 WASM)
*/
export async function createEmbeddingFunctionWithPrecision(): Promise<any> {
const { createEmbeddingFunction } = await import('../utils/embedding.js')
// Create embedding function - always Q8 WASM
return createEmbeddingFunction({
precision: 'q8',
verbose: false // Silent by default in zero-config
})
}

View file

@ -77,11 +77,6 @@ export type {
// Export Aggregation Engine // Export Aggregation Engine
export { AggregationIndex, AggregateMaterializer, bucketTimestamp, parseBucketRange } from './aggregation/index.js' export { AggregationIndex, AggregateMaterializer, bucketTimestamp, parseBucketRange } from './aggregation/index.js'
// Export zero-configuration input type
export {
BrainyZeroConfig
} from './config/index.js'
// Export Neural Import (AI data understanding) // Export Neural Import (AI data understanding)
export { NeuralImport } from './neural/neuralImport.js' export { NeuralImport } from './neural/neuralImport.js'
export type { export type {

View file

@ -18,44 +18,6 @@ import { StorageBatchConfig } from '../baseStorage.js'
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js' import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js' import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
// =============================================================================
// Progressive Initialization Types
// =============================================================================
/**
* Initialization mode for cloud storage adapters.
*
* Controls how storage adapters initialize during startup, enabling
* fast cold starts in serverless environments while maintaining strict
* validation for local development.
*
*
* | Mode | Description | Use Case |
* |------|-------------|----------|
* | `'auto'` | Auto-detect environment (progressive in cloud, strict locally) | Default, zero-config |
* | `'progressive'` | Fast init (<200ms), validate lazily on first write | Serverless, Cloud Run |
* | `'strict'` | Blocking validation during init (traditional behavior) | Local dev, testing |
*
* @example
* ```typescript
* // Zero-config: auto-detects Cloud Run, Lambda, etc.
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
*
* // Force progressive mode for all environments
* const storage = new FileSystemStorage({
* bucket: 'my-bucket',
* initMode: 'progressive'
* })
*
* // Force strict validation (useful for testing)
* const storage = new FileSystemStorage({
* bucket: 'my-bucket',
* initMode: 'strict'
* })
* ```
*/
export type InitMode = 'progressive' | 'strict' | 'auto'
/** /**
* Base class for storage adapters that implements statistics tracking * Base class for storage adapters that implements statistics tracking
*/ */
@ -372,71 +334,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
status: 'normal' | 'throttled' | 'recovering' status: 'normal' | 'throttled' | 'recovering'
}> = new Map() }> = new Map()
// =============================================
// Progressive Initialization State
// =============================================
// These properties enable fast cold starts in cloud environments
// by deferring validation and count loading to background tasks.
/**
* Initialization mode for this storage adapter.
*
* - `'auto'` (default): Progressive in cloud environments, strict locally
* - `'progressive'`: Always use fast init with lazy validation
* - `'strict'`: Always validate during init (traditional behavior)
*
* @protected
*/
protected initMode: InitMode = 'auto'
/**
* Whether the bucket/container has been validated to exist.
*
* In progressive mode, this starts as `false` and is set to `true`
* after the first successful write operation or background validation.
*
* @protected
*/
protected bucketValidated = false
/**
* Error from bucket validation, if any.
*
* Stored here so subsequent operations can fail fast with the same error.
*
* @protected
*/
protected bucketValidationError: Error | null = null
/**
* Whether counts have been loaded from storage.
*
* In progressive mode, counts start at 0 and are loaded in the background.
* Operations work immediately; counts become accurate after background load.
*
* @protected
*/
protected countsLoaded = false
/**
* Whether all background initialization tasks have completed.
*
* Useful for tests and diagnostics to ensure full initialization.
*
* @protected
*/
protected backgroundTasksComplete = false
/**
* Promise that resolves when background initialization completes.
*
* Can be awaited by callers who need to ensure full initialization
* before proceeding (e.g., in tests).
*
* @protected
*/
protected backgroundInitPromise: Promise<void> | null = null
// Statistics-specific methods that must be implemented by subclasses // Statistics-specific methods that must be implemented by subclasses
protected abstract saveStatisticsData( protected abstract saveStatisticsData(
statistics: StatisticsData statistics: StatisticsData
@ -1129,18 +1026,11 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL
// ============================================= // =============================================
// Smart Count Batching // Count Persistence
// ============================================= // =============================================
// Count batching state - mirrors statistics batching pattern // Counts changed since the last persist? Drives the write-through flush.
protected pendingCountPersist = false // Counts changed since last persist? protected pendingCountPersist = false
protected lastCountPersistTime = 0 // Timestamp of last persist
protected scheduledCountPersistTimeout: NodeJS.Timeout | null = null // Scheduled persist timer
protected pendingCountOperations = 0 // Operations since last persist
// Batching configuration (overridable by subclasses for custom strategies)
protected countPersistBatchSize = 10 // Operations before forcing persist (cloud storage)
protected countPersistInterval = 5000 // Milliseconds before forcing persist (cloud storage)
/** /**
* Get total noun count - O(1) operation * Get total noun count - O(1) operation
@ -1295,230 +1185,23 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
// ============================================= // =============================================
/** /**
* Detect if this storage adapter uses cloud storage (network I/O) * Persist counts immediately.
* Cloud storage benefits from batching; local storage does not.
* *
* Override this method in subclasses for accurate detection. * Filesystem and memory storage have no network latency, so counts are
* Default implementation checks storage type from getStorageStatus(). * written through on every change rather than batched.
*
* @returns true if cloud storage (GCS, S3, R2), false if local (File, Memory)
*/
protected isCloudStorage(): boolean {
// Default: assume local storage (conservative, prefers reliability over performance)
// Subclasses should override this for accurate detection
return false
}
// =============================================
// Progressive Initialization Methods
// =============================================
/**
* Detect if running in a cloud/serverless environment.
*
* Cloud environments benefit from progressive initialization to minimize
* cold start times. This method checks for common environment variables
* set by cloud providers.
*
* | Provider | Environment Variable |
* |----------|---------------------|
* | Cloud Run | `K_SERVICE`, `K_REVISION` |
* | Lambda | `AWS_LAMBDA_FUNCTION_NAME` |
* | Cloud Functions | `FUNCTIONS_TARGET` |
* | Azure Functions | `AZURE_FUNCTIONS_ENVIRONMENT` |
*
* @returns `true` if running in a detected cloud environment
* @protected
*/
protected detectCloudEnvironment(): boolean {
return !!(
process.env.K_SERVICE || // Google Cloud Run
process.env.K_REVISION || // Google Cloud Run (revision)
process.env.AWS_LAMBDA_FUNCTION_NAME || // AWS Lambda
process.env.FUNCTIONS_TARGET || // Google Cloud Functions
process.env.AZURE_FUNCTIONS_ENVIRONMENT // Azure Functions
)
}
/**
* Determine the effective initialization mode.
*
* Resolves `'auto'` to either `'progressive'` or `'strict'` based on
* the detected environment.
*
* @returns The resolved init mode (`'progressive'` or `'strict'`)
* @protected
*/
protected resolveInitMode(): 'progressive' | 'strict' {
if (this.initMode === 'auto') {
return this.detectCloudEnvironment() ? 'progressive' : 'strict'
}
return this.initMode
}
/**
* Schedule background initialization tasks.
*
* This method is called in progressive mode after the adapter is marked
* as initialized. It schedules non-blocking tasks to:
* 1. Validate bucket/container existence
* 2. Load counts from storage
*
* Override in subclasses to add storage-specific background tasks.
* Always call `super.scheduleBackgroundInit()` first.
*
* @protected
*/
protected scheduleBackgroundInit(): void {
// Create a promise that tracks all background work
this.backgroundInitPromise = this.runBackgroundInit()
// Don't await - let it run in background
this.backgroundInitPromise
.then(() => {
this.backgroundTasksComplete = true
})
.catch((error) => {
// Log but don't throw - progressive mode is best-effort
console.error('[Progressive Init] Background initialization failed:', error)
this.backgroundTasksComplete = true // Mark complete even on error
})
}
/**
* Run background initialization tasks.
*
* Override in subclasses to add storage-specific tasks.
* The default implementation does nothing (for local storage adapters).
*
* @protected
*/
protected async runBackgroundInit(): Promise<void> {
// Default implementation: nothing to do for local storage
// Cloud storage adapters override this to:
// 1. Validate bucket/container
// 2. Load counts from storage
}
/**
* Wait for background initialization to complete.
*
* Useful in tests or when you need to ensure all background tasks
* have finished before proceeding.
*
* @example
* ```typescript
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
* await storage.init()
*
* // Wait for background validation and count loading
* await storage.awaitBackgroundInit()
*
* // Now counts are accurate and bucket is validated
* const count = await storage.getNounCount()
* ```
*
* @public
*/
public async awaitBackgroundInit(): Promise<void> {
if (this.backgroundInitPromise) {
await this.backgroundInitPromise
}
}
/**
* Check if background initialization has completed.
*
* @returns `true` if all background tasks are complete
* @public
*/
public isBackgroundInitComplete(): boolean {
return this.backgroundTasksComplete
}
/**
* Ensure bucket/container is validated before write operations.
*
* In progressive mode, bucket validation is deferred until the first
* write operation. This method performs lazy validation and caches
* the result (or error) for subsequent calls.
*
* Override in subclasses to implement storage-specific validation.
* The base implementation does nothing (for local storage adapters).
*
* @throws Error if bucket validation fails
* @protected
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do
if (this.bucketValidated) {
return
}
// If we have a cached validation error, throw it
if (this.bucketValidationError) {
throw this.bucketValidationError
}
// Default implementation: no validation needed (local storage)
// Cloud storage adapters override this to validate bucket/container
this.bucketValidated = true
}
/**
* Schedule a smart batched persist operation.
*
* Strategy:
* - Local Storage: Persist immediately (fast, no network latency)
* - Cloud Storage: Batch persist (10 ops OR 5 seconds, whichever first)
*
* This mirrors the statistics batching pattern for consistency.
*/ */
protected async scheduleCountPersist(): Promise<void> { protected async scheduleCountPersist(): Promise<void> {
// Mark counts as pending persist
this.pendingCountPersist = true this.pendingCountPersist = true
this.pendingCountOperations++ await this.flushCounts()
// Local storage: persist immediately (fast enough, no benefit from batching)
if (!this.isCloudStorage()) {
await this.flushCounts()
return
}
// Cloud storage: use smart batching
// Persist if we've hit the batch size threshold
if (this.pendingCountOperations >= this.countPersistBatchSize) {
await this.flushCounts()
return
}
// Otherwise, schedule a time-based persist if not already scheduled
if (!this.scheduledCountPersistTimeout) {
this.scheduledCountPersistTimeout = setTimeout(() => {
this.flushCounts().catch(error => {
console.error('Failed to flush counts on timer:', error)
})
}, this.countPersistInterval)
}
} }
/** /**
* Flush counts immediately to storage. * Flush pending counts to storage.
* *
* Used for: * Used for graceful shutdown (SIGTERM handler) and the immediate
* - Graceful shutdown (SIGTERM handler) * write-through path. This is the public API that shutdown hooks can call.
* - Forced persist (batch threshold reached)
* - Local storage immediate persist
*
* This is the public API that shutdown hooks can call.
*/ */
async flushCounts(): Promise<void> { async flushCounts(): Promise<void> {
// Clear any scheduled persist
if (this.scheduledCountPersistTimeout) {
clearTimeout(this.scheduledCountPersistTimeout)
this.scheduledCountPersistTimeout = null
}
// Nothing to flush? // Nothing to flush?
if (!this.pendingCountPersist) { if (!this.pendingCountPersist) {
return return
@ -1527,13 +1210,9 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
try { try {
// Persist to storage (implemented by subclass) // Persist to storage (implemented by subclass)
await this.persistCounts() await this.persistCounts()
// Update state
this.lastCountPersistTime = Date.now()
this.pendingCountPersist = false this.pendingCountPersist = false
this.pendingCountOperations = 0
} catch (error) { } catch (error) {
console.error('CRITICAL: Failed to flush counts to storage:', error) console.error('CRITICAL: Failed to flush counts to storage:', error)
// Keep pending flag set so we retry on next operation // Keep pending flag set so we retry on next operation
throw error throw error
} }

View file

@ -29,17 +29,6 @@ export interface BrainyInterface<T = unknown> {
*/ */
readonly isInitialized: boolean readonly isInitialized: boolean
/**
* Check if all initialization including background tasks is complete
*/
isFullyInitialized(): boolean
/**
* Wait for all background initialization tasks to complete
* For cloud storage adapters, this waits for bucket validation and count sync.
*/
awaitBackgroundInit(): Promise<void>
/** /**
* Modern add method - unified entity creation * Modern add method - unified entity creation
* @param params Parameters for adding entities * @param params Parameters for adding entities

View file

@ -1,428 +0,0 @@
/**
* Automatic Configuration System for Brainy Vector Database
* Detects environment, resources, and data patterns to provide optimal settings
*/
import { isNode } from './environment.js'
export interface AutoConfigResult {
// Environment details
environment: 'nodejs' | 'serverless' | 'unknown'
// Resource detection
availableMemory: number // bytes
cpuCores: number
// Storage capabilities
persistentStorageAvailable: boolean
s3StorageDetected: boolean
// Recommended configuration
recommendedConfig: {
expectedDatasetSize: number
maxMemoryUsage: number
targetSearchLatency: number
enablePartitioning: boolean
enableCompression: boolean
enablePredictiveCaching: boolean
partitionStrategy: 'semantic' | 'hash'
maxNodesPerPartition: number
semanticClusters: number
}
// Performance optimization flags
optimizationFlags: {
useMemoryMapping: boolean
aggressiveCaching: boolean
backgroundOptimization: boolean
compressionLevel: 'none' | 'light' | 'aggressive'
}
}
export interface DatasetAnalysis {
estimatedSize: number
vectorDimension?: number
growthRate?: number // vectors per second
accessPatterns?: 'read-heavy' | 'write-heavy' | 'balanced'
}
/**
* Automatic configuration system that detects environment and optimizes settings
*/
export class AutoConfiguration {
private static instance: AutoConfiguration
private cachedConfig: AutoConfigResult | null = null
private datasetStats: DatasetAnalysis = { estimatedSize: 0 }
private constructor() {}
public static getInstance(): AutoConfiguration {
if (!AutoConfiguration.instance) {
AutoConfiguration.instance = new AutoConfiguration()
}
return AutoConfiguration.instance
}
/**
* Detect environment and generate optimal configuration
*/
public async detectAndConfigure(hints?: {
expectedDataSize?: number
s3Available?: boolean
memoryBudget?: number
}): Promise<AutoConfigResult> {
if (this.cachedConfig && !hints) {
return this.cachedConfig
}
const environment = this.detectEnvironment()
const resources = await this.detectResources()
const storage = await this.detectStorageCapabilities(hints?.s3Available)
const config: AutoConfigResult = {
environment,
...resources,
...storage,
recommendedConfig: this.generateRecommendedConfig(environment, resources, hints),
optimizationFlags: this.generateOptimizationFlags(environment, resources)
}
this.cachedConfig = config
return config
}
/**
* Update configuration based on runtime dataset analysis
*/
public async adaptToDataset(analysis: DatasetAnalysis): Promise<AutoConfigResult> {
this.datasetStats = analysis
// Regenerate configuration with dataset insights
const currentConfig = await this.detectAndConfigure()
const adaptedConfig = this.adaptConfigurationToData(currentConfig, analysis)
this.cachedConfig = adaptedConfig
return adaptedConfig
}
/**
* Learn from performance metrics and adjust configuration
*/
public async learnFromPerformance(metrics: {
averageSearchTime: number
memoryUsage: number
cacheHitRate: number
errorRate: number
}): Promise<Partial<AutoConfigResult['recommendedConfig']>> {
const adjustments: Partial<AutoConfigResult['recommendedConfig']> = {}
// Learn from search performance
if (metrics.averageSearchTime > 200) {
// Too slow - optimize for speed
adjustments.maxNodesPerPartition = Math.max(10000, (this.cachedConfig?.recommendedConfig.maxNodesPerPartition || 50000) * 0.8)
} else if (metrics.averageSearchTime < 50) {
// Very fast - can optimize for quality
adjustments.maxNodesPerPartition = Math.min(100000, (this.cachedConfig?.recommendedConfig.maxNodesPerPartition || 50000) * 1.2)
}
// Learn from memory usage
if (metrics.memoryUsage > (this.cachedConfig?.recommendedConfig.maxMemoryUsage || 0) * 0.9) {
// High memory usage - enable compression
adjustments.enableCompression = true
}
// Learn from cache performance
if (metrics.cacheHitRate < 0.7) {
// Poor cache performance - enable predictive caching
adjustments.enablePredictiveCaching = true
}
// Update cached config with learned adjustments
if (this.cachedConfig) {
this.cachedConfig.recommendedConfig = {
...this.cachedConfig.recommendedConfig,
...adjustments
}
}
return adjustments
}
/**
* Get minimal configuration for quick setup
*/
public async getQuickSetupConfig(scenario: 'small' | 'medium' | 'large' | 'enterprise'): Promise<{
expectedDatasetSize: number
maxMemoryUsage: number
targetSearchLatency: number
s3Required: boolean
}> {
const environment = this.detectEnvironment()
const resources = await this.detectResources()
switch (scenario) {
case 'small':
return {
expectedDatasetSize: 10000,
maxMemoryUsage: Math.min(resources.availableMemory * 0.3, 1024 * 1024 * 1024), // 1GB max
targetSearchLatency: 100,
s3Required: false
}
case 'medium':
return {
expectedDatasetSize: 100000,
maxMemoryUsage: Math.min(resources.availableMemory * 0.5, 4 * 1024 * 1024 * 1024), // 4GB max
targetSearchLatency: 150,
s3Required: environment === 'serverless'
}
case 'large':
return {
expectedDatasetSize: 1000000,
maxMemoryUsage: Math.min(resources.availableMemory * 0.7, 8 * 1024 * 1024 * 1024), // 8GB max
targetSearchLatency: 200,
s3Required: true
}
case 'enterprise':
return {
expectedDatasetSize: 10000000,
maxMemoryUsage: Math.min(resources.availableMemory * 0.8, 32 * 1024 * 1024 * 1024), // 32GB max
targetSearchLatency: 300,
s3Required: true
}
}
}
/**
* Detect the current runtime environment
*/
private detectEnvironment(): 'nodejs' | 'serverless' | 'unknown' {
if (isNode()) {
// Check for serverless environment indicators
if (process.env.AWS_LAMBDA_FUNCTION_NAME ||
process.env.VERCEL ||
process.env.NETLIFY ||
process.env.CLOUDFLARE_WORKERS) {
return 'serverless'
}
return 'nodejs'
}
return 'unknown'
}
/**
* Detect available system resources
*/
private async detectResources(): Promise<{
availableMemory: number
cpuCores: number
}> {
let availableMemory = 2 * 1024 * 1024 * 1024 // Default 2GB
let cpuCores = 4 // Default 4 cores
if (isNode()) {
try {
const os = await import('node:os')
availableMemory = os.totalmem() * 0.7 // Use 70% of total memory
cpuCores = os.cpus().length
} catch {
// Fallback to defaults
}
}
return { availableMemory, cpuCores }
}
/**
* Detect available storage capabilities
*/
private async detectStorageCapabilities(s3Hint?: boolean): Promise<{
persistentStorageAvailable: boolean
s3StorageDetected: boolean
}> {
let persistentStorageAvailable = false
let s3StorageDetected = s3Hint || false
if (isNode()) {
persistentStorageAvailable = true
// 8.0 dropped cloud storage adapters; the s3StorageDetected flag is
// preserved as a hint for operator backup tooling, not adapter wiring.
s3StorageDetected = s3Hint ||
!!(process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY) ||
!!(process.env.S3_BUCKET_NAME)
}
return {
persistentStorageAvailable,
s3StorageDetected
}
}
/**
* Generate recommended configuration based on detected environment and resources
*/
private generateRecommendedConfig(
environment: string,
resources: { availableMemory: number; cpuCores: number },
hints?: { expectedDataSize?: number; memoryBudget?: number }
): AutoConfigResult['recommendedConfig'] {
const datasetSize = hints?.expectedDataSize || this.estimateDatasetSize()
const memoryBudget = hints?.memoryBudget || Math.floor(resources.availableMemory * 0.6)
// Base configuration
let config = {
expectedDatasetSize: datasetSize,
maxMemoryUsage: memoryBudget,
targetSearchLatency: 150,
enablePartitioning: datasetSize > 25000,
enableCompression: memoryBudget < 2 * 1024 * 1024 * 1024,
enablePredictiveCaching: true,
partitionStrategy: 'semantic' as const,
maxNodesPerPartition: 50000,
semanticClusters: 8
}
// Environment-specific adjustments
switch (environment) {
case 'serverless':
config = {
...config,
targetSearchLatency: 500, // Account for cold starts
enablePredictiveCaching: false, // Avoid background processes
maxNodesPerPartition: 30000 // Moderate partition size
}
break
case 'nodejs':
config = {
...config,
targetSearchLatency: 100, // Aggressive for Node.js
maxNodesPerPartition: Math.min(100000, Math.floor(datasetSize / 10)), // Larger partitions
semanticClusters: Math.min(16, Math.max(4, Math.floor(datasetSize / 50000))) // Scale clusters with data
}
break
}
// Dataset size adjustments
if (datasetSize > 1000000) {
config.semanticClusters = Math.min(32, Math.floor(datasetSize / 100000))
config.maxNodesPerPartition = 100000
} else if (datasetSize < 10000) {
config.enablePartitioning = false
config.partitionStrategy = 'semantic' // Keep semantic but disable partitioning
}
return config
}
/**
* Generate optimization flags based on environment and resources
*/
private generateOptimizationFlags(
environment: string,
resources: { availableMemory: number; cpuCores: number }
): AutoConfigResult['optimizationFlags'] {
return {
useMemoryMapping: environment === 'nodejs' && resources.availableMemory > 4 * 1024 * 1024 * 1024,
aggressiveCaching: resources.availableMemory > 2 * 1024 * 1024 * 1024,
backgroundOptimization: environment !== 'serverless' && resources.cpuCores > 2,
compressionLevel: resources.availableMemory < 1024 * 1024 * 1024 ? 'aggressive' :
resources.availableMemory < 4 * 1024 * 1024 * 1024 ? 'light' : 'none'
}
}
/**
* Adapt configuration based on actual dataset analysis
*/
private adaptConfigurationToData(
baseConfig: AutoConfigResult,
analysis: DatasetAnalysis
): AutoConfigResult {
const updatedConfig = { ...baseConfig }
// Adjust based on actual dataset size
if (analysis.estimatedSize !== baseConfig.recommendedConfig.expectedDatasetSize) {
const sizeRatio = analysis.estimatedSize / baseConfig.recommendedConfig.expectedDatasetSize
updatedConfig.recommendedConfig.expectedDatasetSize = analysis.estimatedSize
// Scale partition size with dataset
if (sizeRatio > 2) {
updatedConfig.recommendedConfig.maxNodesPerPartition = Math.min(
100000,
Math.floor(updatedConfig.recommendedConfig.maxNodesPerPartition * 1.5)
)
updatedConfig.recommendedConfig.semanticClusters = Math.min(
32,
Math.floor(updatedConfig.recommendedConfig.semanticClusters * 1.5)
)
}
}
// Adjust based on vector dimension
if (analysis.vectorDimension) {
if (analysis.vectorDimension > 1024) {
// High-dimensional vectors - optimize for compression
updatedConfig.recommendedConfig.enableCompression = true
updatedConfig.optimizationFlags.compressionLevel = 'aggressive'
}
}
// Adjust based on access patterns
if (analysis.accessPatterns === 'read-heavy') {
updatedConfig.recommendedConfig.enablePredictiveCaching = true
updatedConfig.optimizationFlags.aggressiveCaching = true
} else if (analysis.accessPatterns === 'write-heavy') {
updatedConfig.recommendedConfig.enablePredictiveCaching = false
updatedConfig.optimizationFlags.backgroundOptimization = false
}
return updatedConfig
}
/**
* Estimate dataset size if not provided
*/
private estimateDatasetSize(): number {
// Start with conservative estimate
const environment = this.detectEnvironment()
switch (environment) {
case 'serverless': return 50000
case 'nodejs': return 100000
default: return 25000
}
}
/**
* Reset cached configuration (for testing or manual refresh)
*/
public resetCache(): void {
this.cachedConfig = null
this.datasetStats = { estimatedSize: 0 }
}
}
/**
* Convenience function for quick auto-configuration
*/
export async function autoConfigureBrainy(hints?: {
expectedDataSize?: number
s3Available?: boolean
memoryBudget?: number
}): Promise<AutoConfigResult> {
const autoConfig = AutoConfiguration.getInstance()
return autoConfig.detectAndConfigure(hints)
}
/**
* Get quick setup configuration for common scenarios
*/
export async function getQuickSetup(scenario: 'small' | 'medium' | 'large' | 'enterprise') {
const autoConfig = AutoConfiguration.getInstance()
return autoConfig.getQuickSetupConfig(scenario)
}