refactor(8.0)!: remove orphaned zero-config subsystem + dead cloud/progressive-init storage vestige
The old config-generation subsystem (src/config/ + autoConfiguration.ts) was
superseded during the 8.0 rework and never wired into init(): it emitted settings
for a partitioning subsystem that no longer exists and probed deleted cloud env
vars. The live zero-config path is inline — recall preset → HNSW knobs, storage
auto-detect, auto persistMode, container-memory-aware cache sizing.
The storage progressive-init / cloud-detection cluster was equally dead after the
cloud adapters were dropped: isCloudStorage() is permanently false (no overriders),
scheduleBackgroundInit/runBackgroundInit were never called (the latter an empty
body), initMode was never assigned, and Brainy.isFullyInitialized()/
awaitBackgroundInit() were always-trivial with zero callers. scheduleCountPersist()
collapses to its only-ever-taken immediate write-through path.
Removed:
- src/config/{index,zeroConfig,storageAutoConfig,modelAutoConfig,sharedConfigManager}.ts
- src/utils/autoConfiguration.ts + the inert BrainyZeroConfig export
- Brainy.isFullyInitialized()/awaitBackgroundInit() (+ BrainyInterface decls)
- InitMode type, isCloudStorage/detectCloudEnvironment/resolveInitMode,
scheduleBackgroundInit/runBackgroundInit/ensureValidatedForWrite and their state
- Dead cloud env-var probes (K_SERVICE/K_REVISION/AWS_LAMBDA_FUNCTION_NAME/
FUNCTIONS_TARGET/AZURE_FUNCTIONS_ENVIRONMENT)
Kept (verified live): production-detection logging (environment.ts), container-
memory cache sizing (memoryDetection/paramValidation), on-disk hash bucketing
(sharding.ts).
Docs: scrubbed deleted-subsystem references (JS quantization knobs, cloud/OPFS
adapters, partitioning, old zero-config API) across 14 files; deleted two wholly-
obsolete feature docs (complete-feature-list, v3-features); rewrote
architecture/zero-config for 8.0.
~3,700 LOC removed. Build clean; 1392 unit + 24 db-mvcc green.
This commit is contained in:
parent
00d3203d68
commit
35b9d7ef43
28 changed files with 596 additions and 3752 deletions
|
|
@ -172,19 +172,6 @@ interface MetadataIndexOptionalHooks {
|
|||
close?: () => Promise<void> | void
|
||||
}
|
||||
|
||||
/**
|
||||
* Optional progressive-initialization surface implemented by cloud storage
|
||||
* adapters (plugin-provided in 8.0 — the built-in filesystem/memory adapters
|
||||
* complete their full init synchronously inside `init()` and don't expose
|
||||
* these). Feature-detected with `typeof x === 'function'` before invoking.
|
||||
*/
|
||||
interface StorageBackgroundInitHooks {
|
||||
/** Whether all background init tasks (bucket validation, count sync) finished. */
|
||||
isBackgroundInitComplete?: () => boolean
|
||||
/** Resolves when all background init tasks complete. */
|
||||
awaitBackgroundInit?: () => Promise<void>
|
||||
}
|
||||
|
||||
/**
|
||||
* Brainy's internal resolved configuration. `normalizeConfig()` defaults every
|
||||
* field at construction except the genuinely-optional ones listed in the
|
||||
|
|
@ -1075,13 +1062,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* This Promise is created in the constructor and resolves when init() completes.
|
||||
* It can be awaited multiple times safely - the result is cached.
|
||||
*
|
||||
* This enables reliable readiness detection for consumers,
|
||||
* especially in cloud environments where progressive initialization means
|
||||
* init() returns quickly but background tasks may still be running.
|
||||
* This enables reliable readiness detection for consumers — await it
|
||||
* before issuing queries when init() was fired without awaiting.
|
||||
*
|
||||
* @example Waiting for readiness before API calls
|
||||
* ```typescript
|
||||
* const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||
* const brain = new Brainy()
|
||||
* brain.init() // Fire and forget
|
||||
*
|
||||
* // Elsewhere in your code (e.g., API handler)
|
||||
|
|
@ -1115,82 +1101,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
return this._readyPromise
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if Brainy is fully initialized including all background tasks
|
||||
*
|
||||
* This checks both:
|
||||
* 1. Basic initialization complete (init() returned)
|
||||
* 2. Storage background tasks complete (bucket validation, count sync)
|
||||
*
|
||||
* Useful for determining if all lazy/progressive initialization is done.
|
||||
*
|
||||
* @returns true if all initialization including background tasks is complete
|
||||
*
|
||||
* @example Health check with background status
|
||||
* ```typescript
|
||||
* app.get('/health', (req, res) => {
|
||||
* res.json({
|
||||
* ready: brain.isInitialized,
|
||||
* fullyInitialized: brain.isFullyInitialized(),
|
||||
* status: brain.isFullyInitialized() ? 'ready' : 'warming'
|
||||
* })
|
||||
* })
|
||||
* ```
|
||||
*/
|
||||
isFullyInitialized(): boolean {
|
||||
if (!this.initialized) return false
|
||||
|
||||
// Check if storage has background init methods (cloud storage adapters)
|
||||
const storage = this.storage as BaseStorage & StorageBackgroundInitHooks
|
||||
if (typeof storage?.isBackgroundInitComplete === 'function') {
|
||||
return storage.isBackgroundInitComplete()
|
||||
}
|
||||
|
||||
// Non-cloud storage adapters are fully initialized after init()
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for all background initialization tasks to complete
|
||||
*
|
||||
* For cloud storage adapters with progressive initialization,
|
||||
* this waits for:
|
||||
* - Bucket/container validation
|
||||
* - Count synchronization
|
||||
* - Any other background tasks
|
||||
*
|
||||
* For non-cloud storage, this resolves immediately.
|
||||
*
|
||||
* **Use Case**: Call this when you need guaranteed consistency, such as:
|
||||
* - Before running batch operations
|
||||
* - Before reporting full system health
|
||||
* - When transitioning from "initializing" to "ready" status
|
||||
*
|
||||
* @returns Promise that resolves when all background tasks complete
|
||||
*
|
||||
* @example Ensuring full initialization
|
||||
* ```typescript
|
||||
* const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||
* await brain.init() // Fast return in cloud (<200ms)
|
||||
*
|
||||
* // Optional: wait for background tasks if needed
|
||||
* await brain.awaitBackgroundInit()
|
||||
* console.log('All background tasks complete')
|
||||
* ```
|
||||
*/
|
||||
async awaitBackgroundInit(): Promise<void> {
|
||||
// Must be initialized first
|
||||
await this.ready
|
||||
|
||||
// Check if storage has background init methods (cloud storage adapters)
|
||||
const storage = this.storage as BaseStorage & StorageBackgroundInitHooks
|
||||
if (typeof storage?.awaitBackgroundInit === 'function') {
|
||||
await storage.awaitBackgroundInit()
|
||||
}
|
||||
|
||||
// Non-cloud storage: no background init to wait for
|
||||
}
|
||||
|
||||
// ============= CORE CRUD OPERATIONS =============
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -1,54 +0,0 @@
|
|||
/**
|
||||
* Zero-Configuration System
|
||||
* Main entry point for all auto-configuration features
|
||||
*/
|
||||
|
||||
// Model configuration (simplified - always Q8 WASM)
|
||||
export {
|
||||
getModelPrecision,
|
||||
shouldAutoDownloadModels,
|
||||
getModelPath
|
||||
} from './modelAutoConfig.js'
|
||||
|
||||
// Storage configuration
|
||||
export {
|
||||
autoDetectStorage,
|
||||
StorageType,
|
||||
StoragePreset,
|
||||
StorageConfigResult,
|
||||
logStorageConfig,
|
||||
// Backward compatibility types
|
||||
type StorageTypeString,
|
||||
type StoragePresetString
|
||||
} from './storageAutoConfig.js'
|
||||
|
||||
// Shared configuration for multi-instance
|
||||
export {
|
||||
SharedConfig,
|
||||
SharedConfigManager
|
||||
} from './sharedConfigManager.js'
|
||||
|
||||
// Main zero-config processor
|
||||
export {
|
||||
BrainyZeroConfig,
|
||||
processZeroConfig,
|
||||
createEmbeddingFunctionWithPrecision
|
||||
} from './zeroConfig.js'
|
||||
|
||||
/**
|
||||
* Main zero-config processor
|
||||
* This is what Brainy will call
|
||||
*/
|
||||
export async function applyZeroConfig(input?: string | any): Promise<any> {
|
||||
// Handle legacy config (full object) by detecting known legacy properties
|
||||
if (input && typeof input === 'object' &&
|
||||
(input.storage?.forceMemoryStorage || input.storage?.forceFileSystemStorage || input.storage?.s3Storage)) {
|
||||
// This is a legacy config object - pass through unchanged
|
||||
console.log('📦 Using legacy configuration format')
|
||||
return input
|
||||
}
|
||||
|
||||
// Process as zero-config (includes new object format)
|
||||
const { processZeroConfig } = await import('./zeroConfig.js')
|
||||
return processZeroConfig(input)
|
||||
}
|
||||
|
|
@ -1,71 +0,0 @@
|
|||
/**
|
||||
* Model Configuration
|
||||
* Brainy uses Q8 WASM embeddings - no configuration needed (zero-config)
|
||||
*/
|
||||
|
||||
import { isNode } from '../utils/environment.js'
|
||||
|
||||
interface ModelConfigResult {
|
||||
precision: 'q8'
|
||||
reason: string
|
||||
autoSelected: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Get model precision configuration
|
||||
* Always returns Q8 - the optimal balance of size and accuracy
|
||||
*/
|
||||
export function getModelPrecision(): ModelConfigResult {
|
||||
return {
|
||||
precision: 'q8',
|
||||
reason: 'Q8 WASM (23MB bundled, no downloads)',
|
||||
autoSelected: true
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if running in a serverless environment
|
||||
*/
|
||||
function isServerlessEnvironment(): boolean {
|
||||
if (!isNode()) return false
|
||||
|
||||
return !!(
|
||||
process.env.AWS_LAMBDA_FUNCTION_NAME ||
|
||||
process.env.VERCEL ||
|
||||
process.env.NETLIFY ||
|
||||
process.env.CLOUDFLARE_WORKERS ||
|
||||
process.env.FUNCTIONS_WORKER_RUNTIME ||
|
||||
process.env.K_SERVICE // Google Cloud Run
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if models need to be downloaded
|
||||
* With bundled WASM model, this is rarely needed
|
||||
*/
|
||||
export function shouldAutoDownloadModels(): boolean {
|
||||
// Model is bundled - no downloads needed in normal operation
|
||||
// This flag exists for edge cases only
|
||||
const explicitlyDisabled = process.env.BRAINY_ALLOW_REMOTE_MODELS === 'false'
|
||||
return !explicitlyDisabled
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the model path
|
||||
* With bundled WASM model, this points to the package assets
|
||||
*/
|
||||
export function getModelPath(): string {
|
||||
// Check if user explicitly set a path (for advanced users)
|
||||
if (process.env.BRAINY_MODELS_PATH) {
|
||||
return process.env.BRAINY_MODELS_PATH
|
||||
}
|
||||
|
||||
// Serverless - use /tmp for ephemeral storage
|
||||
if (isServerlessEnvironment()) {
|
||||
return '/tmp/.brainy/models'
|
||||
}
|
||||
|
||||
// Node-like runtime - use home directory for persistent storage
|
||||
const homeDir = process.env.HOME || process.env.USERPROFILE || '~'
|
||||
return `${homeDir}/.brainy/models`
|
||||
}
|
||||
|
|
@ -1,266 +0,0 @@
|
|||
/**
|
||||
* Shared Configuration Manager
|
||||
* Ensures configuration consistency across multiple instances using shared storage
|
||||
*/
|
||||
|
||||
import { StorageType } from './storageAutoConfig.js'
|
||||
import { getBrainyVersion } from '../utils/version.js'
|
||||
|
||||
export interface SharedConfig {
|
||||
// Critical parameters that MUST match across instances
|
||||
version: string
|
||||
precision: 'q8'
|
||||
dimensions: number
|
||||
hnswM: number
|
||||
hnswEfConstruction: number
|
||||
distanceFunction: string
|
||||
createdAt: string
|
||||
lastUpdated: string
|
||||
|
||||
// Informational (can differ between instances)
|
||||
instanceCount?: number
|
||||
lastAccessedBy?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Manages configuration consistency for shared storage
|
||||
*/
|
||||
export class SharedConfigManager {
|
||||
private static CONFIG_FILE = '.brainy/config.json'
|
||||
|
||||
/**
|
||||
* Load or create shared configuration
|
||||
* When connecting to existing data, this OVERRIDES auto-configuration!
|
||||
*/
|
||||
static async loadOrCreateSharedConfig(
|
||||
storage: any,
|
||||
localConfig: any
|
||||
): Promise<{ config: any; warnings: string[] }> {
|
||||
const warnings: string[] = []
|
||||
|
||||
try {
|
||||
// Check if we're using shared storage
|
||||
if (!this.isSharedStorage(localConfig.storageType)) {
|
||||
// Local storage - use local config
|
||||
return { config: localConfig, warnings: [] }
|
||||
}
|
||||
|
||||
// Try to load existing configuration from shared storage
|
||||
const existingConfig = await this.loadConfigFromStorage(storage)
|
||||
|
||||
if (existingConfig) {
|
||||
// EXISTING SHARED DATA - Must use its configuration!
|
||||
console.log('📁 Found existing shared data configuration')
|
||||
|
||||
// Check for critical mismatches
|
||||
const mismatches = this.checkCriticalMismatches(localConfig, existingConfig)
|
||||
|
||||
if (mismatches.length > 0) {
|
||||
console.warn('⚠️ Configuration override required for shared storage:')
|
||||
mismatches.forEach(m => {
|
||||
console.warn(` - ${m.param}: ${m.local} → ${m.shared} (using shared)`)
|
||||
warnings.push(`${m.param} overridden: ${m.local} → ${m.shared}`)
|
||||
})
|
||||
}
|
||||
|
||||
// Override critical parameters with shared values
|
||||
const mergedConfig = this.mergeWithSharedConfig(localConfig, existingConfig)
|
||||
|
||||
// Update last accessed
|
||||
await this.updateAccessInfo(storage, existingConfig)
|
||||
|
||||
return { config: mergedConfig, warnings }
|
||||
|
||||
} else {
|
||||
// NEW SHARED STORAGE - Save our configuration
|
||||
console.log('📝 Initializing new shared storage with configuration')
|
||||
|
||||
const sharedConfig = this.createSharedConfig(localConfig)
|
||||
await this.saveConfigToStorage(storage, sharedConfig)
|
||||
|
||||
return { config: localConfig, warnings: [] }
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('Failed to manage shared configuration:', error)
|
||||
warnings.push('Could not verify shared configuration - proceeding with caution')
|
||||
return { config: localConfig, warnings }
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if storage type is shared (multi-instance)
|
||||
*/
|
||||
private static isSharedStorage(storageType: StorageType): boolean {
|
||||
return ['s3', 'gcs', 'r2'].includes(storageType)
|
||||
}
|
||||
|
||||
/**
|
||||
* Load configuration from shared storage
|
||||
*/
|
||||
private static async loadConfigFromStorage(storage: any): Promise<SharedConfig | null> {
|
||||
try {
|
||||
const configData = await storage.get(this.CONFIG_FILE)
|
||||
if (!configData) return null
|
||||
|
||||
const config = JSON.parse(configData)
|
||||
return config as SharedConfig
|
||||
} catch (error) {
|
||||
// Config doesn't exist yet
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Save configuration to shared storage
|
||||
*/
|
||||
private static async saveConfigToStorage(storage: any, config: SharedConfig): Promise<void> {
|
||||
try {
|
||||
await storage.set(this.CONFIG_FILE, JSON.stringify(config, null, 2))
|
||||
} catch (error) {
|
||||
console.error('Failed to save shared configuration:', error)
|
||||
throw new Error('Cannot initialize shared storage without saving configuration')
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create shared configuration from local config
|
||||
*/
|
||||
private static createSharedConfig(localConfig: any): SharedConfig {
|
||||
return {
|
||||
version: getBrainyVersion(), // Brainy version
|
||||
precision: localConfig.embeddingOptions?.precision || 'fp32',
|
||||
dimensions: 384, // Fixed for all-MiniLM-L6-v2
|
||||
hnswM: localConfig.hnsw?.M || 16,
|
||||
hnswEfConstruction: localConfig.hnsw?.efConstruction || 200,
|
||||
distanceFunction: localConfig.distanceFunction || 'cosine',
|
||||
createdAt: new Date().toISOString(),
|
||||
lastUpdated: new Date().toISOString(),
|
||||
instanceCount: 1,
|
||||
lastAccessedBy: this.getInstanceIdentifier()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check for critical configuration mismatches
|
||||
*/
|
||||
private static checkCriticalMismatches(
|
||||
localConfig: any,
|
||||
sharedConfig: SharedConfig
|
||||
): Array<{ param: string; local: any; shared: any }> {
|
||||
const mismatches: Array<{ param: string; local: any; shared: any }> = []
|
||||
|
||||
// Model precision - CRITICAL!
|
||||
const localPrecision = localConfig.embeddingOptions?.precision || 'fp32'
|
||||
if (localPrecision !== sharedConfig.precision) {
|
||||
mismatches.push({
|
||||
param: 'Model Precision',
|
||||
local: localPrecision,
|
||||
shared: sharedConfig.precision
|
||||
})
|
||||
}
|
||||
|
||||
// HNSW parameters - Important for index consistency
|
||||
const localM = localConfig.hnsw?.M || 16
|
||||
if (localM !== sharedConfig.hnswM) {
|
||||
mismatches.push({
|
||||
param: 'HNSW M',
|
||||
local: localM,
|
||||
shared: sharedConfig.hnswM
|
||||
})
|
||||
}
|
||||
|
||||
const localEf = localConfig.hnsw?.efConstruction || 200
|
||||
if (localEf !== sharedConfig.hnswEfConstruction) {
|
||||
mismatches.push({
|
||||
param: 'HNSW efConstruction',
|
||||
local: localEf,
|
||||
shared: sharedConfig.hnswEfConstruction
|
||||
})
|
||||
}
|
||||
|
||||
// Distance function
|
||||
const localDistance = localConfig.distanceFunction || 'cosine'
|
||||
if (localDistance !== sharedConfig.distanceFunction) {
|
||||
mismatches.push({
|
||||
param: 'Distance Function',
|
||||
local: localDistance,
|
||||
shared: sharedConfig.distanceFunction
|
||||
})
|
||||
}
|
||||
|
||||
return mismatches
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge local config with shared config (shared takes precedence for critical params)
|
||||
*/
|
||||
private static mergeWithSharedConfig(localConfig: any, sharedConfig: SharedConfig): any {
|
||||
return {
|
||||
...localConfig,
|
||||
|
||||
// Override critical parameters with shared values
|
||||
embeddingOptions: {
|
||||
...localConfig.embeddingOptions,
|
||||
precision: sharedConfig.precision // MUST use shared precision!
|
||||
},
|
||||
|
||||
hnsw: {
|
||||
...localConfig.hnsw,
|
||||
M: sharedConfig.hnswM,
|
||||
efConstruction: sharedConfig.hnswEfConstruction
|
||||
},
|
||||
|
||||
distanceFunction: sharedConfig.distanceFunction,
|
||||
|
||||
// Add metadata about shared configuration
|
||||
_sharedConfig: {
|
||||
loaded: true,
|
||||
version: sharedConfig.version,
|
||||
createdAt: sharedConfig.createdAt,
|
||||
precision: sharedConfig.precision
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Update access information in shared config
|
||||
*/
|
||||
private static async updateAccessInfo(storage: any, config: SharedConfig): Promise<void> {
|
||||
try {
|
||||
config.lastUpdated = new Date().toISOString()
|
||||
config.instanceCount = (config.instanceCount || 0) + 1
|
||||
config.lastAccessedBy = this.getInstanceIdentifier()
|
||||
|
||||
await this.saveConfigToStorage(storage, config)
|
||||
} catch {
|
||||
// Non-critical - don't fail if we can't update access info
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get unique identifier for this instance
|
||||
*/
|
||||
private static getInstanceIdentifier(): string {
|
||||
if (process.env.HOSTNAME) return process.env.HOSTNAME
|
||||
if (process.env.CONTAINER_ID) return process.env.CONTAINER_ID
|
||||
if (process.env.K_SERVICE) return process.env.K_SERVICE
|
||||
if (process.env.AWS_LAMBDA_FUNCTION_NAME) return process.env.AWS_LAMBDA_FUNCTION_NAME
|
||||
|
||||
// Generate a random identifier
|
||||
return `instance-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate that a configuration is compatible with shared data
|
||||
*/
|
||||
static validateCompatibility(config1: SharedConfig, config2: SharedConfig): boolean {
|
||||
return (
|
||||
config1.precision === config2.precision &&
|
||||
config1.dimensions === config2.dimensions &&
|
||||
config1.hnswM === config2.hnswM &&
|
||||
config1.hnswEfConstruction === config2.hnswEfConstruction &&
|
||||
config1.distanceFunction === config2.distanceFunction
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,113 +0,0 @@
|
|||
/**
|
||||
* @module config/storageAutoConfig
|
||||
* @description Storage configuration auto-detection for Brainy 8.0.
|
||||
*
|
||||
* Brainy 8.0 ships **two storage adapters**:
|
||||
* - `FILESYSTEM` — persistent on-disk storage (Node, Bun, Deno).
|
||||
* - `MEMORY` — in-memory, ephemeral.
|
||||
*
|
||||
* Cloud storage adapters (GCS / S3 / R2 / Azure) and the browser-only OPFS
|
||||
* adapter were removed. Cloud backup is now an operator concern handled by
|
||||
* `gsutil` / `aws s3 cp` / `rclone` / `azcopy` against the on-disk artefact.
|
||||
*
|
||||
* This module is kept thin: a two-value enum, a tiny preset enum that mirrors
|
||||
* it, and an `autoDetectStorage()` that picks filesystem when available and
|
||||
* memory otherwise.
|
||||
*/
|
||||
|
||||
import { isNode } from '../utils/environment.js'
|
||||
|
||||
/**
|
||||
* Storage backend types shipped in Brainy 8.0.
|
||||
*/
|
||||
export enum StorageType {
|
||||
MEMORY = 'memory',
|
||||
FILESYSTEM = 'filesystem',
|
||||
}
|
||||
|
||||
/**
|
||||
* High-level storage presets. `AUTO` picks filesystem when available, memory
|
||||
* otherwise; `MEMORY` and `DISK` are explicit overrides.
|
||||
*/
|
||||
export enum StoragePreset {
|
||||
AUTO = 'auto',
|
||||
MEMORY = 'memory',
|
||||
DISK = 'disk',
|
||||
}
|
||||
|
||||
export type StorageTypeString = 'memory' | 'filesystem'
|
||||
export type StoragePresetString = 'auto' | 'memory' | 'disk'
|
||||
|
||||
export interface StorageConfigResult {
|
||||
type: StorageType | StorageTypeString
|
||||
config: { type: StorageType | StorageTypeString; rootDirectory?: string }
|
||||
reason: string
|
||||
autoSelected: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a storage configuration. Accepts a {@link StorageType},
|
||||
* {@link StoragePreset}, plain string equivalent, or a free-form storage
|
||||
* config object. Returns a normalized result with the picked type, the
|
||||
* underlying config payload, a human-readable reason, and whether the
|
||||
* pick was auto-selected.
|
||||
*/
|
||||
export async function autoDetectStorage(
|
||||
override?: StorageType | StoragePreset | StorageTypeString | StoragePresetString | any
|
||||
): Promise<StorageConfigResult> {
|
||||
if (override && typeof override === 'object') {
|
||||
return {
|
||||
type: (override.type as StorageType) ?? StorageType.MEMORY,
|
||||
config: override,
|
||||
reason: 'Manually configured storage',
|
||||
autoSelected: false,
|
||||
}
|
||||
}
|
||||
|
||||
if (override === StorageType.MEMORY || override === StoragePreset.MEMORY || override === 'memory') {
|
||||
return {
|
||||
type: StorageType.MEMORY,
|
||||
config: { type: StorageType.MEMORY },
|
||||
reason: 'Explicit memory storage',
|
||||
autoSelected: false,
|
||||
}
|
||||
}
|
||||
|
||||
if (override === StorageType.FILESYSTEM || override === StoragePreset.DISK || override === 'filesystem' || override === 'disk') {
|
||||
return {
|
||||
type: StorageType.FILESYSTEM,
|
||||
config: { type: StorageType.FILESYSTEM, rootDirectory: './brainy-data' },
|
||||
reason: 'Explicit filesystem storage',
|
||||
autoSelected: false,
|
||||
}
|
||||
}
|
||||
|
||||
// Default: filesystem on Node-like runtimes, memory in browsers.
|
||||
if (isNode()) {
|
||||
return {
|
||||
type: StorageType.FILESYSTEM,
|
||||
config: { type: StorageType.FILESYSTEM, rootDirectory: './brainy-data' },
|
||||
reason: 'Auto-selected: filesystem (Node-like runtime detected)',
|
||||
autoSelected: true,
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
type: StorageType.MEMORY,
|
||||
config: { type: StorageType.MEMORY },
|
||||
reason: 'Auto-selected: memory (no Node-like runtime detected)',
|
||||
autoSelected: true,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Log a human-readable summary of a resolved storage config. Used by the
|
||||
* zero-config init path for transparency.
|
||||
*/
|
||||
export function logStorageConfig(config: StorageConfigResult, verbose: boolean = false): void {
|
||||
const prefix = config.autoSelected ? '[brainy] auto-selected' : '[brainy] using'
|
||||
console.log(`${prefix} storage: ${config.type} — ${config.reason}`)
|
||||
if (verbose) {
|
||||
console.log('[brainy] storage config:', JSON.stringify(config.config))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,354 +0,0 @@
|
|||
/**
|
||||
* Zero-Configuration System for Brainy
|
||||
* Provides intelligent defaults while preserving full control
|
||||
*/
|
||||
|
||||
import { getModelPrecision, getModelPath, shouldAutoDownloadModels } from './modelAutoConfig.js'
|
||||
import { autoDetectStorage, StorageType, StoragePreset } from './storageAutoConfig.js'
|
||||
import { AutoConfiguration } from '../utils/autoConfiguration.js'
|
||||
|
||||
/**
|
||||
* Simplified configuration interface
|
||||
* Everything is optional - zero config by default!
|
||||
*/
|
||||
export interface BrainyZeroConfig {
|
||||
/**
|
||||
* Configuration preset for common scenarios
|
||||
* - 'production': Optimized for production (disk storage, auto model, default features)
|
||||
* - 'development': Optimized for development (memory storage, fp32, verbose logging)
|
||||
* - 'minimal': Minimal footprint (memory storage, q8, minimal features)
|
||||
* - 'zero': True zero config (all auto-detected)
|
||||
* - 'writer': Writer process (read + write) — the single mutating process in
|
||||
* Brainy's multi-process model
|
||||
* - 'reader': Reader process (read-only, all mutations throw) — any number of
|
||||
* these may run against the same on-disk store as the writer
|
||||
*/
|
||||
mode?: 'production' | 'development' | 'minimal' | 'zero' | 'writer' | 'reader'
|
||||
|
||||
/**
|
||||
* Storage configuration
|
||||
* - 'memory': In-memory only (no persistence)
|
||||
* - 'disk': Local disk (filesystem or OPFS)
|
||||
* - 'cloud': Cloud storage (S3/GCS/R2 if configured)
|
||||
* - 'auto': Auto-detect best option (default)
|
||||
* - Object: Custom storage configuration
|
||||
*/
|
||||
storage?: StorageType | StoragePreset | any
|
||||
|
||||
/**
|
||||
* Feature set configuration
|
||||
* - 'minimal': Core features only (fastest startup)
|
||||
* - 'default': Standard features (balanced)
|
||||
* - 'full': All features enabled (most capable)
|
||||
* - Array: Specific features to enable
|
||||
*/
|
||||
features?: 'minimal' | 'default' | 'full' | string[]
|
||||
|
||||
/**
|
||||
* Logging verbosity
|
||||
* - true: Show configuration decisions and progress
|
||||
* - false: Silent operation (default in production)
|
||||
*/
|
||||
verbose?: boolean
|
||||
|
||||
/**
|
||||
* Advanced configuration (escape hatch for power users)
|
||||
* Any additional configuration can be passed here
|
||||
*/
|
||||
advanced?: any
|
||||
}
|
||||
|
||||
/**
|
||||
* Configuration presets for common scenarios
|
||||
*/
|
||||
const PRESETS = {
|
||||
production: {
|
||||
storage: 'disk' as const,
|
||||
features: 'default' as const,
|
||||
verbose: false
|
||||
},
|
||||
development: {
|
||||
storage: 'memory' as const,
|
||||
features: 'full' as const,
|
||||
verbose: true
|
||||
},
|
||||
minimal: {
|
||||
storage: 'memory' as const,
|
||||
features: 'minimal' as const,
|
||||
verbose: false
|
||||
},
|
||||
zero: {
|
||||
storage: 'auto' as const,
|
||||
features: 'default' as const,
|
||||
verbose: false
|
||||
},
|
||||
writer: {
|
||||
storage: 'auto' as const,
|
||||
features: 'minimal' as const,
|
||||
verbose: false,
|
||||
// The single mutating process: HybridMode (read + write).
|
||||
mode: 'writer' as const
|
||||
},
|
||||
reader: {
|
||||
storage: 'auto' as const,
|
||||
features: 'default' as const,
|
||||
verbose: false,
|
||||
// Read-only process: ReaderMode — every mutation throws.
|
||||
mode: 'reader' as const
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Feature sets configuration
|
||||
*/
|
||||
const FEATURE_SETS = {
|
||||
minimal: [
|
||||
'core',
|
||||
'search',
|
||||
'storage'
|
||||
],
|
||||
default: [
|
||||
'core',
|
||||
'search',
|
||||
'storage',
|
||||
'cache',
|
||||
'metadata-index',
|
||||
'batch-processing',
|
||||
'entity-registry',
|
||||
'request-deduplicator'
|
||||
],
|
||||
full: [
|
||||
'core',
|
||||
'search',
|
||||
'storage',
|
||||
'cache',
|
||||
'metadata-index',
|
||||
'batch-processing',
|
||||
'entity-registry',
|
||||
'request-deduplicator',
|
||||
'connection-pool',
|
||||
'wal',
|
||||
'monitoring',
|
||||
'metrics',
|
||||
'intelligent-verb-scoring',
|
||||
'triple-intelligence',
|
||||
'neural-api'
|
||||
]
|
||||
}
|
||||
|
||||
/**
|
||||
* Process zero-config input into full configuration
|
||||
*/
|
||||
export async function processZeroConfig(input?: string | BrainyZeroConfig): Promise<any> {
|
||||
let config: BrainyZeroConfig = {}
|
||||
|
||||
// Handle string shorthand (preset name)
|
||||
if (typeof input === 'string') {
|
||||
if (input in PRESETS) {
|
||||
// The `in` guard above proved the string is a preset key, which is
|
||||
// exactly the BrainyZeroConfig mode union.
|
||||
config = { mode: input as keyof typeof PRESETS }
|
||||
} else {
|
||||
throw new Error(`Unknown preset: ${input}. Valid presets: ${Object.keys(PRESETS).join(', ')}`)
|
||||
}
|
||||
} else if (input) {
|
||||
config = input
|
||||
}
|
||||
|
||||
// Apply preset if specified
|
||||
if (config.mode && config.mode in PRESETS) {
|
||||
const preset = PRESETS[config.mode]
|
||||
config = {
|
||||
...preset,
|
||||
...config,
|
||||
// Preserve explicit overrides
|
||||
storage: config.storage ?? preset.storage,
|
||||
features: config.features ?? preset.features,
|
||||
verbose: config.verbose ?? preset.verbose
|
||||
}
|
||||
}
|
||||
|
||||
// Auto-detect environment if not in preset mode
|
||||
const environment = detectEnvironmentMode()
|
||||
|
||||
// Get model configuration (always Q8 WASM)
|
||||
const modelConfig = getModelPrecision()
|
||||
|
||||
// Process storage configuration
|
||||
const storageConfig = await autoDetectStorage(config.storage)
|
||||
|
||||
// Process features configuration
|
||||
const features = processFeatures(config.features)
|
||||
|
||||
// Get auto-configuration recommendations. Brainy 8.0 ships filesystem +
|
||||
// memory only; there is no cloud-storage tier.
|
||||
const autoConfig = await AutoConfiguration.getInstance().detectAndConfigure({
|
||||
expectedDataSize: estimateDataSize(environment),
|
||||
s3Available: false,
|
||||
memoryBudget: undefined // Let it auto-detect
|
||||
})
|
||||
|
||||
// Determine verbosity
|
||||
const verbose = config.verbose ?? (process.env.NODE_ENV === 'development')
|
||||
|
||||
// Log configuration decisions if verbose
|
||||
if (verbose) {
|
||||
logConfigurationSummary({
|
||||
mode: config.mode || 'auto',
|
||||
model: modelConfig,
|
||||
storage: storageConfig,
|
||||
features: features,
|
||||
environment: environment,
|
||||
autoConfig: autoConfig
|
||||
})
|
||||
}
|
||||
|
||||
// Build final configuration
|
||||
const finalConfig: any = {
|
||||
// Model configuration
|
||||
embeddingFunction: undefined, // Will be created with correct precision
|
||||
embeddingOptions: {
|
||||
precision: modelConfig.precision,
|
||||
modelPath: getModelPath(),
|
||||
allowRemoteDownload: shouldAutoDownloadModels()
|
||||
},
|
||||
|
||||
// Storage configuration
|
||||
storage: storageConfig.config,
|
||||
storageType: storageConfig.type,
|
||||
|
||||
// HNSW configuration from auto-config
|
||||
hnsw: {
|
||||
M: autoConfig.recommendedConfig.enablePartitioning ? 32 : 16,
|
||||
efConstruction: autoConfig.recommendedConfig.enablePartitioning ? 400 : 200,
|
||||
maxDatasetSize: autoConfig.recommendedConfig.expectedDatasetSize,
|
||||
partitioning: autoConfig.recommendedConfig.enablePartitioning,
|
||||
maxNodesPerPartition: autoConfig.recommendedConfig.maxNodesPerPartition
|
||||
},
|
||||
|
||||
// Cache configuration from auto-config
|
||||
cache: {
|
||||
autoTune: true,
|
||||
hotCacheMaxSize: Math.floor(autoConfig.recommendedConfig.maxMemoryUsage / (1024 * 1024 * 10)), // 10% of memory budget
|
||||
batchSize: autoConfig.recommendedConfig.enablePartitioning ? 100 : 50
|
||||
},
|
||||
|
||||
// Features configuration
|
||||
enabledFeatures: features,
|
||||
|
||||
// Metadata index configuration
|
||||
metadataIndex: features.includes('metadata-index') ? {
|
||||
enabled: true,
|
||||
autoRebuild: true
|
||||
} : undefined,
|
||||
|
||||
// Intelligent verb scoring
|
||||
intelligentVerbScoring: features.includes('intelligent-verb-scoring') ? {
|
||||
enabled: true
|
||||
} : undefined,
|
||||
|
||||
// Logging configuration
|
||||
logging: {
|
||||
verbose: verbose
|
||||
},
|
||||
|
||||
// Performance flags from auto-config
|
||||
optimizations: autoConfig.optimizationFlags,
|
||||
|
||||
// Advanced overrides (if any)
|
||||
...config.advanced
|
||||
}
|
||||
|
||||
// Multi-process role: writer (read + write) or reader (read-only). This is the
|
||||
// only signal Brainy needs — `mode` drives the operational mode (HybridMode vs
|
||||
// ReaderMode) that gates every mutation path.
|
||||
if (config.mode === 'writer' || config.mode === 'reader') {
|
||||
finalConfig.mode = config.mode
|
||||
|
||||
if (verbose) {
|
||||
console.log(`📡 Process role: ${config.mode.toUpperCase()}`)
|
||||
}
|
||||
}
|
||||
|
||||
return finalConfig
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect environment mode if not specified
|
||||
*/
|
||||
function detectEnvironmentMode(): 'production' | 'development' | 'unknown' {
|
||||
if (process.env.NODE_ENV === 'production') return 'production'
|
||||
if (process.env.NODE_ENV === 'development') return 'development'
|
||||
if (process.env.NODE_ENV === 'test') return 'development'
|
||||
|
||||
// Check for CI environments
|
||||
if (process.env.CI || process.env.GITHUB_ACTIONS) return 'production'
|
||||
|
||||
// Check for production indicators
|
||||
if (process.env.VERCEL_ENV === 'production' ||
|
||||
process.env.NETLIFY_ENV === 'production' ||
|
||||
process.env.RAILWAY_ENVIRONMENT === 'production') {
|
||||
return 'production'
|
||||
}
|
||||
|
||||
return 'unknown'
|
||||
}
|
||||
|
||||
/**
|
||||
* Process features configuration
|
||||
*/
|
||||
function processFeatures(features?: 'minimal' | 'default' | 'full' | string[]): string[] {
|
||||
if (Array.isArray(features)) {
|
||||
return features
|
||||
}
|
||||
|
||||
if (features && features in FEATURE_SETS) {
|
||||
return FEATURE_SETS[features]
|
||||
}
|
||||
|
||||
// Default based on environment
|
||||
const env = detectEnvironmentMode()
|
||||
if (env === 'production') return FEATURE_SETS.default
|
||||
if (env === 'development') return FEATURE_SETS.full
|
||||
return FEATURE_SETS.default
|
||||
}
|
||||
|
||||
/**
|
||||
* Estimate dataset size based on environment
|
||||
*/
|
||||
function estimateDataSize(environment: string): number {
|
||||
switch (environment) {
|
||||
case 'production': return 100000
|
||||
case 'development': return 10000
|
||||
default: return 50000
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Log configuration summary
|
||||
*/
|
||||
function logConfigurationSummary(config: any): void {
|
||||
console.log('\n🧠 Brainy Zero-Config Summary')
|
||||
console.log('================================')
|
||||
console.log(`Mode: ${config.mode}`)
|
||||
console.log(`Environment: ${config.environment}`)
|
||||
console.log(`Model: ${config.model.precision.toUpperCase()} (${config.model.reason})`)
|
||||
console.log(`Storage: ${config.storage.type.toUpperCase()} (${config.storage.reason})`)
|
||||
console.log(`Features: ${config.features.length} enabled`)
|
||||
console.log(`Memory Budget: ${Math.floor(config.autoConfig.recommendedConfig.maxMemoryUsage / (1024 * 1024))}MB`)
|
||||
console.log(`Expected Dataset: ${config.autoConfig.recommendedConfig.expectedDatasetSize.toLocaleString()} items`)
|
||||
console.log('================================\n')
|
||||
}
|
||||
|
||||
/**
|
||||
* Create embedding function (always Q8 WASM)
|
||||
*/
|
||||
export async function createEmbeddingFunctionWithPrecision(): Promise<any> {
|
||||
const { createEmbeddingFunction } = await import('../utils/embedding.js')
|
||||
|
||||
// Create embedding function - always Q8 WASM
|
||||
return createEmbeddingFunction({
|
||||
precision: 'q8',
|
||||
verbose: false // Silent by default in zero-config
|
||||
})
|
||||
}
|
||||
|
|
@ -77,11 +77,6 @@ export type {
|
|||
// Export Aggregation Engine
|
||||
export { AggregationIndex, AggregateMaterializer, bucketTimestamp, parseBucketRange } from './aggregation/index.js'
|
||||
|
||||
// Export zero-configuration input type
|
||||
export {
|
||||
BrainyZeroConfig
|
||||
} from './config/index.js'
|
||||
|
||||
// Export Neural Import (AI data understanding)
|
||||
export { NeuralImport } from './neural/neuralImport.js'
|
||||
export type {
|
||||
|
|
|
|||
|
|
@ -18,44 +18,6 @@ import { StorageBatchConfig } from '../baseStorage.js'
|
|||
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
|
||||
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
|
||||
|
||||
// =============================================================================
|
||||
// Progressive Initialization Types
|
||||
// =============================================================================
|
||||
|
||||
/**
|
||||
* Initialization mode for cloud storage adapters.
|
||||
*
|
||||
* Controls how storage adapters initialize during startup, enabling
|
||||
* fast cold starts in serverless environments while maintaining strict
|
||||
* validation for local development.
|
||||
*
|
||||
*
|
||||
* | Mode | Description | Use Case |
|
||||
* |------|-------------|----------|
|
||||
* | `'auto'` | Auto-detect environment (progressive in cloud, strict locally) | Default, zero-config |
|
||||
* | `'progressive'` | Fast init (<200ms), validate lazily on first write | Serverless, Cloud Run |
|
||||
* | `'strict'` | Blocking validation during init (traditional behavior) | Local dev, testing |
|
||||
*
|
||||
* @example
|
||||
* ```typescript
|
||||
* // Zero-config: auto-detects Cloud Run, Lambda, etc.
|
||||
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
|
||||
*
|
||||
* // Force progressive mode for all environments
|
||||
* const storage = new FileSystemStorage({
|
||||
* bucket: 'my-bucket',
|
||||
* initMode: 'progressive'
|
||||
* })
|
||||
*
|
||||
* // Force strict validation (useful for testing)
|
||||
* const storage = new FileSystemStorage({
|
||||
* bucket: 'my-bucket',
|
||||
* initMode: 'strict'
|
||||
* })
|
||||
* ```
|
||||
*/
|
||||
export type InitMode = 'progressive' | 'strict' | 'auto'
|
||||
|
||||
/**
|
||||
* Base class for storage adapters that implements statistics tracking
|
||||
*/
|
||||
|
|
@ -372,71 +334,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
status: 'normal' | 'throttled' | 'recovering'
|
||||
}> = new Map()
|
||||
|
||||
// =============================================
|
||||
// Progressive Initialization State
|
||||
// =============================================
|
||||
// These properties enable fast cold starts in cloud environments
|
||||
// by deferring validation and count loading to background tasks.
|
||||
|
||||
/**
|
||||
* Initialization mode for this storage adapter.
|
||||
*
|
||||
* - `'auto'` (default): Progressive in cloud environments, strict locally
|
||||
* - `'progressive'`: Always use fast init with lazy validation
|
||||
* - `'strict'`: Always validate during init (traditional behavior)
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected initMode: InitMode = 'auto'
|
||||
|
||||
/**
|
||||
* Whether the bucket/container has been validated to exist.
|
||||
*
|
||||
* In progressive mode, this starts as `false` and is set to `true`
|
||||
* after the first successful write operation or background validation.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected bucketValidated = false
|
||||
|
||||
/**
|
||||
* Error from bucket validation, if any.
|
||||
*
|
||||
* Stored here so subsequent operations can fail fast with the same error.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected bucketValidationError: Error | null = null
|
||||
|
||||
/**
|
||||
* Whether counts have been loaded from storage.
|
||||
*
|
||||
* In progressive mode, counts start at 0 and are loaded in the background.
|
||||
* Operations work immediately; counts become accurate after background load.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected countsLoaded = false
|
||||
|
||||
/**
|
||||
* Whether all background initialization tasks have completed.
|
||||
*
|
||||
* Useful for tests and diagnostics to ensure full initialization.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected backgroundTasksComplete = false
|
||||
|
||||
/**
|
||||
* Promise that resolves when background initialization completes.
|
||||
*
|
||||
* Can be awaited by callers who need to ensure full initialization
|
||||
* before proceeding (e.g., in tests).
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected backgroundInitPromise: Promise<void> | null = null
|
||||
|
||||
// Statistics-specific methods that must be implemented by subclasses
|
||||
protected abstract saveStatisticsData(
|
||||
statistics: StatisticsData
|
||||
|
|
@ -1129,18 +1026,11 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL
|
||||
|
||||
// =============================================
|
||||
// Smart Count Batching
|
||||
// Count Persistence
|
||||
// =============================================
|
||||
|
||||
// Count batching state - mirrors statistics batching pattern
|
||||
protected pendingCountPersist = false // Counts changed since last persist?
|
||||
protected lastCountPersistTime = 0 // Timestamp of last persist
|
||||
protected scheduledCountPersistTimeout: NodeJS.Timeout | null = null // Scheduled persist timer
|
||||
protected pendingCountOperations = 0 // Operations since last persist
|
||||
|
||||
// Batching configuration (overridable by subclasses for custom strategies)
|
||||
protected countPersistBatchSize = 10 // Operations before forcing persist (cloud storage)
|
||||
protected countPersistInterval = 5000 // Milliseconds before forcing persist (cloud storage)
|
||||
// Counts changed since the last persist? Drives the write-through flush.
|
||||
protected pendingCountPersist = false
|
||||
|
||||
/**
|
||||
* Get total noun count - O(1) operation
|
||||
|
|
@ -1295,230 +1185,23 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
// =============================================
|
||||
|
||||
/**
|
||||
* Detect if this storage adapter uses cloud storage (network I/O)
|
||||
* Cloud storage benefits from batching; local storage does not.
|
||||
* Persist counts immediately.
|
||||
*
|
||||
* Override this method in subclasses for accurate detection.
|
||||
* Default implementation checks storage type from getStorageStatus().
|
||||
*
|
||||
* @returns true if cloud storage (GCS, S3, R2), false if local (File, Memory)
|
||||
*/
|
||||
protected isCloudStorage(): boolean {
|
||||
// Default: assume local storage (conservative, prefers reliability over performance)
|
||||
// Subclasses should override this for accurate detection
|
||||
return false
|
||||
}
|
||||
|
||||
// =============================================
|
||||
// Progressive Initialization Methods
|
||||
// =============================================
|
||||
|
||||
/**
|
||||
* Detect if running in a cloud/serverless environment.
|
||||
*
|
||||
* Cloud environments benefit from progressive initialization to minimize
|
||||
* cold start times. This method checks for common environment variables
|
||||
* set by cloud providers.
|
||||
*
|
||||
* | Provider | Environment Variable |
|
||||
* |----------|---------------------|
|
||||
* | Cloud Run | `K_SERVICE`, `K_REVISION` |
|
||||
* | Lambda | `AWS_LAMBDA_FUNCTION_NAME` |
|
||||
* | Cloud Functions | `FUNCTIONS_TARGET` |
|
||||
* | Azure Functions | `AZURE_FUNCTIONS_ENVIRONMENT` |
|
||||
*
|
||||
* @returns `true` if running in a detected cloud environment
|
||||
* @protected
|
||||
*/
|
||||
protected detectCloudEnvironment(): boolean {
|
||||
return !!(
|
||||
process.env.K_SERVICE || // Google Cloud Run
|
||||
process.env.K_REVISION || // Google Cloud Run (revision)
|
||||
process.env.AWS_LAMBDA_FUNCTION_NAME || // AWS Lambda
|
||||
process.env.FUNCTIONS_TARGET || // Google Cloud Functions
|
||||
process.env.AZURE_FUNCTIONS_ENVIRONMENT // Azure Functions
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Determine the effective initialization mode.
|
||||
*
|
||||
* Resolves `'auto'` to either `'progressive'` or `'strict'` based on
|
||||
* the detected environment.
|
||||
*
|
||||
* @returns The resolved init mode (`'progressive'` or `'strict'`)
|
||||
* @protected
|
||||
*/
|
||||
protected resolveInitMode(): 'progressive' | 'strict' {
|
||||
if (this.initMode === 'auto') {
|
||||
return this.detectCloudEnvironment() ? 'progressive' : 'strict'
|
||||
}
|
||||
return this.initMode
|
||||
}
|
||||
|
||||
/**
|
||||
* Schedule background initialization tasks.
|
||||
*
|
||||
* This method is called in progressive mode after the adapter is marked
|
||||
* as initialized. It schedules non-blocking tasks to:
|
||||
* 1. Validate bucket/container existence
|
||||
* 2. Load counts from storage
|
||||
*
|
||||
* Override in subclasses to add storage-specific background tasks.
|
||||
* Always call `super.scheduleBackgroundInit()` first.
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected scheduleBackgroundInit(): void {
|
||||
// Create a promise that tracks all background work
|
||||
this.backgroundInitPromise = this.runBackgroundInit()
|
||||
|
||||
// Don't await - let it run in background
|
||||
this.backgroundInitPromise
|
||||
.then(() => {
|
||||
this.backgroundTasksComplete = true
|
||||
})
|
||||
.catch((error) => {
|
||||
// Log but don't throw - progressive mode is best-effort
|
||||
console.error('[Progressive Init] Background initialization failed:', error)
|
||||
this.backgroundTasksComplete = true // Mark complete even on error
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Run background initialization tasks.
|
||||
*
|
||||
* Override in subclasses to add storage-specific tasks.
|
||||
* The default implementation does nothing (for local storage adapters).
|
||||
*
|
||||
* @protected
|
||||
*/
|
||||
protected async runBackgroundInit(): Promise<void> {
|
||||
// Default implementation: nothing to do for local storage
|
||||
// Cloud storage adapters override this to:
|
||||
// 1. Validate bucket/container
|
||||
// 2. Load counts from storage
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for background initialization to complete.
|
||||
*
|
||||
* Useful in tests or when you need to ensure all background tasks
|
||||
* have finished before proceeding.
|
||||
*
|
||||
* @example
|
||||
* ```typescript
|
||||
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
|
||||
* await storage.init()
|
||||
*
|
||||
* // Wait for background validation and count loading
|
||||
* await storage.awaitBackgroundInit()
|
||||
*
|
||||
* // Now counts are accurate and bucket is validated
|
||||
* const count = await storage.getNounCount()
|
||||
* ```
|
||||
*
|
||||
* @public
|
||||
*/
|
||||
public async awaitBackgroundInit(): Promise<void> {
|
||||
if (this.backgroundInitPromise) {
|
||||
await this.backgroundInitPromise
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if background initialization has completed.
|
||||
*
|
||||
* @returns `true` if all background tasks are complete
|
||||
* @public
|
||||
*/
|
||||
public isBackgroundInitComplete(): boolean {
|
||||
return this.backgroundTasksComplete
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure bucket/container is validated before write operations.
|
||||
*
|
||||
* In progressive mode, bucket validation is deferred until the first
|
||||
* write operation. This method performs lazy validation and caches
|
||||
* the result (or error) for subsequent calls.
|
||||
*
|
||||
* Override in subclasses to implement storage-specific validation.
|
||||
* The base implementation does nothing (for local storage adapters).
|
||||
*
|
||||
* @throws Error if bucket validation fails
|
||||
* @protected
|
||||
*/
|
||||
protected async ensureValidatedForWrite(): Promise<void> {
|
||||
// If already validated, nothing to do
|
||||
if (this.bucketValidated) {
|
||||
return
|
||||
}
|
||||
|
||||
// If we have a cached validation error, throw it
|
||||
if (this.bucketValidationError) {
|
||||
throw this.bucketValidationError
|
||||
}
|
||||
|
||||
// Default implementation: no validation needed (local storage)
|
||||
// Cloud storage adapters override this to validate bucket/container
|
||||
this.bucketValidated = true
|
||||
}
|
||||
|
||||
/**
|
||||
* Schedule a smart batched persist operation.
|
||||
*
|
||||
* Strategy:
|
||||
* - Local Storage: Persist immediately (fast, no network latency)
|
||||
* - Cloud Storage: Batch persist (10 ops OR 5 seconds, whichever first)
|
||||
*
|
||||
* This mirrors the statistics batching pattern for consistency.
|
||||
* Filesystem and memory storage have no network latency, so counts are
|
||||
* written through on every change rather than batched.
|
||||
*/
|
||||
protected async scheduleCountPersist(): Promise<void> {
|
||||
// Mark counts as pending persist
|
||||
this.pendingCountPersist = true
|
||||
this.pendingCountOperations++
|
||||
|
||||
// Local storage: persist immediately (fast enough, no benefit from batching)
|
||||
if (!this.isCloudStorage()) {
|
||||
await this.flushCounts()
|
||||
return
|
||||
}
|
||||
|
||||
// Cloud storage: use smart batching
|
||||
// Persist if we've hit the batch size threshold
|
||||
if (this.pendingCountOperations >= this.countPersistBatchSize) {
|
||||
await this.flushCounts()
|
||||
return
|
||||
}
|
||||
|
||||
// Otherwise, schedule a time-based persist if not already scheduled
|
||||
if (!this.scheduledCountPersistTimeout) {
|
||||
this.scheduledCountPersistTimeout = setTimeout(() => {
|
||||
this.flushCounts().catch(error => {
|
||||
console.error('Failed to flush counts on timer:', error)
|
||||
})
|
||||
}, this.countPersistInterval)
|
||||
}
|
||||
await this.flushCounts()
|
||||
}
|
||||
|
||||
/**
|
||||
* Flush counts immediately to storage.
|
||||
* Flush pending counts to storage.
|
||||
*
|
||||
* Used for:
|
||||
* - Graceful shutdown (SIGTERM handler)
|
||||
* - Forced persist (batch threshold reached)
|
||||
* - Local storage immediate persist
|
||||
*
|
||||
* This is the public API that shutdown hooks can call.
|
||||
* Used for graceful shutdown (SIGTERM handler) and the immediate
|
||||
* write-through path. This is the public API that shutdown hooks can call.
|
||||
*/
|
||||
async flushCounts(): Promise<void> {
|
||||
// Clear any scheduled persist
|
||||
if (this.scheduledCountPersistTimeout) {
|
||||
clearTimeout(this.scheduledCountPersistTimeout)
|
||||
this.scheduledCountPersistTimeout = null
|
||||
}
|
||||
|
||||
// Nothing to flush?
|
||||
if (!this.pendingCountPersist) {
|
||||
return
|
||||
|
|
@ -1527,13 +1210,9 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
|||
try {
|
||||
// Persist to storage (implemented by subclass)
|
||||
await this.persistCounts()
|
||||
|
||||
// Update state
|
||||
this.lastCountPersistTime = Date.now()
|
||||
this.pendingCountPersist = false
|
||||
this.pendingCountOperations = 0
|
||||
} catch (error) {
|
||||
console.error('❌ CRITICAL: Failed to flush counts to storage:', error)
|
||||
console.error('CRITICAL: Failed to flush counts to storage:', error)
|
||||
// Keep pending flag set so we retry on next operation
|
||||
throw error
|
||||
}
|
||||
|
|
|
|||
|
|
@ -29,17 +29,6 @@ export interface BrainyInterface<T = unknown> {
|
|||
*/
|
||||
readonly isInitialized: boolean
|
||||
|
||||
/**
|
||||
* Check if all initialization including background tasks is complete
|
||||
*/
|
||||
isFullyInitialized(): boolean
|
||||
|
||||
/**
|
||||
* Wait for all background initialization tasks to complete
|
||||
* For cloud storage adapters, this waits for bucket validation and count sync.
|
||||
*/
|
||||
awaitBackgroundInit(): Promise<void>
|
||||
|
||||
/**
|
||||
* Modern add method - unified entity creation
|
||||
* @param params Parameters for adding entities
|
||||
|
|
|
|||
|
|
@ -1,428 +0,0 @@
|
|||
/**
|
||||
* Automatic Configuration System for Brainy Vector Database
|
||||
* Detects environment, resources, and data patterns to provide optimal settings
|
||||
*/
|
||||
|
||||
import { isNode } from './environment.js'
|
||||
|
||||
export interface AutoConfigResult {
|
||||
// Environment details
|
||||
environment: 'nodejs' | 'serverless' | 'unknown'
|
||||
|
||||
// Resource detection
|
||||
availableMemory: number // bytes
|
||||
cpuCores: number
|
||||
|
||||
|
||||
// Storage capabilities
|
||||
persistentStorageAvailable: boolean
|
||||
s3StorageDetected: boolean
|
||||
|
||||
// Recommended configuration
|
||||
recommendedConfig: {
|
||||
expectedDatasetSize: number
|
||||
maxMemoryUsage: number
|
||||
targetSearchLatency: number
|
||||
enablePartitioning: boolean
|
||||
enableCompression: boolean
|
||||
enablePredictiveCaching: boolean
|
||||
partitionStrategy: 'semantic' | 'hash'
|
||||
maxNodesPerPartition: number
|
||||
semanticClusters: number
|
||||
}
|
||||
|
||||
// Performance optimization flags
|
||||
optimizationFlags: {
|
||||
useMemoryMapping: boolean
|
||||
aggressiveCaching: boolean
|
||||
backgroundOptimization: boolean
|
||||
compressionLevel: 'none' | 'light' | 'aggressive'
|
||||
}
|
||||
}
|
||||
|
||||
export interface DatasetAnalysis {
|
||||
estimatedSize: number
|
||||
vectorDimension?: number
|
||||
growthRate?: number // vectors per second
|
||||
accessPatterns?: 'read-heavy' | 'write-heavy' | 'balanced'
|
||||
}
|
||||
|
||||
/**
|
||||
* Automatic configuration system that detects environment and optimizes settings
|
||||
*/
|
||||
export class AutoConfiguration {
|
||||
private static instance: AutoConfiguration
|
||||
private cachedConfig: AutoConfigResult | null = null
|
||||
private datasetStats: DatasetAnalysis = { estimatedSize: 0 }
|
||||
|
||||
private constructor() {}
|
||||
|
||||
public static getInstance(): AutoConfiguration {
|
||||
if (!AutoConfiguration.instance) {
|
||||
AutoConfiguration.instance = new AutoConfiguration()
|
||||
}
|
||||
return AutoConfiguration.instance
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect environment and generate optimal configuration
|
||||
*/
|
||||
public async detectAndConfigure(hints?: {
|
||||
expectedDataSize?: number
|
||||
s3Available?: boolean
|
||||
memoryBudget?: number
|
||||
}): Promise<AutoConfigResult> {
|
||||
if (this.cachedConfig && !hints) {
|
||||
return this.cachedConfig
|
||||
}
|
||||
|
||||
const environment = this.detectEnvironment()
|
||||
const resources = await this.detectResources()
|
||||
const storage = await this.detectStorageCapabilities(hints?.s3Available)
|
||||
|
||||
const config: AutoConfigResult = {
|
||||
environment,
|
||||
...resources,
|
||||
...storage,
|
||||
recommendedConfig: this.generateRecommendedConfig(environment, resources, hints),
|
||||
optimizationFlags: this.generateOptimizationFlags(environment, resources)
|
||||
}
|
||||
|
||||
this.cachedConfig = config
|
||||
return config
|
||||
}
|
||||
|
||||
/**
|
||||
* Update configuration based on runtime dataset analysis
|
||||
*/
|
||||
public async adaptToDataset(analysis: DatasetAnalysis): Promise<AutoConfigResult> {
|
||||
this.datasetStats = analysis
|
||||
|
||||
// Regenerate configuration with dataset insights
|
||||
const currentConfig = await this.detectAndConfigure()
|
||||
const adaptedConfig = this.adaptConfigurationToData(currentConfig, analysis)
|
||||
|
||||
this.cachedConfig = adaptedConfig
|
||||
return adaptedConfig
|
||||
}
|
||||
|
||||
/**
|
||||
* Learn from performance metrics and adjust configuration
|
||||
*/
|
||||
public async learnFromPerformance(metrics: {
|
||||
averageSearchTime: number
|
||||
memoryUsage: number
|
||||
cacheHitRate: number
|
||||
errorRate: number
|
||||
}): Promise<Partial<AutoConfigResult['recommendedConfig']>> {
|
||||
const adjustments: Partial<AutoConfigResult['recommendedConfig']> = {}
|
||||
|
||||
// Learn from search performance
|
||||
if (metrics.averageSearchTime > 200) {
|
||||
// Too slow - optimize for speed
|
||||
adjustments.maxNodesPerPartition = Math.max(10000, (this.cachedConfig?.recommendedConfig.maxNodesPerPartition || 50000) * 0.8)
|
||||
} else if (metrics.averageSearchTime < 50) {
|
||||
// Very fast - can optimize for quality
|
||||
adjustments.maxNodesPerPartition = Math.min(100000, (this.cachedConfig?.recommendedConfig.maxNodesPerPartition || 50000) * 1.2)
|
||||
}
|
||||
|
||||
// Learn from memory usage
|
||||
if (metrics.memoryUsage > (this.cachedConfig?.recommendedConfig.maxMemoryUsage || 0) * 0.9) {
|
||||
// High memory usage - enable compression
|
||||
adjustments.enableCompression = true
|
||||
}
|
||||
|
||||
// Learn from cache performance
|
||||
if (metrics.cacheHitRate < 0.7) {
|
||||
// Poor cache performance - enable predictive caching
|
||||
adjustments.enablePredictiveCaching = true
|
||||
}
|
||||
|
||||
// Update cached config with learned adjustments
|
||||
if (this.cachedConfig) {
|
||||
this.cachedConfig.recommendedConfig = {
|
||||
...this.cachedConfig.recommendedConfig,
|
||||
...adjustments
|
||||
}
|
||||
}
|
||||
|
||||
return adjustments
|
||||
}
|
||||
|
||||
/**
|
||||
* Get minimal configuration for quick setup
|
||||
*/
|
||||
public async getQuickSetupConfig(scenario: 'small' | 'medium' | 'large' | 'enterprise'): Promise<{
|
||||
expectedDatasetSize: number
|
||||
maxMemoryUsage: number
|
||||
targetSearchLatency: number
|
||||
s3Required: boolean
|
||||
}> {
|
||||
const environment = this.detectEnvironment()
|
||||
const resources = await this.detectResources()
|
||||
|
||||
switch (scenario) {
|
||||
case 'small':
|
||||
return {
|
||||
expectedDatasetSize: 10000,
|
||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.3, 1024 * 1024 * 1024), // 1GB max
|
||||
targetSearchLatency: 100,
|
||||
s3Required: false
|
||||
}
|
||||
|
||||
case 'medium':
|
||||
return {
|
||||
expectedDatasetSize: 100000,
|
||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.5, 4 * 1024 * 1024 * 1024), // 4GB max
|
||||
targetSearchLatency: 150,
|
||||
s3Required: environment === 'serverless'
|
||||
}
|
||||
|
||||
case 'large':
|
||||
return {
|
||||
expectedDatasetSize: 1000000,
|
||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.7, 8 * 1024 * 1024 * 1024), // 8GB max
|
||||
targetSearchLatency: 200,
|
||||
s3Required: true
|
||||
}
|
||||
|
||||
case 'enterprise':
|
||||
return {
|
||||
expectedDatasetSize: 10000000,
|
||||
maxMemoryUsage: Math.min(resources.availableMemory * 0.8, 32 * 1024 * 1024 * 1024), // 32GB max
|
||||
targetSearchLatency: 300,
|
||||
s3Required: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect the current runtime environment
|
||||
*/
|
||||
private detectEnvironment(): 'nodejs' | 'serverless' | 'unknown' {
|
||||
if (isNode()) {
|
||||
// Check for serverless environment indicators
|
||||
if (process.env.AWS_LAMBDA_FUNCTION_NAME ||
|
||||
process.env.VERCEL ||
|
||||
process.env.NETLIFY ||
|
||||
process.env.CLOUDFLARE_WORKERS) {
|
||||
return 'serverless'
|
||||
}
|
||||
return 'nodejs'
|
||||
}
|
||||
|
||||
return 'unknown'
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect available system resources
|
||||
*/
|
||||
private async detectResources(): Promise<{
|
||||
availableMemory: number
|
||||
cpuCores: number
|
||||
}> {
|
||||
let availableMemory = 2 * 1024 * 1024 * 1024 // Default 2GB
|
||||
let cpuCores = 4 // Default 4 cores
|
||||
|
||||
if (isNode()) {
|
||||
try {
|
||||
const os = await import('node:os')
|
||||
availableMemory = os.totalmem() * 0.7 // Use 70% of total memory
|
||||
cpuCores = os.cpus().length
|
||||
} catch {
|
||||
// Fallback to defaults
|
||||
}
|
||||
}
|
||||
|
||||
return { availableMemory, cpuCores }
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect available storage capabilities
|
||||
*/
|
||||
private async detectStorageCapabilities(s3Hint?: boolean): Promise<{
|
||||
persistentStorageAvailable: boolean
|
||||
s3StorageDetected: boolean
|
||||
}> {
|
||||
let persistentStorageAvailable = false
|
||||
let s3StorageDetected = s3Hint || false
|
||||
|
||||
if (isNode()) {
|
||||
persistentStorageAvailable = true
|
||||
// 8.0 dropped cloud storage adapters; the s3StorageDetected flag is
|
||||
// preserved as a hint for operator backup tooling, not adapter wiring.
|
||||
s3StorageDetected = s3Hint ||
|
||||
!!(process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY) ||
|
||||
!!(process.env.S3_BUCKET_NAME)
|
||||
}
|
||||
|
||||
return {
|
||||
persistentStorageAvailable,
|
||||
s3StorageDetected
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate recommended configuration based on detected environment and resources
|
||||
*/
|
||||
private generateRecommendedConfig(
|
||||
environment: string,
|
||||
resources: { availableMemory: number; cpuCores: number },
|
||||
hints?: { expectedDataSize?: number; memoryBudget?: number }
|
||||
): AutoConfigResult['recommendedConfig'] {
|
||||
const datasetSize = hints?.expectedDataSize || this.estimateDatasetSize()
|
||||
const memoryBudget = hints?.memoryBudget || Math.floor(resources.availableMemory * 0.6)
|
||||
|
||||
// Base configuration
|
||||
let config = {
|
||||
expectedDatasetSize: datasetSize,
|
||||
maxMemoryUsage: memoryBudget,
|
||||
targetSearchLatency: 150,
|
||||
enablePartitioning: datasetSize > 25000,
|
||||
enableCompression: memoryBudget < 2 * 1024 * 1024 * 1024,
|
||||
enablePredictiveCaching: true,
|
||||
partitionStrategy: 'semantic' as const,
|
||||
maxNodesPerPartition: 50000,
|
||||
semanticClusters: 8
|
||||
}
|
||||
|
||||
// Environment-specific adjustments
|
||||
switch (environment) {
|
||||
case 'serverless':
|
||||
config = {
|
||||
...config,
|
||||
targetSearchLatency: 500, // Account for cold starts
|
||||
enablePredictiveCaching: false, // Avoid background processes
|
||||
maxNodesPerPartition: 30000 // Moderate partition size
|
||||
}
|
||||
break
|
||||
|
||||
case 'nodejs':
|
||||
config = {
|
||||
...config,
|
||||
targetSearchLatency: 100, // Aggressive for Node.js
|
||||
maxNodesPerPartition: Math.min(100000, Math.floor(datasetSize / 10)), // Larger partitions
|
||||
semanticClusters: Math.min(16, Math.max(4, Math.floor(datasetSize / 50000))) // Scale clusters with data
|
||||
}
|
||||
break
|
||||
}
|
||||
|
||||
// Dataset size adjustments
|
||||
if (datasetSize > 1000000) {
|
||||
config.semanticClusters = Math.min(32, Math.floor(datasetSize / 100000))
|
||||
config.maxNodesPerPartition = 100000
|
||||
} else if (datasetSize < 10000) {
|
||||
config.enablePartitioning = false
|
||||
config.partitionStrategy = 'semantic' // Keep semantic but disable partitioning
|
||||
}
|
||||
|
||||
return config
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate optimization flags based on environment and resources
|
||||
*/
|
||||
private generateOptimizationFlags(
|
||||
environment: string,
|
||||
resources: { availableMemory: number; cpuCores: number }
|
||||
): AutoConfigResult['optimizationFlags'] {
|
||||
return {
|
||||
useMemoryMapping: environment === 'nodejs' && resources.availableMemory > 4 * 1024 * 1024 * 1024,
|
||||
aggressiveCaching: resources.availableMemory > 2 * 1024 * 1024 * 1024,
|
||||
backgroundOptimization: environment !== 'serverless' && resources.cpuCores > 2,
|
||||
compressionLevel: resources.availableMemory < 1024 * 1024 * 1024 ? 'aggressive' :
|
||||
resources.availableMemory < 4 * 1024 * 1024 * 1024 ? 'light' : 'none'
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Adapt configuration based on actual dataset analysis
|
||||
*/
|
||||
private adaptConfigurationToData(
|
||||
baseConfig: AutoConfigResult,
|
||||
analysis: DatasetAnalysis
|
||||
): AutoConfigResult {
|
||||
const updatedConfig = { ...baseConfig }
|
||||
|
||||
// Adjust based on actual dataset size
|
||||
if (analysis.estimatedSize !== baseConfig.recommendedConfig.expectedDatasetSize) {
|
||||
const sizeRatio = analysis.estimatedSize / baseConfig.recommendedConfig.expectedDatasetSize
|
||||
|
||||
updatedConfig.recommendedConfig.expectedDatasetSize = analysis.estimatedSize
|
||||
|
||||
// Scale partition size with dataset
|
||||
if (sizeRatio > 2) {
|
||||
updatedConfig.recommendedConfig.maxNodesPerPartition = Math.min(
|
||||
100000,
|
||||
Math.floor(updatedConfig.recommendedConfig.maxNodesPerPartition * 1.5)
|
||||
)
|
||||
updatedConfig.recommendedConfig.semanticClusters = Math.min(
|
||||
32,
|
||||
Math.floor(updatedConfig.recommendedConfig.semanticClusters * 1.5)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// Adjust based on vector dimension
|
||||
if (analysis.vectorDimension) {
|
||||
if (analysis.vectorDimension > 1024) {
|
||||
// High-dimensional vectors - optimize for compression
|
||||
updatedConfig.recommendedConfig.enableCompression = true
|
||||
updatedConfig.optimizationFlags.compressionLevel = 'aggressive'
|
||||
}
|
||||
}
|
||||
|
||||
// Adjust based on access patterns
|
||||
if (analysis.accessPatterns === 'read-heavy') {
|
||||
updatedConfig.recommendedConfig.enablePredictiveCaching = true
|
||||
updatedConfig.optimizationFlags.aggressiveCaching = true
|
||||
} else if (analysis.accessPatterns === 'write-heavy') {
|
||||
updatedConfig.recommendedConfig.enablePredictiveCaching = false
|
||||
updatedConfig.optimizationFlags.backgroundOptimization = false
|
||||
}
|
||||
|
||||
return updatedConfig
|
||||
}
|
||||
|
||||
/**
|
||||
* Estimate dataset size if not provided
|
||||
*/
|
||||
private estimateDatasetSize(): number {
|
||||
// Start with conservative estimate
|
||||
const environment = this.detectEnvironment()
|
||||
|
||||
switch (environment) {
|
||||
case 'serverless': return 50000
|
||||
case 'nodejs': return 100000
|
||||
default: return 25000
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset cached configuration (for testing or manual refresh)
|
||||
*/
|
||||
public resetCache(): void {
|
||||
this.cachedConfig = null
|
||||
this.datasetStats = { estimatedSize: 0 }
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience function for quick auto-configuration
|
||||
*/
|
||||
export async function autoConfigureBrainy(hints?: {
|
||||
expectedDataSize?: number
|
||||
s3Available?: boolean
|
||||
memoryBudget?: number
|
||||
}): Promise<AutoConfigResult> {
|
||||
const autoConfig = AutoConfiguration.getInstance()
|
||||
return autoConfig.detectAndConfigure(hints)
|
||||
}
|
||||
|
||||
/**
|
||||
* Get quick setup configuration for common scenarios
|
||||
*/
|
||||
export async function getQuickSetup(scenario: 'small' | 'medium' | 'large' | 'enterprise') {
|
||||
const autoConfig = AutoConfiguration.getInstance()
|
||||
return autoConfig.getQuickSetupConfig(scenario)
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue