soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

refactor(8.0)!: remove orphaned zero-config subsystem + dead cloud/progressive-init storage vestige

Browse source 
The old config-generation subsystem (src/config/ + autoConfiguration.ts) was
superseded during the 8.0 rework and never wired into init(): it emitted settings
for a partitioning subsystem that no longer exists and probed deleted cloud env
vars. The live zero-config path is inline — recall preset → HNSW knobs, storage
auto-detect, auto persistMode, container-memory-aware cache sizing.

The storage progressive-init / cloud-detection cluster was equally dead after the
cloud adapters were dropped: isCloudStorage() is permanently false (no overriders),
scheduleBackgroundInit/runBackgroundInit were never called (the latter an empty
body), initMode was never assigned, and Brainy.isFullyInitialized()/
awaitBackgroundInit() were always-trivial with zero callers. scheduleCountPersist()
collapses to its only-ever-taken immediate write-through path.

Removed:
- src/config/{index,zeroConfig,storageAutoConfig,modelAutoConfig,sharedConfigManager}.ts
- src/utils/autoConfiguration.ts + the inert BrainyZeroConfig export
- Brainy.isFullyInitialized()/awaitBackgroundInit() (+ BrainyInterface decls)
- InitMode type, isCloudStorage/detectCloudEnvironment/resolveInitMode,
  scheduleBackgroundInit/runBackgroundInit/ensureValidatedForWrite and their state
- Dead cloud env-var probes (K_SERVICE/K_REVISION/AWS_LAMBDA_FUNCTION_NAME/
  FUNCTIONS_TARGET/AZURE_FUNCTIONS_ENVIRONMENT)

Kept (verified live): production-detection logging (environment.ts), container-
memory cache sizing (memoryDetection/paramValidation), on-disk hash bucketing
(sharding.ts).

Docs: scrubbed deleted-subsystem references (JS quantization knobs, cloud/OPFS
adapters, partitioning, old zero-config API) across 14 files; deleted two wholly-
obsolete feature docs (complete-feature-list, v3-features); rewrote
architecture/zero-config for 8.0.

~3,700 LOC removed. Build clean; 1392 unit + 24 db-mvcc green.
This commit is contained in:
David Snelling 2026-06-15 11:11:21 -07:00
parent 00d3203d68
commit 35b9d7ef43
28 changed files with 596 additions and 3752 deletions
Download patch file Download diff file Expand all files Collapse all files

343
src/storage/adapters/baseStorageAdapter.ts
View file

@ -18,44 +18,6 @@ import { StorageBatchConfig } from '../baseStorage.js'
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
// =============================================================================
// Progressive Initialization Types
// =============================================================================
/**
* Initialization mode for cloud storage adapters.
*
* Controls how storage adapters initialize during startup, enabling
* fast cold starts in serverless environments while maintaining strict
* validation for local development.
*
*
* | Mode | Description | Use Case |
* |------|-------------|----------|
* | `'auto'` | Auto-detect environment (progressive in cloud, strict locally) | Default, zero-config |
* | `'progressive'` | Fast init (<200ms), validate lazily on first write | Serverless, Cloud Run |
* | `'strict'` | Blocking validation during init (traditional behavior) | Local dev, testing |
*
* @example
* ```typescript
* // Zero-config: auto-detects Cloud Run, Lambda, etc.
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
*
* // Force progressive mode for all environments
* const storage = new FileSystemStorage({
* bucket: 'my-bucket',
* initMode: 'progressive'
* })
*
* // Force strict validation (useful for testing)
* const storage = new FileSystemStorage({
* bucket: 'my-bucket',
* initMode: 'strict'
* })
* ```
*/
export type InitMode = 'progressive' | 'strict' | 'auto'
/**
* Base class for storage adapters that implements statistics tracking
*/
@ -372,71 +334,6 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
status: 'normal' | 'throttled' | 'recovering'
}> = new Map()
// =============================================
// Progressive Initialization State
// =============================================
// These properties enable fast cold starts in cloud environments
// by deferring validation and count loading to background tasks.
/**
* Initialization mode for this storage adapter.
*
* - `'auto'` (default): Progressive in cloud environments, strict locally
* - `'progressive'`: Always use fast init with lazy validation
* - `'strict'`: Always validate during init (traditional behavior)
*
* @protected
*/
protected initMode: InitMode = 'auto'
/**
* Whether the bucket/container has been validated to exist.
*
* In progressive mode, this starts as `false` and is set to `true`
* after the first successful write operation or background validation.
*
* @protected
*/
protected bucketValidated = false
/**
* Error from bucket validation, if any.
*
* Stored here so subsequent operations can fail fast with the same error.
*
* @protected
*/
protected bucketValidationError: Error | null = null
/**
* Whether counts have been loaded from storage.
*
* In progressive mode, counts start at 0 and are loaded in the background.
* Operations work immediately; counts become accurate after background load.
*
* @protected
*/
protected countsLoaded = false
/**
* Whether all background initialization tasks have completed.
*
* Useful for tests and diagnostics to ensure full initialization.
*
* @protected
*/
protected backgroundTasksComplete = false
/**
* Promise that resolves when background initialization completes.
*
* Can be awaited by callers who need to ensure full initialization
* before proceeding (e.g., in tests).
*
* @protected
*/
protected backgroundInitPromise: Promise<void> | null = null
// Statistics-specific methods that must be implemented by subclasses
protected abstract saveStatisticsData(
statistics: StatisticsData
@ -1129,18 +1026,11 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
protected readonly COUNT_CACHE_TTL = 60000 // 1 minute cache TTL
// =============================================
// Smart Count Batching
// Count Persistence
// =============================================
// Count batching state - mirrors statistics batching pattern
protected pendingCountPersist = false // Counts changed since last persist?
protected lastCountPersistTime = 0 // Timestamp of last persist
protected scheduledCountPersistTimeout: NodeJS.Timeout | null = null // Scheduled persist timer
protected pendingCountOperations = 0 // Operations since last persist
// Batching configuration (overridable by subclasses for custom strategies)
protected countPersistBatchSize = 10 // Operations before forcing persist (cloud storage)
protected countPersistInterval = 5000 // Milliseconds before forcing persist (cloud storage)
// Counts changed since the last persist? Drives the write-through flush.
protected pendingCountPersist = false
/**
* Get total noun count - O(1) operation
@ -1295,230 +1185,23 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
// =============================================
/**
* Detect if this storage adapter uses cloud storage (network I/O)
* Cloud storage benefits from batching; local storage does not.
* Persist counts immediately.
*
* Override this method in subclasses for accurate detection.
* Default implementation checks storage type from getStorageStatus().
*
* @returns true if cloud storage (GCS, S3, R2), false if local (File, Memory)
*/
protected isCloudStorage(): boolean {
// Default: assume local storage (conservative, prefers reliability over performance)
// Subclasses should override this for accurate detection
return false
}
// =============================================
// Progressive Initialization Methods
// =============================================
/**
* Detect if running in a cloud/serverless environment.
*
* Cloud environments benefit from progressive initialization to minimize
* cold start times. This method checks for common environment variables
* set by cloud providers.
*
* | Provider | Environment Variable |
* |----------|---------------------|
* | Cloud Run | `K_SERVICE`, `K_REVISION` |
* | Lambda | `AWS_LAMBDA_FUNCTION_NAME` |
* | Cloud Functions | `FUNCTIONS_TARGET` |
* | Azure Functions | `AZURE_FUNCTIONS_ENVIRONMENT` |
*
* @returns `true` if running in a detected cloud environment
* @protected
*/
protected detectCloudEnvironment(): boolean {
return !!(
process.env.K_SERVICE || // Google Cloud Run
process.env.K_REVISION || // Google Cloud Run (revision)
process.env.AWS_LAMBDA_FUNCTION_NAME || // AWS Lambda
process.env.FUNCTIONS_TARGET || // Google Cloud Functions
process.env.AZURE_FUNCTIONS_ENVIRONMENT // Azure Functions
)
}
/**
* Determine the effective initialization mode.
*
* Resolves `'auto'` to either `'progressive'` or `'strict'` based on
* the detected environment.
*
* @returns The resolved init mode (`'progressive'` or `'strict'`)
* @protected
*/
protected resolveInitMode(): 'progressive' | 'strict' {
if (this.initMode === 'auto') {
return this.detectCloudEnvironment() ? 'progressive' : 'strict'
}
return this.initMode
}
/**
* Schedule background initialization tasks.
*
* This method is called in progressive mode after the adapter is marked
* as initialized. It schedules non-blocking tasks to:
* 1. Validate bucket/container existence
* 2. Load counts from storage
*
* Override in subclasses to add storage-specific background tasks.
* Always call `super.scheduleBackgroundInit()` first.
*
* @protected
*/
protected scheduleBackgroundInit(): void {
// Create a promise that tracks all background work
this.backgroundInitPromise = this.runBackgroundInit()
// Don't await - let it run in background
this.backgroundInitPromise
.then(() => {
this.backgroundTasksComplete = true
})
.catch((error) => {
// Log but don't throw - progressive mode is best-effort
console.error('[Progressive Init] Background initialization failed:', error)
this.backgroundTasksComplete = true // Mark complete even on error
})
}
/**
* Run background initialization tasks.
*
* Override in subclasses to add storage-specific tasks.
* The default implementation does nothing (for local storage adapters).
*
* @protected
*/
protected async runBackgroundInit(): Promise<void> {
// Default implementation: nothing to do for local storage
// Cloud storage adapters override this to:
// 1. Validate bucket/container
// 2. Load counts from storage
}
/**
* Wait for background initialization to complete.
*
* Useful in tests or when you need to ensure all background tasks
* have finished before proceeding.
*
* @example
* ```typescript
* const storage = new FileSystemStorage({ bucket: 'my-bucket' })
* await storage.init()
*
* // Wait for background validation and count loading
* await storage.awaitBackgroundInit()
*
* // Now counts are accurate and bucket is validated
* const count = await storage.getNounCount()
* ```
*
* @public
*/
public async awaitBackgroundInit(): Promise<void> {
if (this.backgroundInitPromise) {
await this.backgroundInitPromise
}
}
/**
* Check if background initialization has completed.
*
* @returns `true` if all background tasks are complete
* @public
*/
public isBackgroundInitComplete(): boolean {
return this.backgroundTasksComplete
}
/**
* Ensure bucket/container is validated before write operations.
*
* In progressive mode, bucket validation is deferred until the first
* write operation. This method performs lazy validation and caches
* the result (or error) for subsequent calls.
*
* Override in subclasses to implement storage-specific validation.
* The base implementation does nothing (for local storage adapters).
*
* @throws Error if bucket validation fails
* @protected
*/
protected async ensureValidatedForWrite(): Promise<void> {
// If already validated, nothing to do
if (this.bucketValidated) {
return
}
// If we have a cached validation error, throw it
if (this.bucketValidationError) {
throw this.bucketValidationError
}
// Default implementation: no validation needed (local storage)
// Cloud storage adapters override this to validate bucket/container
this.bucketValidated = true
}
/**
* Schedule a smart batched persist operation.
*
* Strategy:
* - Local Storage: Persist immediately (fast, no network latency)
* - Cloud Storage: Batch persist (10 ops OR 5 seconds, whichever first)
*
* This mirrors the statistics batching pattern for consistency.
* Filesystem and memory storage have no network latency, so counts are
* written through on every change rather than batched.
*/
protected async scheduleCountPersist(): Promise<void> {
// Mark counts as pending persist
this.pendingCountPersist = true
this.pendingCountOperations++
// Local storage: persist immediately (fast enough, no benefit from batching)
if (!this.isCloudStorage()) {
await this.flushCounts()
return
}
// Cloud storage: use smart batching
// Persist if we've hit the batch size threshold
if (this.pendingCountOperations >= this.countPersistBatchSize) {
await this.flushCounts()
return
}
// Otherwise, schedule a time-based persist if not already scheduled
if (!this.scheduledCountPersistTimeout) {
this.scheduledCountPersistTimeout = setTimeout(() => {
this.flushCounts().catch(error => {
console.error('Failed to flush counts on timer:', error)
})
}, this.countPersistInterval)
}
await this.flushCounts()
}
/**
* Flush counts immediately to storage.
* Flush pending counts to storage.
*
* Used for:
* - Graceful shutdown (SIGTERM handler)
* - Forced persist (batch threshold reached)
* - Local storage immediate persist
*
* This is the public API that shutdown hooks can call.
* Used for graceful shutdown (SIGTERM handler) and the immediate
* write-through path. This is the public API that shutdown hooks can call.
*/
async flushCounts(): Promise<void> {
// Clear any scheduled persist
if (this.scheduledCountPersistTimeout) {
clearTimeout(this.scheduledCountPersistTimeout)
this.scheduledCountPersistTimeout = null
}
// Nothing to flush?
if (!this.pendingCountPersist) {
return
@ -1527,13 +1210,9 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
try {
// Persist to storage (implemented by subclass)
await this.persistCounts()
// Update state
this.lastCountPersistTime = Date.now()
this.pendingCountPersist = false
this.pendingCountOperations = 0
} catch (error) {
console.error('❌ CRITICAL: Failed to flush counts to storage:', error)
console.error('CRITICAL: Failed to flush counts to storage:', error)
// Keep pending flag set so we retry on next operation
throw error
}