feat: progressive init and readiness API for cloud storage
- Add `initMode` option to all cloud storage adapters (GCS, S3, Azure) - 'auto' (default): progressive in cloud, strict locally - 'progressive': <200ms cold starts, lazy bucket validation - 'strict': blocking validation (current behavior) - Add cloud environment auto-detection for: - Cloud Run (K_SERVICE, K_REVISION) - AWS Lambda (AWS_LAMBDA_FUNCTION_NAME) - Cloud Functions (FUNCTIONS_TARGET) - Azure Functions (AZURE_FUNCTIONS_ENVIRONMENT) - Add readiness API to Brainy class: - `brain.ready`: Promise that resolves when init() completes - `brain.isFullyInitialized()`: checks if background tasks done - `brain.awaitBackgroundInit()`: waits for background tasks - Lazy bucket validation on first write (not during init) - Background count synchronization - Update AWS/GCP deployment docs with readiness patterns 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
19c5e67035
commit
d938a6bd4f
10 changed files with 1470 additions and 62 deletions
|
|
@ -20,6 +20,66 @@ const brain = new Brainy({
|
||||||
await brain.init()
|
await brain.init()
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Readiness API (v7.3.0+)
|
||||||
|
|
||||||
|
For reliable initialization detection, especially in cloud environments with progressive initialization:
|
||||||
|
|
||||||
|
### `brain.ready` - Await Initialization
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const brain = new Brainy()
|
||||||
|
brain.init() // Fire and forget
|
||||||
|
|
||||||
|
// Elsewhere (e.g., API handler)
|
||||||
|
await brain.ready // Wait until init() completes
|
||||||
|
const results = await brain.find({ query: 'test' })
|
||||||
|
```
|
||||||
|
|
||||||
|
### `brain.isInitialized` - Check Basic Readiness
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
if (brain.isInitialized) {
|
||||||
|
// Safe to use brain methods
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `brain.isFullyInitialized()` - Check Background Tasks
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Returns true when ALL initialization is complete, including background tasks
|
||||||
|
// Useful for cloud storage adapters with progressive initialization
|
||||||
|
if (brain.isFullyInitialized()) {
|
||||||
|
console.log('All background tasks complete')
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `brain.awaitBackgroundInit()` - Wait for Background Tasks
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||||
|
await brain.init() // Fast return in cloud (<200ms)
|
||||||
|
|
||||||
|
// Optional: wait for all background tasks (bucket validation, count sync)
|
||||||
|
await brain.awaitBackgroundInit()
|
||||||
|
console.log('Fully initialized including background tasks')
|
||||||
|
```
|
||||||
|
|
||||||
|
### Health Check Pattern
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
app.get('/health', async (req, res) => {
|
||||||
|
try {
|
||||||
|
await brain.ready
|
||||||
|
res.json({
|
||||||
|
status: 'ready',
|
||||||
|
fullyInitialized: brain.isFullyInitialized()
|
||||||
|
})
|
||||||
|
} catch (error) {
|
||||||
|
res.status(503).json({ status: 'initializing', error: error.message })
|
||||||
|
}
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
## Core CRUD Operations
|
## Core CRUD Operations
|
||||||
|
|
||||||
### `brain.add(params)` - Add Entity
|
### `brain.add(params)` - Add Entity
|
||||||
|
|
|
||||||
|
|
@ -386,8 +386,53 @@ jobs:
|
||||||
```
|
```
|
||||||
|
|
||||||
3. **Cold Starts (Lambda)**
|
3. **Cold Starts (Lambda)**
|
||||||
|
|
||||||
|
**v7.3.0+ Progressive Initialization (Zero-Config)**
|
||||||
|
|
||||||
|
Brainy automatically detects Lambda environments (AWS_LAMBDA_FUNCTION_NAME)
|
||||||
|
and uses progressive initialization for <200ms cold starts:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// Zero-config - Brainy auto-detects Lambda
|
||||||
|
const brain = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 's3',
|
||||||
|
s3Storage: {
|
||||||
|
bucketName: 'my-bucket',
|
||||||
|
region: 'us-east-1',
|
||||||
|
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||||
|
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
await brain.init() // Returns in <200ms
|
||||||
|
|
||||||
|
// First write validates bucket (lazy validation)
|
||||||
|
await brain.add('noun', { name: 'test' }) // Validates here
|
||||||
|
```
|
||||||
|
|
||||||
|
**Manual Override (if needed)**
|
||||||
|
```javascript
|
||||||
|
const brain = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 's3',
|
||||||
|
s3Storage: {
|
||||||
|
bucketName: 'my-bucket',
|
||||||
|
// Force specific mode
|
||||||
|
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
| Mode | Cold Start | Best For |
|
||||||
|
|------|------------|----------|
|
||||||
|
| `auto` (default) | <200ms in Lambda | Zero-config, auto-detects |
|
||||||
|
| `progressive` | <200ms always | Force fast init everywhere |
|
||||||
|
| `strict` | 100-500ms+ | Local dev, tests, debugging |
|
||||||
|
|
||||||
|
**Warm-up (Alternative)**
|
||||||
```javascript
|
```javascript
|
||||||
// Warm-up configuration
|
|
||||||
exports.warmup = async () => {
|
exports.warmup = async () => {
|
||||||
if (!brain) {
|
if (!brain) {
|
||||||
brain = new Brainy({ warmup: true })
|
brain = new Brainy({ warmup: true })
|
||||||
|
|
@ -396,6 +441,50 @@ jobs:
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Readiness Detection (v7.3.0+)**
|
||||||
|
|
||||||
|
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
let brain
|
||||||
|
|
||||||
|
exports.handler = async (event) => {
|
||||||
|
if (!brain) {
|
||||||
|
brain = new Brainy({ storage: { type: 's3', ... } })
|
||||||
|
brain.init() // Fire and forget
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait for initialization to complete
|
||||||
|
await brain.ready
|
||||||
|
|
||||||
|
// Now safe to use brain methods
|
||||||
|
const results = await brain.find({ query: event.queryStringParameters.q })
|
||||||
|
return { statusCode: 200, body: JSON.stringify(results) }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
For health checks, use `isFullyInitialized()` to verify all background tasks are complete:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
exports.healthCheck = async () => {
|
||||||
|
try {
|
||||||
|
await brain.ready
|
||||||
|
return {
|
||||||
|
statusCode: 200,
|
||||||
|
body: JSON.stringify({
|
||||||
|
status: 'ready',
|
||||||
|
fullyInitialized: brain.isFullyInitialized()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
return {
|
||||||
|
statusCode: 503,
|
||||||
|
body: JSON.stringify({ status: 'initializing' })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
## Production Checklist
|
## Production Checklist
|
||||||
|
|
||||||
- [ ] IAM roles configured with minimal permissions
|
- [ ] IAM roles configured with minimal permissions
|
||||||
|
|
|
||||||
|
|
@ -506,6 +506,47 @@ const brain = new Brainy({
|
||||||
```
|
```
|
||||||
|
|
||||||
2. **Cold Starts**
|
2. **Cold Starts**
|
||||||
|
|
||||||
|
**v7.3.0+ Progressive Initialization (Zero-Config)**
|
||||||
|
|
||||||
|
Brainy automatically detects Cloud Run and Cloud Functions environments
|
||||||
|
and uses progressive initialization for <200ms cold starts:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// Zero-config - Brainy auto-detects Cloud Run (K_SERVICE env var)
|
||||||
|
const brain = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 'gcs',
|
||||||
|
gcsNativeStorage: { bucketName: 'my-bucket' }
|
||||||
|
}
|
||||||
|
})
|
||||||
|
await brain.init() // Returns in <200ms
|
||||||
|
|
||||||
|
// First write validates bucket (lazy validation)
|
||||||
|
await brain.add('noun', { name: 'test' }) // Validates here
|
||||||
|
```
|
||||||
|
|
||||||
|
**Manual Override (if needed)**
|
||||||
|
```javascript
|
||||||
|
const brain = new Brainy({
|
||||||
|
storage: {
|
||||||
|
type: 'gcs',
|
||||||
|
gcsNativeStorage: {
|
||||||
|
bucketName: 'my-bucket',
|
||||||
|
// Force specific mode
|
||||||
|
initMode: 'progressive' // 'auto' | 'progressive' | 'strict'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
| Mode | Cold Start | Best For |
|
||||||
|
|------|------------|----------|
|
||||||
|
| `auto` (default) | <200ms in cloud | Zero-config, auto-detects |
|
||||||
|
| `progressive` | <200ms always | Force fast init everywhere |
|
||||||
|
| `strict` | 100-500ms+ | Local dev, tests, debugging |
|
||||||
|
|
||||||
|
**Keep Warm (Alternative)**
|
||||||
```javascript
|
```javascript
|
||||||
// Keep minimum instances warm
|
// Keep minimum instances warm
|
||||||
const brain = new Brainy({
|
const brain = new Brainy({
|
||||||
|
|
@ -516,6 +557,45 @@ const brain = new Brainy({
|
||||||
})
|
})
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Readiness Detection (v7.3.0+)**
|
||||||
|
|
||||||
|
Use the `brain.ready` Promise to ensure Brainy is initialized before handling requests:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
let brain
|
||||||
|
|
||||||
|
async function handleRequest(req, res) {
|
||||||
|
if (!brain) {
|
||||||
|
brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||||
|
brain.init() // Fire and forget
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait for initialization to complete
|
||||||
|
await brain.ready
|
||||||
|
|
||||||
|
// Now safe to use brain methods
|
||||||
|
const results = await brain.find({ query: req.query.q })
|
||||||
|
res.json(results)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
For Cloud Run health checks, use `isFullyInitialized()` to verify all background tasks are complete:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// Health check endpoint for Cloud Run
|
||||||
|
app.get('/health', async (req, res) => {
|
||||||
|
try {
|
||||||
|
await brain.ready
|
||||||
|
res.json({
|
||||||
|
status: 'ready',
|
||||||
|
fullyInitialized: brain.isFullyInitialized()
|
||||||
|
})
|
||||||
|
} catch (error) {
|
||||||
|
res.status(503).json({ status: 'initializing' })
|
||||||
|
}
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
3. **Memory Pressure**
|
3. **Memory Pressure**
|
||||||
```javascript
|
```javascript
|
||||||
// Optimize for GCP memory constraints
|
// Optimize for GCP memory constraints
|
||||||
|
|
|
||||||
144
src/brainy.ts
144
src/brainy.ts
|
|
@ -134,6 +134,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
private initialized = false
|
private initialized = false
|
||||||
private dimensions?: number
|
private dimensions?: number
|
||||||
|
|
||||||
|
// Ready Promise state (v7.3.0 - Unified readiness API)
|
||||||
|
// Allows consumers to await brain.ready for initialization completion
|
||||||
|
private _readyPromise: Promise<void> | null = null
|
||||||
|
private _readyResolve: (() => void) | null = null
|
||||||
|
private _readyReject: ((error: Error) => void) | null = null
|
||||||
|
|
||||||
// Lazy rebuild state (v5.7.7 - Production-scale lazy loading)
|
// Lazy rebuild state (v5.7.7 - Production-scale lazy loading)
|
||||||
// Prevents race conditions when multiple queries trigger rebuild simultaneously
|
// Prevents race conditions when multiple queries trigger rebuild simultaneously
|
||||||
private lazyRebuildInProgress = false
|
private lazyRebuildInProgress = false
|
||||||
|
|
@ -164,6 +170,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
this.setupDistributedComponents()
|
this.setupDistributedComponents()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Initialize ready Promise (v7.3.0)
|
||||||
|
// This allows consumers to await brain.ready before using the database
|
||||||
|
this._readyPromise = new Promise<void>((resolve, reject) => {
|
||||||
|
this._readyResolve = resolve
|
||||||
|
this._readyReject = reject
|
||||||
|
})
|
||||||
|
|
||||||
// Track this instance for shutdown hooks
|
// Track this instance for shutdown hooks
|
||||||
Brainy.instances.push(this)
|
Brainy.instances.push(this)
|
||||||
|
|
||||||
|
|
@ -308,7 +321,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
await embeddingManager.init()
|
await embeddingManager.init()
|
||||||
console.log('✅ Embedding engine ready')
|
console.log('✅ Embedding engine ready')
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// v7.3.0: Resolve ready Promise - consumers awaiting brain.ready will now proceed
|
||||||
|
if (this._readyResolve) {
|
||||||
|
this._readyResolve()
|
||||||
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
// v7.3.0: Reject ready Promise - consumers awaiting brain.ready will receive error
|
||||||
|
if (this._readyReject) {
|
||||||
|
this._readyReject(error instanceof Error ? error : new Error(String(error)))
|
||||||
|
}
|
||||||
throw new Error(`Failed to initialize Brainy: ${error}`)
|
throw new Error(`Failed to initialize Brainy: ${error}`)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -378,6 +400,128 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return this.initialized
|
return this.initialized
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Promise that resolves when Brainy is fully initialized and ready to use
|
||||||
|
*
|
||||||
|
* This Promise is created in the constructor and resolves when init() completes.
|
||||||
|
* It can be awaited multiple times safely - the result is cached.
|
||||||
|
*
|
||||||
|
* **v7.3.0 Feature**: This enables reliable readiness detection for consumers,
|
||||||
|
* especially in cloud environments where progressive initialization means
|
||||||
|
* init() returns quickly but background tasks may still be running.
|
||||||
|
*
|
||||||
|
* @example Waiting for readiness before API calls
|
||||||
|
* ```typescript
|
||||||
|
* const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||||
|
* brain.init() // Fire and forget
|
||||||
|
*
|
||||||
|
* // Elsewhere in your code (e.g., API handler)
|
||||||
|
* await brain.ready
|
||||||
|
* const results = await brain.find({ query: 'test' })
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* @example Server startup pattern
|
||||||
|
* ```typescript
|
||||||
|
* const brain = new Brainy()
|
||||||
|
* await brain.init()
|
||||||
|
*
|
||||||
|
* // For health check endpoint
|
||||||
|
* app.get('/health', async (req, res) => {
|
||||||
|
* try {
|
||||||
|
* await brain.ready
|
||||||
|
* res.json({ status: 'ready' })
|
||||||
|
* } catch (error) {
|
||||||
|
* res.status(503).json({ status: 'initializing', error: error.message })
|
||||||
|
* }
|
||||||
|
* })
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* @returns Promise that resolves when init() completes, or rejects if init fails
|
||||||
|
*/
|
||||||
|
get ready(): Promise<void> {
|
||||||
|
if (!this._readyPromise) {
|
||||||
|
// This should never happen if constructor ran, but handle gracefully
|
||||||
|
return Promise.reject(new Error('Brainy not constructed properly'))
|
||||||
|
}
|
||||||
|
return this._readyPromise
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if Brainy is fully initialized including all background tasks
|
||||||
|
*
|
||||||
|
* This checks both:
|
||||||
|
* 1. Basic initialization complete (init() returned)
|
||||||
|
* 2. Storage background tasks complete (bucket validation, count sync)
|
||||||
|
*
|
||||||
|
* Useful for determining if all lazy/progressive initialization is done.
|
||||||
|
*
|
||||||
|
* @returns true if all initialization including background tasks is complete
|
||||||
|
*
|
||||||
|
* @example Health check with background status
|
||||||
|
* ```typescript
|
||||||
|
* app.get('/health', (req, res) => {
|
||||||
|
* res.json({
|
||||||
|
* ready: brain.isInitialized,
|
||||||
|
* fullyInitialized: brain.isFullyInitialized(),
|
||||||
|
* status: brain.isFullyInitialized() ? 'ready' : 'warming'
|
||||||
|
* })
|
||||||
|
* })
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
isFullyInitialized(): boolean {
|
||||||
|
if (!this.initialized) return false
|
||||||
|
|
||||||
|
// Check if storage has background init methods (cloud storage adapters)
|
||||||
|
const storage = this.storage as any
|
||||||
|
if (typeof storage?.isBackgroundInitComplete === 'function') {
|
||||||
|
return storage.isBackgroundInitComplete()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Non-cloud storage adapters are fully initialized after init()
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Wait for all background initialization tasks to complete
|
||||||
|
*
|
||||||
|
* For cloud storage adapters with progressive initialization (v7.3.0+),
|
||||||
|
* this waits for:
|
||||||
|
* - Bucket/container validation
|
||||||
|
* - Count synchronization
|
||||||
|
* - Any other background tasks
|
||||||
|
*
|
||||||
|
* For non-cloud storage, this resolves immediately.
|
||||||
|
*
|
||||||
|
* **Use Case**: Call this when you need guaranteed consistency, such as:
|
||||||
|
* - Before running batch operations
|
||||||
|
* - Before reporting full system health
|
||||||
|
* - When transitioning from "initializing" to "ready" status
|
||||||
|
*
|
||||||
|
* @returns Promise that resolves when all background tasks complete
|
||||||
|
*
|
||||||
|
* @example Ensuring full initialization
|
||||||
|
* ```typescript
|
||||||
|
* const brain = new Brainy({ storage: { type: 'gcs', ... } })
|
||||||
|
* await brain.init() // Fast return in cloud (<200ms)
|
||||||
|
*
|
||||||
|
* // Optional: wait for background tasks if needed
|
||||||
|
* await brain.awaitBackgroundInit()
|
||||||
|
* console.log('All background tasks complete')
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
async awaitBackgroundInit(): Promise<void> {
|
||||||
|
// Must be initialized first
|
||||||
|
await this.ready
|
||||||
|
|
||||||
|
// Check if storage has background init methods (cloud storage adapters)
|
||||||
|
const storage = this.storage as any
|
||||||
|
if (typeof storage?.awaitBackgroundInit === 'function') {
|
||||||
|
await storage.awaitBackgroundInit()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Non-cloud storage: no background init to wait for
|
||||||
|
}
|
||||||
|
|
||||||
// ============= CORE CRUD OPERATIONS =============
|
// ============= CORE CRUD OPERATIONS =============
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -41,6 +41,7 @@ import { getGlobalBackpressure } from '../../utils/adaptiveBackpressure.js'
|
||||||
import { getWriteBuffer, WriteBuffer } from '../../utils/writeBuffer.js'
|
import { getWriteBuffer, WriteBuffer } from '../../utils/writeBuffer.js'
|
||||||
import { getCoalescer, RequestCoalescer } from '../../utils/requestCoalescer.js'
|
import { getCoalescer, RequestCoalescer } from '../../utils/requestCoalescer.js'
|
||||||
import { getShardIdFromUuid, getAllShardIds, getShardIdByIndex, TOTAL_SHARDS } from '../sharding.js'
|
import { getShardIdFromUuid, getAllShardIds, getShardIdByIndex, TOTAL_SHARDS } from '../sharding.js'
|
||||||
|
import { InitMode } from './baseStorageAdapter.js'
|
||||||
|
|
||||||
// Type aliases for better readability
|
// Type aliases for better readability
|
||||||
type HNSWNode = HNSWNoun
|
type HNSWNode = HNSWNoun
|
||||||
|
|
@ -119,7 +120,26 @@ export class AzureBlobStorage extends BaseStorage {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initialize the storage adapter
|
* Initialize the storage adapter
|
||||||
|
*
|
||||||
* @param options Configuration options for Azure Blob Storage
|
* @param options Configuration options for Azure Blob Storage
|
||||||
|
*
|
||||||
|
* @example Zero-config (recommended) - auto-detects Azure Functions for fast init
|
||||||
|
* ```typescript
|
||||||
|
* const storage = new AzureBlobStorage({
|
||||||
|
* containerName: 'my-container',
|
||||||
|
* accountName: 'myaccount'
|
||||||
|
* })
|
||||||
|
* await storage.init() // <200ms in Azure Functions, blocking locally
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* @example Force progressive mode for all environments
|
||||||
|
* ```typescript
|
||||||
|
* const storage = new AzureBlobStorage({
|
||||||
|
* containerName: 'my-container',
|
||||||
|
* accountName: 'myaccount',
|
||||||
|
* initMode: 'progressive' // Always <200ms init
|
||||||
|
* })
|
||||||
|
* ```
|
||||||
*/
|
*/
|
||||||
constructor(options: {
|
constructor(options: {
|
||||||
containerName: string
|
containerName: string
|
||||||
|
|
@ -140,6 +160,21 @@ export class AzureBlobStorage extends BaseStorage {
|
||||||
hotCacheEvictionThreshold?: number
|
hotCacheEvictionThreshold?: number
|
||||||
warmCacheTTL?: number
|
warmCacheTTL?: number
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Initialization mode for fast cold starts (v7.3.0+)
|
||||||
|
*
|
||||||
|
* - `'auto'` (default): Progressive in cloud environments (Azure Functions),
|
||||||
|
* strict locally. Zero-config optimization.
|
||||||
|
* - `'progressive'`: Always use fast init (<200ms). Container validation and
|
||||||
|
* count loading happen in background. First write validates container.
|
||||||
|
* - `'strict'`: Traditional blocking init. Validates container and loads counts
|
||||||
|
* before init() returns.
|
||||||
|
*
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
initMode?: InitMode
|
||||||
|
|
||||||
readOnly?: boolean
|
readOnly?: boolean
|
||||||
}) {
|
}) {
|
||||||
super()
|
super()
|
||||||
|
|
@ -150,6 +185,11 @@ export class AzureBlobStorage extends BaseStorage {
|
||||||
this.sasToken = options.sasToken
|
this.sasToken = options.sasToken
|
||||||
this.readOnly = options.readOnly || false
|
this.readOnly = options.readOnly || false
|
||||||
|
|
||||||
|
// v7.3.0: Handle initMode
|
||||||
|
if (options.initMode) {
|
||||||
|
this.initMode = options.initMode
|
||||||
|
}
|
||||||
|
|
||||||
// Set up prefixes for different types of data using entity-based structure
|
// Set up prefixes for different types of data using entity-based structure
|
||||||
this.nounPrefix = `${getDirectoryPath('noun', 'vector')}/`
|
this.nounPrefix = `${getDirectoryPath('noun', 'vector')}/`
|
||||||
this.verbPrefix = `${getDirectoryPath('verb', 'vector')}/`
|
this.verbPrefix = `${getDirectoryPath('verb', 'vector')}/`
|
||||||
|
|
@ -256,6 +296,22 @@ export class AzureBlobStorage extends BaseStorage {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initialize the storage adapter
|
* Initialize the storage adapter
|
||||||
|
*
|
||||||
|
* v7.3.0: Supports progressive initialization for fast cold starts
|
||||||
|
*
|
||||||
|
* | Mode | Init Time | When |
|
||||||
|
* |------|-----------|------|
|
||||||
|
* | `progressive` | <200ms | Azure Functions |
|
||||||
|
* | `strict` | 100-500ms+ | Local development, tests |
|
||||||
|
* | `auto` | Detected | Default - best of both |
|
||||||
|
*
|
||||||
|
* In progressive mode:
|
||||||
|
* - SDK import and client creation: ~50ms (unavoidable)
|
||||||
|
* - Write buffers and caches: ~10ms
|
||||||
|
* - Mark as initialized: READY
|
||||||
|
* - Background: validate container, load counts
|
||||||
|
*
|
||||||
|
* First write operation validates container existence (lazy validation).
|
||||||
*/
|
*/
|
||||||
public async init(): Promise<void> {
|
public async init(): Promise<void> {
|
||||||
if (this.isInitialized) {
|
if (this.isInitialized) {
|
||||||
|
|
@ -263,7 +319,7 @@ export class AzureBlobStorage extends BaseStorage {
|
||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
// Import Azure Storage SDK only when needed
|
// Import Azure Storage SDK only when needed (~50ms)
|
||||||
const { BlobServiceClient } = await import('@azure/storage-blob')
|
const { BlobServiceClient } = await import('@azure/storage-blob')
|
||||||
|
|
||||||
// Configure the Azure Blob Storage client based on available credentials
|
// Configure the Azure Blob Storage client based on available credentials
|
||||||
|
|
@ -306,17 +362,14 @@ export class AzureBlobStorage extends BaseStorage {
|
||||||
throw new Error('Azure Blob Storage requires either connectionString, accountName+accountKey, accountName+sasToken, or accountName (for Managed Identity)')
|
throw new Error('Azure Blob Storage requires either connectionString, accountName+accountKey, accountName+sasToken, or accountName (for Managed Identity)')
|
||||||
}
|
}
|
||||||
|
|
||||||
// Get reference to the container
|
// Get reference to the container (no network calls)
|
||||||
this.containerClient = this.blobServiceClient.getContainerClient(this.containerName)
|
this.containerClient = this.blobServiceClient.getContainerClient(this.containerName)
|
||||||
|
|
||||||
// Create container if it doesn't exist
|
// Determine initialization mode
|
||||||
const exists = await this.containerClient.exists()
|
const effectiveMode = this.resolveInitMode()
|
||||||
if (!exists) {
|
const isCloud = this.detectCloudEnvironment()
|
||||||
await this.containerClient.create()
|
|
||||||
prodLog.info(`✅ Created Azure container: ${this.containerName}`)
|
prodLog.info(`🚀 Azure init mode: ${effectiveMode} (detected cloud: ${isCloud})`)
|
||||||
} else {
|
|
||||||
prodLog.info(`✅ Connected to Azure container: ${this.containerName}`)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Initialize write buffers for high-volume mode
|
// Initialize write buffers for high-volume mode
|
||||||
const storageId = `azure-${this.containerName}`
|
const storageId = `azure-${this.containerName}`
|
||||||
|
|
@ -345,23 +398,183 @@ export class AzureBlobStorage extends BaseStorage {
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
|
||||||
// Initialize counts from storage
|
|
||||||
await this.initializeCounts()
|
|
||||||
|
|
||||||
// Clear any stale cache entries from previous runs
|
// Clear any stale cache entries from previous runs
|
||||||
prodLog.info('🧹 Clearing cache from previous run to prevent cache poisoning')
|
prodLog.info('🧹 Clearing cache from previous run to prevent cache poisoning')
|
||||||
this.nounCacheManager.clear()
|
this.nounCacheManager.clear()
|
||||||
this.verbCacheManager.clear()
|
this.verbCacheManager.clear()
|
||||||
prodLog.info('✅ Cache cleared - starting fresh')
|
prodLog.info('✅ Cache cleared - starting fresh')
|
||||||
|
|
||||||
|
// v7.3.0: Progressive vs Strict initialization
|
||||||
|
if (effectiveMode === 'progressive') {
|
||||||
|
// PROGRESSIVE MODE: Fast init, background validation
|
||||||
|
// Mark as initialized immediately - ready to accept operations
|
||||||
|
// Container validation happens lazily on first write
|
||||||
|
// Count loading happens in background
|
||||||
|
|
||||||
|
prodLog.info(`✅ Azure progressive init complete: ${this.containerName} (validation deferred)`)
|
||||||
|
|
||||||
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
|
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
|
||||||
await super.init()
|
await super.init()
|
||||||
|
|
||||||
|
// Schedule background tasks (non-blocking)
|
||||||
|
this.scheduleBackgroundInit()
|
||||||
|
} else {
|
||||||
|
// STRICT MODE: Traditional blocking initialization
|
||||||
|
// Verify container exists or create it (blocking)
|
||||||
|
const exists = await this.containerClient.exists()
|
||||||
|
if (!exists) {
|
||||||
|
await this.containerClient.create()
|
||||||
|
prodLog.info(`✅ Created Azure container: ${this.containerName}`)
|
||||||
|
} else {
|
||||||
|
prodLog.info(`✅ Connected to Azure container: ${this.containerName}`)
|
||||||
|
}
|
||||||
|
this.bucketValidated = true
|
||||||
|
|
||||||
|
// Initialize counts from storage (blocking)
|
||||||
|
await this.initializeCounts()
|
||||||
|
this.countsLoaded = true
|
||||||
|
|
||||||
|
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
|
||||||
|
await super.init()
|
||||||
|
|
||||||
|
// Mark background tasks as complete (nothing to do in background)
|
||||||
|
this.backgroundTasksComplete = true
|
||||||
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
this.logger.error('Failed to initialize Azure Blob Storage:', error)
|
this.logger.error('Failed to initialize Azure Blob Storage:', error)
|
||||||
throw new Error(`Failed to initialize Azure Blob Storage: ${error}`)
|
throw new Error(`Failed to initialize Azure Blob Storage: ${error}`)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// =============================================
|
||||||
|
// Progressive Initialization (v7.3.0+)
|
||||||
|
// =============================================
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Run background initialization tasks for Azure.
|
||||||
|
*
|
||||||
|
* Called in progressive mode after init() returns. Performs:
|
||||||
|
* 1. Container validation (in background)
|
||||||
|
* 2. Count loading from storage (in background)
|
||||||
|
*
|
||||||
|
* These tasks don't block the main thread, allowing fast cold starts.
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @override
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected async runBackgroundInit(): Promise<void> {
|
||||||
|
const startTime = Date.now()
|
||||||
|
prodLog.info('[Azure Background] Starting background initialization...')
|
||||||
|
|
||||||
|
// Run validation and count loading in parallel
|
||||||
|
const validationPromise = this.validateContainerInBackground()
|
||||||
|
const countsPromise = this.loadCountsInBackground()
|
||||||
|
|
||||||
|
// Wait for both to complete (but we're already initialized)
|
||||||
|
await Promise.all([validationPromise, countsPromise])
|
||||||
|
|
||||||
|
const elapsed = Date.now() - startTime
|
||||||
|
prodLog.info(`[Azure Background] Background init complete in ${elapsed}ms`)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validate container existence in background.
|
||||||
|
*
|
||||||
|
* Creates container if it doesn't exist. Stores result in
|
||||||
|
* bucketValidated/bucketValidationError for lazy use.
|
||||||
|
*
|
||||||
|
* @private
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
private async validateContainerInBackground(): Promise<void> {
|
||||||
|
try {
|
||||||
|
const exists = await this.containerClient!.exists()
|
||||||
|
if (!exists) {
|
||||||
|
// Try to create container
|
||||||
|
try {
|
||||||
|
await this.containerClient!.create()
|
||||||
|
prodLog.info(`[Azure Background] Created container: ${this.containerName}`)
|
||||||
|
} catch (createError: any) {
|
||||||
|
// Another process might have created it - check again
|
||||||
|
const existsNow = await this.containerClient!.exists()
|
||||||
|
if (!existsNow) {
|
||||||
|
throw createError
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this.bucketValidated = true
|
||||||
|
prodLog.info(`[Azure Background] Container validated: ${this.containerName}`)
|
||||||
|
} catch (error: any) {
|
||||||
|
this.bucketValidationError = new Error(
|
||||||
|
`Container ${this.containerName} validation failed: ${error.message || error}`
|
||||||
|
)
|
||||||
|
prodLog.warn(`[Azure Background] Container validation failed: ${this.containerName}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Load counts from storage in background.
|
||||||
|
*
|
||||||
|
* @private
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
private async loadCountsInBackground(): Promise<void> {
|
||||||
|
try {
|
||||||
|
await this.initializeCounts()
|
||||||
|
this.countsLoaded = true
|
||||||
|
prodLog.info(`[Azure Background] Counts loaded: ${this.totalNounCount} nouns, ${this.totalVerbCount} verbs`)
|
||||||
|
} catch (error: any) {
|
||||||
|
// Non-fatal in progressive mode - counts start at 0
|
||||||
|
prodLog.warn(`[Azure Background] Failed to load counts (starting at 0): ${error.message}`)
|
||||||
|
this.countsLoaded = true // Mark as loaded even on error (0 is valid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Ensure container is validated before write operations.
|
||||||
|
*
|
||||||
|
* In progressive mode, container validation is deferred until the first
|
||||||
|
* write operation. This method validates the container and caches the
|
||||||
|
* result (or error) for subsequent calls.
|
||||||
|
*
|
||||||
|
* @throws Error if container validation fails
|
||||||
|
* @protected
|
||||||
|
* @override
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected async ensureValidatedForWrite(): Promise<void> {
|
||||||
|
// If already validated, nothing to do
|
||||||
|
if (this.bucketValidated) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// If we have a cached validation error from background init, throw it
|
||||||
|
if (this.bucketValidationError) {
|
||||||
|
throw this.bucketValidationError
|
||||||
|
}
|
||||||
|
|
||||||
|
// Perform synchronous validation (first write in progressive mode)
|
||||||
|
try {
|
||||||
|
prodLog.info(`[Azure] Lazy container validation on first write: ${this.containerName}`)
|
||||||
|
const exists = await this.containerClient!.exists()
|
||||||
|
if (!exists) {
|
||||||
|
// Try to create container
|
||||||
|
await this.containerClient!.create()
|
||||||
|
prodLog.info(`[Azure] Created container: ${this.containerName}`)
|
||||||
|
}
|
||||||
|
this.bucketValidated = true
|
||||||
|
prodLog.info(`[Azure] Container validated successfully: ${this.containerName}`)
|
||||||
|
} catch (error: any) {
|
||||||
|
// Cache the error for fast-fail on subsequent writes
|
||||||
|
const wrappedError = new Error(
|
||||||
|
`Container ${this.containerName} validation failed: ${error.message || error}`
|
||||||
|
)
|
||||||
|
this.bucketValidationError = wrappedError
|
||||||
|
throw wrappedError
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get the Azure blob name for a noun using UUID-based sharding
|
* Get the Azure blob name for a noun using UUID-based sharding
|
||||||
*
|
*
|
||||||
|
|
@ -681,10 +894,14 @@ export class AzureBlobStorage extends BaseStorage {
|
||||||
/**
|
/**
|
||||||
* Write an object to a specific path in Azure
|
* Write an object to a specific path in Azure
|
||||||
* Primitive operation required by base class
|
* Primitive operation required by base class
|
||||||
|
*
|
||||||
|
* v7.3.0: Performs lazy container validation on first write in progressive mode.
|
||||||
* @protected
|
* @protected
|
||||||
*/
|
*/
|
||||||
protected async writeObjectToPath(path: string, data: any): Promise<void> {
|
protected async writeObjectToPath(path: string, data: any): Promise<void> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
// v7.3.0: Lazy container validation for progressive init
|
||||||
|
await this.ensureValidatedForWrite()
|
||||||
|
|
||||||
try {
|
try {
|
||||||
this.logger.trace(`Writing object to path: ${path}`)
|
this.logger.trace(`Writing object to path: ${path}`)
|
||||||
|
|
@ -736,10 +953,14 @@ export class AzureBlobStorage extends BaseStorage {
|
||||||
/**
|
/**
|
||||||
* Delete an object from a specific path in Azure
|
* Delete an object from a specific path in Azure
|
||||||
* Primitive operation required by base class
|
* Primitive operation required by base class
|
||||||
|
*
|
||||||
|
* v7.3.0: Performs lazy container validation on first delete in progressive mode.
|
||||||
* @protected
|
* @protected
|
||||||
*/
|
*/
|
||||||
protected async deleteObjectFromPath(path: string): Promise<void> {
|
protected async deleteObjectFromPath(path: string): Promise<void> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
// v7.3.0: Lazy container validation for progressive init
|
||||||
|
await this.ensureValidatedForWrite()
|
||||||
|
|
||||||
try {
|
try {
|
||||||
this.logger.trace(`Deleting object at path: ${path}`)
|
this.logger.trace(`Deleting object at path: ${path}`)
|
||||||
|
|
|
||||||
|
|
@ -18,6 +18,45 @@ import { StorageBatchConfig } from '../baseStorage.js'
|
||||||
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
|
import { extractFieldNamesFromJson, mapToStandardField } from '../../utils/fieldNameTracking.js'
|
||||||
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
|
import { getGlobalMutex, cleanupMutexes } from '../../utils/mutex.js'
|
||||||
|
|
||||||
|
// =============================================================================
|
||||||
|
// Progressive Initialization Types (v7.3.0+)
|
||||||
|
// =============================================================================
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Initialization mode for cloud storage adapters.
|
||||||
|
*
|
||||||
|
* Controls how storage adapters initialize during startup, enabling
|
||||||
|
* fast cold starts in serverless environments while maintaining strict
|
||||||
|
* validation for local development.
|
||||||
|
*
|
||||||
|
* @since v7.3.0
|
||||||
|
*
|
||||||
|
* | Mode | Description | Use Case |
|
||||||
|
* |------|-------------|----------|
|
||||||
|
* | `'auto'` | Auto-detect environment (progressive in cloud, strict locally) | Default, zero-config |
|
||||||
|
* | `'progressive'` | Fast init (<200ms), validate lazily on first write | Serverless, Cloud Run |
|
||||||
|
* | `'strict'` | Blocking validation during init (traditional behavior) | Local dev, testing |
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* ```typescript
|
||||||
|
* // Zero-config: auto-detects Cloud Run, Lambda, etc.
|
||||||
|
* const storage = new GCSStorageAdapter({ bucket: 'my-bucket' })
|
||||||
|
*
|
||||||
|
* // Force progressive mode for all environments
|
||||||
|
* const storage = new GCSStorageAdapter({
|
||||||
|
* bucket: 'my-bucket',
|
||||||
|
* initMode: 'progressive'
|
||||||
|
* })
|
||||||
|
*
|
||||||
|
* // Force strict validation (useful for testing)
|
||||||
|
* const storage = new GCSStorageAdapter({
|
||||||
|
* bucket: 'my-bucket',
|
||||||
|
* initMode: 'strict'
|
||||||
|
* })
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
export type InitMode = 'progressive' | 'strict' | 'auto'
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Base class for storage adapters that implements statistics tracking
|
* Base class for storage adapters that implements statistics tracking
|
||||||
*/
|
*/
|
||||||
|
|
@ -255,6 +294,77 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
||||||
status: 'normal' | 'throttled' | 'recovering'
|
status: 'normal' | 'throttled' | 'recovering'
|
||||||
}> = new Map()
|
}> = new Map()
|
||||||
|
|
||||||
|
// =============================================
|
||||||
|
// Progressive Initialization State (v7.3.0+)
|
||||||
|
// =============================================
|
||||||
|
// These properties enable fast cold starts in cloud environments
|
||||||
|
// by deferring validation and count loading to background tasks.
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Initialization mode for this storage adapter.
|
||||||
|
*
|
||||||
|
* - `'auto'` (default): Progressive in cloud environments, strict locally
|
||||||
|
* - `'progressive'`: Always use fast init with lazy validation
|
||||||
|
* - `'strict'`: Always validate during init (traditional behavior)
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected initMode: InitMode = 'auto'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether the bucket/container has been validated to exist.
|
||||||
|
*
|
||||||
|
* In progressive mode, this starts as `false` and is set to `true`
|
||||||
|
* after the first successful write operation or background validation.
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected bucketValidated = false
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Error from bucket validation, if any.
|
||||||
|
*
|
||||||
|
* Stored here so subsequent operations can fail fast with the same error.
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected bucketValidationError: Error | null = null
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether counts have been loaded from storage.
|
||||||
|
*
|
||||||
|
* In progressive mode, counts start at 0 and are loaded in the background.
|
||||||
|
* Operations work immediately; counts become accurate after background load.
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected countsLoaded = false
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether all background initialization tasks have completed.
|
||||||
|
*
|
||||||
|
* Useful for tests and diagnostics to ensure full initialization.
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected backgroundTasksComplete = false
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Promise that resolves when background initialization completes.
|
||||||
|
*
|
||||||
|
* Can be awaited by callers who need to ensure full initialization
|
||||||
|
* before proceeding (e.g., in tests).
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected backgroundInitPromise: Promise<void> | null = null
|
||||||
|
|
||||||
// Statistics-specific methods that must be implemented by subclasses
|
// Statistics-specific methods that must be implemented by subclasses
|
||||||
protected abstract saveStatisticsData(
|
protected abstract saveStatisticsData(
|
||||||
statistics: StatisticsData
|
statistics: StatisticsData
|
||||||
|
|
@ -1124,6 +1234,169 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// =============================================
|
||||||
|
// Progressive Initialization Methods (v7.3.0+)
|
||||||
|
// =============================================
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Detect if running in a cloud/serverless environment.
|
||||||
|
*
|
||||||
|
* Cloud environments benefit from progressive initialization to minimize
|
||||||
|
* cold start times. This method checks for common environment variables
|
||||||
|
* set by cloud providers.
|
||||||
|
*
|
||||||
|
* | Provider | Environment Variable |
|
||||||
|
* |----------|---------------------|
|
||||||
|
* | Cloud Run | `K_SERVICE`, `K_REVISION` |
|
||||||
|
* | Lambda | `AWS_LAMBDA_FUNCTION_NAME` |
|
||||||
|
* | Cloud Functions | `FUNCTIONS_TARGET` |
|
||||||
|
* | Azure Functions | `AZURE_FUNCTIONS_ENVIRONMENT` |
|
||||||
|
*
|
||||||
|
* @returns `true` if running in a detected cloud environment
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected detectCloudEnvironment(): boolean {
|
||||||
|
return !!(
|
||||||
|
process.env.K_SERVICE || // Google Cloud Run
|
||||||
|
process.env.K_REVISION || // Google Cloud Run (revision)
|
||||||
|
process.env.AWS_LAMBDA_FUNCTION_NAME || // AWS Lambda
|
||||||
|
process.env.FUNCTIONS_TARGET || // Google Cloud Functions
|
||||||
|
process.env.AZURE_FUNCTIONS_ENVIRONMENT // Azure Functions
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Determine the effective initialization mode.
|
||||||
|
*
|
||||||
|
* Resolves `'auto'` to either `'progressive'` or `'strict'` based on
|
||||||
|
* the detected environment.
|
||||||
|
*
|
||||||
|
* @returns The resolved init mode (`'progressive'` or `'strict'`)
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected resolveInitMode(): 'progressive' | 'strict' {
|
||||||
|
if (this.initMode === 'auto') {
|
||||||
|
return this.detectCloudEnvironment() ? 'progressive' : 'strict'
|
||||||
|
}
|
||||||
|
return this.initMode
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Schedule background initialization tasks.
|
||||||
|
*
|
||||||
|
* This method is called in progressive mode after the adapter is marked
|
||||||
|
* as initialized. It schedules non-blocking tasks to:
|
||||||
|
* 1. Validate bucket/container existence
|
||||||
|
* 2. Load counts from storage
|
||||||
|
*
|
||||||
|
* Override in subclasses to add storage-specific background tasks.
|
||||||
|
* Always call `super.scheduleBackgroundInit()` first.
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected scheduleBackgroundInit(): void {
|
||||||
|
// Create a promise that tracks all background work
|
||||||
|
this.backgroundInitPromise = this.runBackgroundInit()
|
||||||
|
|
||||||
|
// Don't await - let it run in background
|
||||||
|
this.backgroundInitPromise
|
||||||
|
.then(() => {
|
||||||
|
this.backgroundTasksComplete = true
|
||||||
|
})
|
||||||
|
.catch((error) => {
|
||||||
|
// Log but don't throw - progressive mode is best-effort
|
||||||
|
console.error('[Progressive Init] Background initialization failed:', error)
|
||||||
|
this.backgroundTasksComplete = true // Mark complete even on error
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Run background initialization tasks.
|
||||||
|
*
|
||||||
|
* Override in subclasses to add storage-specific tasks.
|
||||||
|
* The default implementation does nothing (for local storage adapters).
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected async runBackgroundInit(): Promise<void> {
|
||||||
|
// Default implementation: nothing to do for local storage
|
||||||
|
// Cloud storage adapters override this to:
|
||||||
|
// 1. Validate bucket/container
|
||||||
|
// 2. Load counts from storage
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Wait for background initialization to complete.
|
||||||
|
*
|
||||||
|
* Useful in tests or when you need to ensure all background tasks
|
||||||
|
* have finished before proceeding.
|
||||||
|
*
|
||||||
|
* @example
|
||||||
|
* ```typescript
|
||||||
|
* const storage = new GCSStorageAdapter({ bucket: 'my-bucket' })
|
||||||
|
* await storage.init()
|
||||||
|
*
|
||||||
|
* // Wait for background validation and count loading
|
||||||
|
* await storage.awaitBackgroundInit()
|
||||||
|
*
|
||||||
|
* // Now counts are accurate and bucket is validated
|
||||||
|
* const count = await storage.getNounCount()
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* @public
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
public async awaitBackgroundInit(): Promise<void> {
|
||||||
|
if (this.backgroundInitPromise) {
|
||||||
|
await this.backgroundInitPromise
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if background initialization has completed.
|
||||||
|
*
|
||||||
|
* @returns `true` if all background tasks are complete
|
||||||
|
* @public
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
public isBackgroundInitComplete(): boolean {
|
||||||
|
return this.backgroundTasksComplete
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Ensure bucket/container is validated before write operations.
|
||||||
|
*
|
||||||
|
* In progressive mode, bucket validation is deferred until the first
|
||||||
|
* write operation. This method performs lazy validation and caches
|
||||||
|
* the result (or error) for subsequent calls.
|
||||||
|
*
|
||||||
|
* Override in subclasses to implement storage-specific validation.
|
||||||
|
* The base implementation does nothing (for local storage adapters).
|
||||||
|
*
|
||||||
|
* @throws Error if bucket validation fails
|
||||||
|
* @protected
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected async ensureValidatedForWrite(): Promise<void> {
|
||||||
|
// If already validated, nothing to do
|
||||||
|
if (this.bucketValidated) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// If we have a cached validation error, throw it
|
||||||
|
if (this.bucketValidationError) {
|
||||||
|
throw this.bucketValidationError
|
||||||
|
}
|
||||||
|
|
||||||
|
// Default implementation: no validation needed (local storage)
|
||||||
|
// Cloud storage adapters override this to validate bucket/container
|
||||||
|
this.bucketValidated = true
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Schedule a smart batched persist operation.
|
* Schedule a smart batched persist operation.
|
||||||
*
|
*
|
||||||
|
|
|
||||||
|
|
@ -39,6 +39,7 @@ import { getGlobalBackpressure } from '../../utils/adaptiveBackpressure.js'
|
||||||
import { getWriteBuffer, WriteBuffer } from '../../utils/writeBuffer.js'
|
import { getWriteBuffer, WriteBuffer } from '../../utils/writeBuffer.js'
|
||||||
import { getCoalescer, RequestCoalescer } from '../../utils/requestCoalescer.js'
|
import { getCoalescer, RequestCoalescer } from '../../utils/requestCoalescer.js'
|
||||||
import { getShardIdFromUuid, getAllShardIds, getShardIdByIndex, TOTAL_SHARDS } from '../sharding.js'
|
import { getShardIdFromUuid, getAllShardIds, getShardIdByIndex, TOTAL_SHARDS } from '../sharding.js'
|
||||||
|
import { InitMode } from './baseStorageAdapter.js'
|
||||||
|
|
||||||
// Type aliases for better readability
|
// Type aliases for better readability
|
||||||
type HNSWNode = HNSWNoun
|
type HNSWNode = HNSWNoun
|
||||||
|
|
@ -128,7 +129,30 @@ export class GcsStorage extends BaseStorage {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initialize the storage adapter
|
* Initialize the storage adapter
|
||||||
|
*
|
||||||
* @param options Configuration options for Google Cloud Storage
|
* @param options Configuration options for Google Cloud Storage
|
||||||
|
*
|
||||||
|
* @example Zero-config (recommended) - auto-detects Cloud Run/Lambda for fast init
|
||||||
|
* ```typescript
|
||||||
|
* const storage = new GcsStorage({ bucketName: 'my-bucket' })
|
||||||
|
* await storage.init() // <200ms in Cloud Run, blocking locally
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* @example Force progressive mode for all environments
|
||||||
|
* ```typescript
|
||||||
|
* const storage = new GcsStorage({
|
||||||
|
* bucketName: 'my-bucket',
|
||||||
|
* initMode: 'progressive' // Always <200ms init
|
||||||
|
* })
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* @example Force strict validation (useful for tests)
|
||||||
|
* ```typescript
|
||||||
|
* const storage = new GcsStorage({
|
||||||
|
* bucketName: 'my-bucket',
|
||||||
|
* initMode: 'strict' // Always validate during init
|
||||||
|
* })
|
||||||
|
* ```
|
||||||
*/
|
*/
|
||||||
constructor(options: {
|
constructor(options: {
|
||||||
bucketName: string
|
bucketName: string
|
||||||
|
|
@ -141,8 +165,30 @@ export class GcsStorage extends BaseStorage {
|
||||||
accessKeyId?: string
|
accessKeyId?: string
|
||||||
secretAccessKey?: string
|
secretAccessKey?: string
|
||||||
|
|
||||||
// Initialization configuration
|
/**
|
||||||
|
* Initialization mode for fast cold starts (v7.3.0+)
|
||||||
|
*
|
||||||
|
* - `'auto'` (default): Progressive in cloud environments (Cloud Run, Lambda),
|
||||||
|
* strict locally. Zero-config optimization.
|
||||||
|
* - `'progressive'`: Always use fast init (<200ms). Bucket validation and
|
||||||
|
* count loading happen in background. First write validates bucket.
|
||||||
|
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
|
||||||
|
* before init() returns.
|
||||||
|
*
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
initMode?: InitMode
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @deprecated Use `initMode: 'progressive'` instead.
|
||||||
|
* Will be removed in v8.0.0.
|
||||||
|
*/
|
||||||
skipInitialScan?: boolean
|
skipInitialScan?: boolean
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @deprecated Use `initMode: 'progressive'` instead.
|
||||||
|
* Will be removed in v8.0.0.
|
||||||
|
*/
|
||||||
skipCountsFile?: boolean
|
skipCountsFile?: boolean
|
||||||
|
|
||||||
// Cache and operation configuration
|
// Cache and operation configuration
|
||||||
|
|
@ -159,8 +205,28 @@ export class GcsStorage extends BaseStorage {
|
||||||
this.credentials = options.credentials
|
this.credentials = options.credentials
|
||||||
this.accessKeyId = options.accessKeyId
|
this.accessKeyId = options.accessKeyId
|
||||||
this.secretAccessKey = options.secretAccessKey
|
this.secretAccessKey = options.secretAccessKey
|
||||||
this.skipInitialScan = options.skipInitialScan || false
|
|
||||||
this.skipCountsFile = options.skipCountsFile || false
|
// v7.3.0: Handle initMode and deprecated skip* flags
|
||||||
|
if (options.initMode) {
|
||||||
|
this.initMode = options.initMode
|
||||||
|
}
|
||||||
|
|
||||||
|
// Deprecation warnings for skip* flags
|
||||||
|
if (options.skipInitialScan) {
|
||||||
|
console.warn(
|
||||||
|
'[GcsStorage] DEPRECATION WARNING: skipInitialScan is deprecated. ' +
|
||||||
|
'Use initMode: "progressive" instead. Will be removed in v8.0.0.'
|
||||||
|
)
|
||||||
|
this.skipInitialScan = true
|
||||||
|
}
|
||||||
|
if (options.skipCountsFile) {
|
||||||
|
console.warn(
|
||||||
|
'[GcsStorage] DEPRECATION WARNING: skipCountsFile is deprecated. ' +
|
||||||
|
'Use initMode: "progressive" instead. Will be removed in v8.0.0.'
|
||||||
|
)
|
||||||
|
this.skipCountsFile = true
|
||||||
|
}
|
||||||
|
|
||||||
this.readOnly = options.readOnly || false
|
this.readOnly = options.readOnly || false
|
||||||
|
|
||||||
// Set up prefixes for different types of data using entity-based structure
|
// Set up prefixes for different types of data using entity-based structure
|
||||||
|
|
@ -179,6 +245,22 @@ export class GcsStorage extends BaseStorage {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initialize the storage adapter
|
* Initialize the storage adapter
|
||||||
|
*
|
||||||
|
* v7.3.0: Supports progressive initialization for fast cold starts
|
||||||
|
*
|
||||||
|
* | Mode | Init Time | When |
|
||||||
|
* |------|-----------|------|
|
||||||
|
* | `progressive` | <200ms | Cloud Run, Lambda, Azure Functions |
|
||||||
|
* | `strict` | 100-500ms+ | Local development, tests |
|
||||||
|
* | `auto` | Detected | Default - best of both |
|
||||||
|
*
|
||||||
|
* In progressive mode:
|
||||||
|
* - SDK import and bucket reference: ~50ms (unavoidable)
|
||||||
|
* - Write buffers and caches: ~10ms
|
||||||
|
* - Mark as initialized: READY
|
||||||
|
* - Background: validate bucket, load counts
|
||||||
|
*
|
||||||
|
* First write operation validates bucket existence (lazy validation).
|
||||||
*/
|
*/
|
||||||
public async init(): Promise<void> {
|
public async init(): Promise<void> {
|
||||||
if (this.isInitialized) {
|
if (this.isInitialized) {
|
||||||
|
|
@ -186,7 +268,7 @@ export class GcsStorage extends BaseStorage {
|
||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
// Import Google Cloud Storage SDK only when needed
|
// Import Google Cloud Storage SDK only when needed (~50ms)
|
||||||
const { Storage } = await import('@google-cloud/storage')
|
const { Storage } = await import('@google-cloud/storage')
|
||||||
|
|
||||||
// Configure the GCS client based on available credentials
|
// Configure the GCS client based on available credentials
|
||||||
|
|
@ -216,19 +298,17 @@ export class GcsStorage extends BaseStorage {
|
||||||
prodLog.info('🔐 GCS: Using Application Default Credentials (ADC)')
|
prodLog.info('🔐 GCS: Using Application Default Credentials (ADC)')
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create the GCS client
|
// Create the GCS client (no network calls)
|
||||||
this.storage = new Storage(clientConfig)
|
this.storage = new Storage(clientConfig)
|
||||||
|
|
||||||
// Get reference to the bucket
|
// Get reference to the bucket (no network calls)
|
||||||
this.bucket = this.storage.bucket(this.bucketName)
|
this.bucket = this.storage.bucket(this.bucketName)
|
||||||
|
|
||||||
// Verify bucket exists and is accessible
|
// Determine initialization mode
|
||||||
const [exists] = await this.bucket.exists()
|
const effectiveMode = this.resolveInitMode()
|
||||||
if (!exists) {
|
const isCloud = this.detectCloudEnvironment()
|
||||||
throw new Error(`Bucket ${this.bucketName} does not exist or is not accessible`)
|
|
||||||
}
|
|
||||||
|
|
||||||
prodLog.info(`✅ Connected to GCS bucket: ${this.bucketName}`)
|
prodLog.info(`🚀 GCS init mode: ${effectiveMode} (detected cloud: ${isCloud})`)
|
||||||
|
|
||||||
// Initialize write buffers for high-volume mode
|
// Initialize write buffers for high-volume mode
|
||||||
const storageId = `gcs-${this.bucketName}`
|
const storageId = `gcs-${this.bucketName}`
|
||||||
|
|
@ -257,9 +337,6 @@ export class GcsStorage extends BaseStorage {
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
|
||||||
// Initialize counts from storage
|
|
||||||
await this.initializeCounts()
|
|
||||||
|
|
||||||
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs
|
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs
|
||||||
// This prevents cache poisoning from causing silent failures on container restart
|
// This prevents cache poisoning from causing silent failures on container restart
|
||||||
prodLog.info('🧹 Clearing cache from previous run to prevent cache poisoning')
|
prodLog.info('🧹 Clearing cache from previous run to prevent cache poisoning')
|
||||||
|
|
@ -267,14 +344,170 @@ export class GcsStorage extends BaseStorage {
|
||||||
this.verbCacheManager.clear()
|
this.verbCacheManager.clear()
|
||||||
prodLog.info('✅ Cache cleared - starting fresh')
|
prodLog.info('✅ Cache cleared - starting fresh')
|
||||||
|
|
||||||
|
// v7.3.0: Progressive vs Strict initialization
|
||||||
|
if (effectiveMode === 'progressive') {
|
||||||
|
// PROGRESSIVE MODE: Fast init, background validation
|
||||||
|
// Mark as initialized immediately - ready to accept operations
|
||||||
|
// Bucket validation happens lazily on first write
|
||||||
|
// Count loading happens in background
|
||||||
|
|
||||||
|
prodLog.info(`✅ GCS progressive init complete: ${this.bucketName} (validation deferred)`)
|
||||||
|
|
||||||
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
|
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
|
||||||
await super.init()
|
await super.init()
|
||||||
|
|
||||||
|
// Schedule background tasks (non-blocking)
|
||||||
|
this.scheduleBackgroundInit()
|
||||||
|
} else {
|
||||||
|
// STRICT MODE: Traditional blocking initialization
|
||||||
|
// Verify bucket exists and is accessible (blocking)
|
||||||
|
const [exists] = await this.bucket.exists()
|
||||||
|
if (!exists) {
|
||||||
|
throw new Error(`Bucket ${this.bucketName} does not exist or is not accessible`)
|
||||||
|
}
|
||||||
|
this.bucketValidated = true
|
||||||
|
|
||||||
|
prodLog.info(`✅ Connected to GCS bucket: ${this.bucketName}`)
|
||||||
|
|
||||||
|
// Initialize counts from storage (blocking)
|
||||||
|
await this.initializeCounts()
|
||||||
|
this.countsLoaded = true
|
||||||
|
|
||||||
|
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
|
||||||
|
await super.init()
|
||||||
|
|
||||||
|
// Mark background tasks as complete (nothing to do in background)
|
||||||
|
this.backgroundTasksComplete = true
|
||||||
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
this.logger.error('Failed to initialize GCS storage:', error)
|
this.logger.error('Failed to initialize GCS storage:', error)
|
||||||
throw new Error(`Failed to initialize GCS storage: ${error}`)
|
throw new Error(`Failed to initialize GCS storage: ${error}`)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// =============================================
|
||||||
|
// Progressive Initialization (v7.3.0+)
|
||||||
|
// =============================================
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Run background initialization tasks for GCS.
|
||||||
|
*
|
||||||
|
* Called in progressive mode after init() returns. Performs:
|
||||||
|
* 1. Bucket validation (in background)
|
||||||
|
* 2. Count loading from storage (in background)
|
||||||
|
*
|
||||||
|
* These tasks don't block the main thread, allowing fast cold starts.
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @override
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected async runBackgroundInit(): Promise<void> {
|
||||||
|
const startTime = Date.now()
|
||||||
|
prodLog.info('[GCS Background] Starting background initialization...')
|
||||||
|
|
||||||
|
// Run validation and count loading in parallel
|
||||||
|
const validationPromise = this.validateBucketInBackground()
|
||||||
|
const countsPromise = this.loadCountsInBackground()
|
||||||
|
|
||||||
|
// Wait for both to complete (but we're already initialized)
|
||||||
|
await Promise.all([validationPromise, countsPromise])
|
||||||
|
|
||||||
|
const elapsed = Date.now() - startTime
|
||||||
|
prodLog.info(`[GCS Background] Background init complete in ${elapsed}ms`)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validate bucket existence in background.
|
||||||
|
*
|
||||||
|
* Stores result in bucketValidated/bucketValidationError for lazy use.
|
||||||
|
*
|
||||||
|
* @private
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
private async validateBucketInBackground(): Promise<void> {
|
||||||
|
try {
|
||||||
|
const [exists] = await this.bucket!.exists()
|
||||||
|
if (exists) {
|
||||||
|
this.bucketValidated = true
|
||||||
|
prodLog.info(`[GCS Background] Bucket validated: ${this.bucketName}`)
|
||||||
|
} else {
|
||||||
|
this.bucketValidationError = new Error(
|
||||||
|
`Bucket ${this.bucketName} does not exist or is not accessible`
|
||||||
|
)
|
||||||
|
prodLog.warn(`[GCS Background] Bucket validation failed: ${this.bucketName}`)
|
||||||
|
}
|
||||||
|
} catch (error: any) {
|
||||||
|
this.bucketValidationError = new Error(
|
||||||
|
`Bucket validation failed: ${error.message || error}`
|
||||||
|
)
|
||||||
|
prodLog.error(`[GCS Background] Bucket validation error:`, error)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Load counts from storage in background.
|
||||||
|
*
|
||||||
|
* Uses the existing initializeCounts() logic but in background.
|
||||||
|
*
|
||||||
|
* @private
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
private async loadCountsInBackground(): Promise<void> {
|
||||||
|
try {
|
||||||
|
await this.initializeCounts()
|
||||||
|
this.countsLoaded = true
|
||||||
|
prodLog.info(`[GCS Background] Counts loaded: ${this.totalNounCount} nouns, ${this.totalVerbCount} verbs`)
|
||||||
|
} catch (error: any) {
|
||||||
|
// Non-fatal in progressive mode - counts start at 0
|
||||||
|
prodLog.warn(`[GCS Background] Failed to load counts (starting at 0): ${error.message}`)
|
||||||
|
this.countsLoaded = true // Mark as loaded even on error (0 is valid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Ensure bucket is validated before write operations.
|
||||||
|
*
|
||||||
|
* In progressive mode, bucket validation is deferred until the first
|
||||||
|
* write operation. This method validates the bucket and caches the
|
||||||
|
* result (or error) for subsequent calls.
|
||||||
|
*
|
||||||
|
* @throws Error if bucket does not exist or is not accessible
|
||||||
|
* @protected
|
||||||
|
* @override
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected async ensureValidatedForWrite(): Promise<void> {
|
||||||
|
// If already validated, nothing to do
|
||||||
|
if (this.bucketValidated) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// If we have a cached validation error from background init, throw it
|
||||||
|
if (this.bucketValidationError) {
|
||||||
|
throw this.bucketValidationError
|
||||||
|
}
|
||||||
|
|
||||||
|
// Perform synchronous validation (first write in progressive mode)
|
||||||
|
try {
|
||||||
|
prodLog.info(`[GCS] Lazy bucket validation on first write: ${this.bucketName}`)
|
||||||
|
const [exists] = await this.bucket!.exists()
|
||||||
|
if (!exists) {
|
||||||
|
const error = new Error(`Bucket ${this.bucketName} does not exist or is not accessible`)
|
||||||
|
this.bucketValidationError = error
|
||||||
|
throw error
|
||||||
|
}
|
||||||
|
this.bucketValidated = true
|
||||||
|
prodLog.info(`[GCS] Bucket validated successfully: ${this.bucketName}`)
|
||||||
|
} catch (error: any) {
|
||||||
|
// Cache the error for fast-fail on subsequent writes
|
||||||
|
if (!this.bucketValidationError) {
|
||||||
|
this.bucketValidationError = error
|
||||||
|
}
|
||||||
|
throw error
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get the GCS object key for a noun using UUID-based sharding
|
* Get the GCS object key for a noun using UUID-based sharding
|
||||||
*
|
*
|
||||||
|
|
@ -423,8 +656,13 @@ export class GcsStorage extends BaseStorage {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Save a node directly to GCS (bypass buffer)
|
* Save a node directly to GCS (bypass buffer)
|
||||||
|
*
|
||||||
|
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
|
||||||
*/
|
*/
|
||||||
private async saveNodeDirect(node: HNSWNode): Promise<void> {
|
private async saveNodeDirect(node: HNSWNode): Promise<void> {
|
||||||
|
// v7.3.0: Lazy bucket validation for progressive init
|
||||||
|
await this.ensureValidatedForWrite()
|
||||||
|
|
||||||
// Apply backpressure before starting operation
|
// Apply backpressure before starting operation
|
||||||
const requestId = await this.applyBackpressure()
|
const requestId = await this.applyBackpressure()
|
||||||
|
|
||||||
|
|
@ -598,10 +836,14 @@ export class GcsStorage extends BaseStorage {
|
||||||
/**
|
/**
|
||||||
* Write an object to a specific path in GCS
|
* Write an object to a specific path in GCS
|
||||||
* Primitive operation required by base class
|
* Primitive operation required by base class
|
||||||
|
*
|
||||||
|
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
|
||||||
* @protected
|
* @protected
|
||||||
*/
|
*/
|
||||||
protected async writeObjectToPath(path: string, data: any): Promise<void> {
|
protected async writeObjectToPath(path: string, data: any): Promise<void> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
// v7.3.0: Lazy bucket validation for progressive init
|
||||||
|
await this.ensureValidatedForWrite()
|
||||||
|
|
||||||
try {
|
try {
|
||||||
this.logger.trace(`Writing object to path: ${path}`)
|
this.logger.trace(`Writing object to path: ${path}`)
|
||||||
|
|
@ -741,10 +983,14 @@ export class GcsStorage extends BaseStorage {
|
||||||
/**
|
/**
|
||||||
* Delete an object from a specific path in GCS
|
* Delete an object from a specific path in GCS
|
||||||
* Primitive operation required by base class
|
* Primitive operation required by base class
|
||||||
|
*
|
||||||
|
* v7.3.0: Performs lazy bucket validation on first delete in progressive mode.
|
||||||
* @protected
|
* @protected
|
||||||
*/
|
*/
|
||||||
protected async deleteObjectFromPath(path: string): Promise<void> {
|
protected async deleteObjectFromPath(path: string): Promise<void> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
// v7.3.0: Lazy bucket validation for progressive init
|
||||||
|
await this.ensureValidatedForWrite()
|
||||||
|
|
||||||
try {
|
try {
|
||||||
this.logger.trace(`Deleting object at path: ${path}`)
|
this.logger.trace(`Deleting object at path: ${path}`)
|
||||||
|
|
@ -814,8 +1060,13 @@ export class GcsStorage extends BaseStorage {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Save an edge directly to GCS (bypass buffer)
|
* Save an edge directly to GCS (bypass buffer)
|
||||||
|
*
|
||||||
|
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
|
||||||
*/
|
*/
|
||||||
private async saveEdgeDirect(edge: Edge): Promise<void> {
|
private async saveEdgeDirect(edge: Edge): Promise<void> {
|
||||||
|
// v7.3.0: Lazy bucket validation for progressive init
|
||||||
|
await this.ensureValidatedForWrite()
|
||||||
|
|
||||||
const requestId = await this.applyBackpressure()
|
const requestId = await this.applyBackpressure()
|
||||||
|
|
||||||
try {
|
try {
|
||||||
|
|
|
||||||
|
|
@ -40,6 +40,7 @@ import { getGlobalBackpressure } from '../../utils/adaptiveBackpressure.js'
|
||||||
import { getWriteBuffer, WriteBuffer } from '../../utils/writeBuffer.js'
|
import { getWriteBuffer, WriteBuffer } from '../../utils/writeBuffer.js'
|
||||||
import { getCoalescer, RequestCoalescer } from '../../utils/requestCoalescer.js'
|
import { getCoalescer, RequestCoalescer } from '../../utils/requestCoalescer.js'
|
||||||
import { getShardIdFromUuid, getAllShardIds, getShardIdByIndex, TOTAL_SHARDS } from '../sharding.js'
|
import { getShardIdFromUuid, getAllShardIds, getShardIdByIndex, TOTAL_SHARDS } from '../sharding.js'
|
||||||
|
import { InitMode } from './baseStorageAdapter.js'
|
||||||
|
|
||||||
// Type aliases for better readability
|
// Type aliases for better readability
|
||||||
type HNSWNode = HNSWNoun
|
type HNSWNode = HNSWNoun
|
||||||
|
|
@ -169,7 +170,28 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initialize the storage adapter
|
* Initialize the storage adapter
|
||||||
|
*
|
||||||
* @param options Configuration options for the S3-compatible storage
|
* @param options Configuration options for the S3-compatible storage
|
||||||
|
*
|
||||||
|
* @example Zero-config (recommended) - auto-detects Lambda/Cloud Run for fast init
|
||||||
|
* ```typescript
|
||||||
|
* const storage = new S3CompatibleStorage({
|
||||||
|
* bucketName: 'my-bucket',
|
||||||
|
* accessKeyId: 'key',
|
||||||
|
* secretAccessKey: 'secret'
|
||||||
|
* })
|
||||||
|
* await storage.init() // <200ms in Lambda, blocking locally
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* @example Force progressive mode for all environments
|
||||||
|
* ```typescript
|
||||||
|
* const storage = new S3CompatibleStorage({
|
||||||
|
* bucketName: 'my-bucket',
|
||||||
|
* accessKeyId: 'key',
|
||||||
|
* secretAccessKey: 'secret',
|
||||||
|
* initMode: 'progressive' // Always <200ms init
|
||||||
|
* })
|
||||||
|
* ```
|
||||||
*/
|
*/
|
||||||
constructor(options: {
|
constructor(options: {
|
||||||
bucketName: string
|
bucketName: string
|
||||||
|
|
@ -186,6 +208,21 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
hotCacheEvictionThreshold?: number
|
hotCacheEvictionThreshold?: number
|
||||||
warmCacheTTL?: number
|
warmCacheTTL?: number
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Initialization mode for fast cold starts (v7.3.0+)
|
||||||
|
*
|
||||||
|
* - `'auto'` (default): Progressive in cloud environments (Lambda, Cloud Run),
|
||||||
|
* strict locally. Zero-config optimization.
|
||||||
|
* - `'progressive'`: Always use fast init (<200ms). Bucket validation and
|
||||||
|
* count loading happen in background. First write validates bucket.
|
||||||
|
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
|
||||||
|
* before init() returns.
|
||||||
|
*
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
initMode?: InitMode
|
||||||
|
|
||||||
readOnly?: boolean
|
readOnly?: boolean
|
||||||
}) {
|
}) {
|
||||||
super()
|
super()
|
||||||
|
|
@ -199,6 +236,11 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
this.serviceType = options.serviceType || 's3'
|
this.serviceType = options.serviceType || 's3'
|
||||||
this.readOnly = options.readOnly || false
|
this.readOnly = options.readOnly || false
|
||||||
|
|
||||||
|
// v7.3.0: Handle initMode
|
||||||
|
if (options.initMode) {
|
||||||
|
this.initMode = options.initMode
|
||||||
|
}
|
||||||
|
|
||||||
// Initialize operation executors with timeout and retry configuration
|
// Initialize operation executors with timeout and retry configuration
|
||||||
this.operationExecutors = new StorageOperationExecutors(
|
this.operationExecutors = new StorageOperationExecutors(
|
||||||
options.operationConfig
|
options.operationConfig
|
||||||
|
|
@ -315,6 +357,22 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Initialize the storage adapter
|
* Initialize the storage adapter
|
||||||
|
*
|
||||||
|
* v7.3.0: Supports progressive initialization for fast cold starts
|
||||||
|
*
|
||||||
|
* | Mode | Init Time | When |
|
||||||
|
* |------|-----------|------|
|
||||||
|
* | `progressive` | <200ms | Lambda, Cloud Run, Azure Functions |
|
||||||
|
* | `strict` | 100-500ms+ | Local development, tests |
|
||||||
|
* | `auto` | Detected | Default - best of both |
|
||||||
|
*
|
||||||
|
* In progressive mode:
|
||||||
|
* - SDK import and client creation: ~50ms (unavoidable)
|
||||||
|
* - Write buffers and caches: ~10ms
|
||||||
|
* - Mark as initialized: READY
|
||||||
|
* - Background: validate bucket, load counts
|
||||||
|
*
|
||||||
|
* First write operation validates bucket existence (lazy validation).
|
||||||
*/
|
*/
|
||||||
public async init(): Promise<void> {
|
public async init(): Promise<void> {
|
||||||
if (this.isInitialized) {
|
if (this.isInitialized) {
|
||||||
|
|
@ -322,7 +380,7 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
// Import AWS SDK modules only when needed
|
// Import AWS SDK modules only when needed (~50ms)
|
||||||
const { S3Client } = await import('@aws-sdk/client-s3')
|
const { S3Client } = await import('@aws-sdk/client-s3')
|
||||||
|
|
||||||
// Configure the S3 client based on the service type
|
// Configure the S3 client based on the service type
|
||||||
|
|
@ -354,16 +412,14 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
clientConfig.endpoint = `https://${this.accountId}.r2.cloudflarestorage.com`
|
clientConfig.endpoint = `https://${this.accountId}.r2.cloudflarestorage.com`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Create the S3 client
|
// Create the S3 client (no network calls)
|
||||||
this.s3Client = new S3Client(clientConfig)
|
this.s3Client = new S3Client(clientConfig)
|
||||||
|
|
||||||
// Ensure the bucket exists and is accessible
|
// Determine initialization mode
|
||||||
const { HeadBucketCommand } = await import('@aws-sdk/client-s3')
|
const effectiveMode = this.resolveInitMode()
|
||||||
await this.s3Client.send(
|
const isCloud = this.detectCloudEnvironment()
|
||||||
new HeadBucketCommand({
|
|
||||||
Bucket: this.bucketName
|
prodLog.info(`🚀 S3 init mode: ${effectiveMode} (detected cloud: ${isCloud})`)
|
||||||
})
|
|
||||||
)
|
|
||||||
|
|
||||||
// Create storage adapter proxies for the cache managers
|
// Create storage adapter proxies for the cache managers
|
||||||
const nounStorageAdapter = {
|
const nounStorageAdapter = {
|
||||||
|
|
@ -458,12 +514,6 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
// Initialize request coalescer
|
// Initialize request coalescer
|
||||||
this.initializeCoalescer()
|
this.initializeCoalescer()
|
||||||
|
|
||||||
// Auto-cleanup legacy /index folder on initialization
|
|
||||||
await this.cleanupLegacyIndexFolder()
|
|
||||||
|
|
||||||
// Initialize counts from storage
|
|
||||||
await this.initializeCounts()
|
|
||||||
|
|
||||||
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs
|
// CRITICAL FIX (v3.37.7): Clear any stale cache entries from previous runs
|
||||||
// This prevents cache poisoning from causing silent failures on container restart
|
// This prevents cache poisoning from causing silent failures on container restart
|
||||||
const nodeCacheSize = this.nodeCache?.size || 0
|
const nodeCacheSize = this.nodeCache?.size || 0
|
||||||
|
|
@ -474,8 +524,45 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
prodLog.info('🧹 Node cache is empty - starting fresh')
|
prodLog.info('🧹 Node cache is empty - starting fresh')
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// v7.3.0: Progressive vs Strict initialization
|
||||||
|
if (effectiveMode === 'progressive') {
|
||||||
|
// PROGRESSIVE MODE: Fast init, background validation
|
||||||
|
// Mark as initialized immediately - ready to accept operations
|
||||||
|
// Bucket validation happens lazily on first write
|
||||||
|
// Count loading happens in background
|
||||||
|
|
||||||
|
prodLog.info(`✅ S3 progressive init complete: ${this.bucketName} (validation deferred)`)
|
||||||
|
|
||||||
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
|
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
|
||||||
await super.init()
|
await super.init()
|
||||||
|
|
||||||
|
// Schedule background tasks (non-blocking)
|
||||||
|
this.scheduleBackgroundInit()
|
||||||
|
} else {
|
||||||
|
// STRICT MODE: Traditional blocking initialization
|
||||||
|
// Ensure the bucket exists and is accessible (blocking)
|
||||||
|
const { HeadBucketCommand } = await import('@aws-sdk/client-s3')
|
||||||
|
await this.s3Client.send(
|
||||||
|
new HeadBucketCommand({
|
||||||
|
Bucket: this.bucketName
|
||||||
|
})
|
||||||
|
)
|
||||||
|
this.bucketValidated = true
|
||||||
|
|
||||||
|
// Auto-cleanup legacy /index folder on initialization
|
||||||
|
await this.cleanupLegacyIndexFolder()
|
||||||
|
|
||||||
|
// Initialize counts from storage (blocking)
|
||||||
|
await this.initializeCounts()
|
||||||
|
this.countsLoaded = true
|
||||||
|
|
||||||
|
// v6.0.0: Initialize GraphAdjacencyIndex and type statistics
|
||||||
|
await super.init()
|
||||||
|
|
||||||
|
// Mark background tasks as complete (nothing to do in background)
|
||||||
|
this.backgroundTasksComplete = true
|
||||||
|
}
|
||||||
|
|
||||||
this.logger.info(`Initialized ${this.serviceType} storage with bucket ${this.bucketName}`)
|
this.logger.info(`Initialized ${this.serviceType} storage with bucket ${this.bucketName}`)
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
this.logger.error(`Failed to initialize ${this.serviceType} storage:`, error)
|
this.logger.error(`Failed to initialize ${this.serviceType} storage:`, error)
|
||||||
|
|
@ -485,6 +572,130 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// =============================================
|
||||||
|
// Progressive Initialization (v7.3.0+)
|
||||||
|
// =============================================
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Run background initialization tasks for S3.
|
||||||
|
*
|
||||||
|
* Called in progressive mode after init() returns. Performs:
|
||||||
|
* 1. Bucket validation (in background)
|
||||||
|
* 2. Legacy folder cleanup (in background)
|
||||||
|
* 3. Count loading from storage (in background)
|
||||||
|
*
|
||||||
|
* These tasks don't block the main thread, allowing fast cold starts.
|
||||||
|
*
|
||||||
|
* @protected
|
||||||
|
* @override
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected async runBackgroundInit(): Promise<void> {
|
||||||
|
const startTime = Date.now()
|
||||||
|
prodLog.info('[S3 Background] Starting background initialization...')
|
||||||
|
|
||||||
|
// Run validation, cleanup, and count loading in parallel
|
||||||
|
const validationPromise = this.validateBucketInBackground()
|
||||||
|
const cleanupPromise = this.cleanupLegacyIndexFolder().catch((err) => {
|
||||||
|
prodLog.warn(`[S3 Background] Legacy cleanup failed (non-fatal): ${err.message}`)
|
||||||
|
})
|
||||||
|
const countsPromise = this.loadCountsInBackground()
|
||||||
|
|
||||||
|
// Wait for all to complete (but we're already initialized)
|
||||||
|
await Promise.all([validationPromise, cleanupPromise, countsPromise])
|
||||||
|
|
||||||
|
const elapsed = Date.now() - startTime
|
||||||
|
prodLog.info(`[S3 Background] Background init complete in ${elapsed}ms`)
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validate bucket existence in background.
|
||||||
|
*
|
||||||
|
* Stores result in bucketValidated/bucketValidationError for lazy use.
|
||||||
|
*
|
||||||
|
* @private
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
private async validateBucketInBackground(): Promise<void> {
|
||||||
|
try {
|
||||||
|
const { HeadBucketCommand } = await import('@aws-sdk/client-s3')
|
||||||
|
await this.s3Client!.send(
|
||||||
|
new HeadBucketCommand({
|
||||||
|
Bucket: this.bucketName
|
||||||
|
})
|
||||||
|
)
|
||||||
|
this.bucketValidated = true
|
||||||
|
prodLog.info(`[S3 Background] Bucket validated: ${this.bucketName}`)
|
||||||
|
} catch (error: any) {
|
||||||
|
this.bucketValidationError = new Error(
|
||||||
|
`Bucket ${this.bucketName} does not exist or is not accessible: ${error.message || error}`
|
||||||
|
)
|
||||||
|
prodLog.warn(`[S3 Background] Bucket validation failed: ${this.bucketName}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Load counts from storage in background.
|
||||||
|
*
|
||||||
|
* @private
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
private async loadCountsInBackground(): Promise<void> {
|
||||||
|
try {
|
||||||
|
await this.initializeCounts()
|
||||||
|
this.countsLoaded = true
|
||||||
|
prodLog.info(`[S3 Background] Counts loaded: ${this.totalNounCount} nouns, ${this.totalVerbCount} verbs`)
|
||||||
|
} catch (error: any) {
|
||||||
|
// Non-fatal in progressive mode - counts start at 0
|
||||||
|
prodLog.warn(`[S3 Background] Failed to load counts (starting at 0): ${error.message}`)
|
||||||
|
this.countsLoaded = true // Mark as loaded even on error (0 is valid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Ensure bucket is validated before write operations.
|
||||||
|
*
|
||||||
|
* In progressive mode, bucket validation is deferred until the first
|
||||||
|
* write operation. This method validates the bucket and caches the
|
||||||
|
* result (or error) for subsequent calls.
|
||||||
|
*
|
||||||
|
* @throws Error if bucket does not exist or is not accessible
|
||||||
|
* @protected
|
||||||
|
* @override
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
protected async ensureValidatedForWrite(): Promise<void> {
|
||||||
|
// If already validated, nothing to do
|
||||||
|
if (this.bucketValidated) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// If we have a cached validation error from background init, throw it
|
||||||
|
if (this.bucketValidationError) {
|
||||||
|
throw this.bucketValidationError
|
||||||
|
}
|
||||||
|
|
||||||
|
// Perform synchronous validation (first write in progressive mode)
|
||||||
|
try {
|
||||||
|
prodLog.info(`[S3] Lazy bucket validation on first write: ${this.bucketName}`)
|
||||||
|
const { HeadBucketCommand } = await import('@aws-sdk/client-s3')
|
||||||
|
await this.s3Client!.send(
|
||||||
|
new HeadBucketCommand({
|
||||||
|
Bucket: this.bucketName
|
||||||
|
})
|
||||||
|
)
|
||||||
|
this.bucketValidated = true
|
||||||
|
prodLog.info(`[S3] Bucket validated successfully: ${this.bucketName}`)
|
||||||
|
} catch (error: any) {
|
||||||
|
// Cache the error for fast-fail on subsequent writes
|
||||||
|
const wrappedError = new Error(
|
||||||
|
`Bucket ${this.bucketName} does not exist or is not accessible: ${error.message || error}`
|
||||||
|
)
|
||||||
|
this.bucketValidationError = wrappedError
|
||||||
|
throw wrappedError
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Set distributed components for multi-node coordination
|
* Set distributed components for multi-node coordination
|
||||||
*
|
*
|
||||||
|
|
@ -1588,9 +1799,13 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
/**
|
/**
|
||||||
* Primitive operation: Write object to path
|
* Primitive operation: Write object to path
|
||||||
* All metadata operations use this internally via base class routing
|
* All metadata operations use this internally via base class routing
|
||||||
|
*
|
||||||
|
* v7.3.0: Performs lazy bucket validation on first write in progressive mode.
|
||||||
*/
|
*/
|
||||||
protected async writeObjectToPath(path: string, data: any): Promise<void> {
|
protected async writeObjectToPath(path: string, data: any): Promise<void> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
// v7.3.0: Lazy bucket validation for progressive init
|
||||||
|
await this.ensureValidatedForWrite()
|
||||||
|
|
||||||
// Apply backpressure before starting operation
|
// Apply backpressure before starting operation
|
||||||
const requestId = await this.applyBackpressure()
|
const requestId = await this.applyBackpressure()
|
||||||
|
|
@ -1681,9 +1896,13 @@ export class S3CompatibleStorage extends BaseStorage {
|
||||||
/**
|
/**
|
||||||
* Primitive operation: Delete object from path
|
* Primitive operation: Delete object from path
|
||||||
* All metadata operations use this internally via base class routing
|
* All metadata operations use this internally via base class routing
|
||||||
|
*
|
||||||
|
* v7.3.0: Performs lazy bucket validation on first delete in progressive mode.
|
||||||
*/
|
*/
|
||||||
protected async deleteObjectFromPath(path: string): Promise<void> {
|
protected async deleteObjectFromPath(path: string): Promise<void> {
|
||||||
await this.ensureInitialized()
|
await this.ensureInitialized()
|
||||||
|
// v7.3.0: Lazy bucket validation for progressive init
|
||||||
|
await this.ensureValidatedForWrite()
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const { DeleteObjectCommand } = await import('@aws-sdk/client-s3')
|
const { DeleteObjectCommand } = await import('@aws-sdk/client-s3')
|
||||||
|
|
|
||||||
|
|
@ -92,6 +92,20 @@ export interface StorageOptions {
|
||||||
* AWS session token (optional)
|
* AWS session token (optional)
|
||||||
*/
|
*/
|
||||||
sessionToken?: string
|
sessionToken?: string
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Initialization mode for fast cold starts (v7.3.0+)
|
||||||
|
*
|
||||||
|
* - `'auto'` (default): Progressive in cloud environments (Lambda),
|
||||||
|
* strict locally. Zero-config optimization.
|
||||||
|
* - `'progressive'`: Always use fast init (<200ms). Bucket validation and
|
||||||
|
* count loading happen in background. First write validates bucket.
|
||||||
|
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
|
||||||
|
* before init() returns.
|
||||||
|
*
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
initMode?: 'progressive' | 'strict' | 'auto'
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -196,8 +210,23 @@ export interface StorageOptions {
|
||||||
* Skip loading and saving the counts file entirely
|
* Skip loading and saving the counts file entirely
|
||||||
* Useful for very large datasets where counts aren't critical
|
* Useful for very large datasets where counts aren't critical
|
||||||
* @default false
|
* @default false
|
||||||
|
* @deprecated Use `initMode: 'progressive'` instead
|
||||||
*/
|
*/
|
||||||
skipCountsFile?: boolean
|
skipCountsFile?: boolean
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Initialization mode for fast cold starts (v7.3.0+)
|
||||||
|
*
|
||||||
|
* - `'auto'` (default): Progressive in cloud environments (Cloud Run),
|
||||||
|
* strict locally. Zero-config optimization.
|
||||||
|
* - `'progressive'`: Always use fast init (<200ms). Bucket validation and
|
||||||
|
* count loading happen in background. First write validates bucket.
|
||||||
|
* - `'strict'`: Traditional blocking init. Validates bucket and loads counts
|
||||||
|
* before init() returns.
|
||||||
|
*
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
initMode?: 'progressive' | 'strict' | 'auto'
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -228,6 +257,20 @@ export interface StorageOptions {
|
||||||
* SAS token (optional, alternative to account key)
|
* SAS token (optional, alternative to account key)
|
||||||
*/
|
*/
|
||||||
sasToken?: string
|
sasToken?: string
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Initialization mode for fast cold starts (v7.3.0+)
|
||||||
|
*
|
||||||
|
* - `'auto'` (default): Progressive in cloud environments (Azure Functions),
|
||||||
|
* strict locally. Zero-config optimization.
|
||||||
|
* - `'progressive'`: Always use fast init (<200ms). Container validation and
|
||||||
|
* count loading happen in background. First write validates container.
|
||||||
|
* - `'strict'`: Traditional blocking init. Validates container and loads counts
|
||||||
|
* before init() returns.
|
||||||
|
*
|
||||||
|
* @since v7.3.0
|
||||||
|
*/
|
||||||
|
initMode?: 'progressive' | 'strict' | 'auto'
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -504,6 +547,7 @@ export async function createStorage(
|
||||||
accessKeyId: options.s3Storage.accessKeyId,
|
accessKeyId: options.s3Storage.accessKeyId,
|
||||||
secretAccessKey: options.s3Storage.secretAccessKey,
|
secretAccessKey: options.s3Storage.secretAccessKey,
|
||||||
sessionToken: options.s3Storage.sessionToken,
|
sessionToken: options.s3Storage.sessionToken,
|
||||||
|
initMode: options.s3Storage.initMode,
|
||||||
serviceType: 's3',
|
serviceType: 's3',
|
||||||
operationConfig: options.operationConfig,
|
operationConfig: options.operationConfig,
|
||||||
cacheConfig: options.cacheConfig
|
cacheConfig: options.cacheConfig
|
||||||
|
|
@ -597,6 +641,7 @@ export async function createStorage(
|
||||||
secretAccessKey: config.secretAccessKey,
|
secretAccessKey: config.secretAccessKey,
|
||||||
skipInitialScan: gcsNative?.skipInitialScan,
|
skipInitialScan: gcsNative?.skipInitialScan,
|
||||||
skipCountsFile: gcsNative?.skipCountsFile,
|
skipCountsFile: gcsNative?.skipCountsFile,
|
||||||
|
initMode: gcsNative?.initMode,
|
||||||
cacheConfig: options.cacheConfig
|
cacheConfig: options.cacheConfig
|
||||||
})
|
})
|
||||||
configureCOW(storage, options)
|
configureCOW(storage, options)
|
||||||
|
|
@ -612,6 +657,7 @@ export async function createStorage(
|
||||||
accountKey: options.azureStorage.accountKey,
|
accountKey: options.azureStorage.accountKey,
|
||||||
connectionString: options.azureStorage.connectionString,
|
connectionString: options.azureStorage.connectionString,
|
||||||
sasToken: options.azureStorage.sasToken,
|
sasToken: options.azureStorage.sasToken,
|
||||||
|
initMode: options.azureStorage.initMode,
|
||||||
cacheConfig: options.cacheConfig
|
cacheConfig: options.cacheConfig
|
||||||
})
|
})
|
||||||
configureCOW(storage, options)
|
configureCOW(storage, options)
|
||||||
|
|
@ -691,6 +737,7 @@ export async function createStorage(
|
||||||
accessKeyId: options.s3Storage.accessKeyId,
|
accessKeyId: options.s3Storage.accessKeyId,
|
||||||
secretAccessKey: options.s3Storage.secretAccessKey,
|
secretAccessKey: options.s3Storage.secretAccessKey,
|
||||||
sessionToken: options.s3Storage.sessionToken,
|
sessionToken: options.s3Storage.sessionToken,
|
||||||
|
initMode: options.s3Storage.initMode,
|
||||||
serviceType: 's3',
|
serviceType: 's3',
|
||||||
cacheConfig: options.cacheConfig
|
cacheConfig: options.cacheConfig
|
||||||
})
|
})
|
||||||
|
|
@ -738,6 +785,7 @@ export async function createStorage(
|
||||||
secretAccessKey: config.secretAccessKey,
|
secretAccessKey: config.secretAccessKey,
|
||||||
skipInitialScan: gcsNative?.skipInitialScan,
|
skipInitialScan: gcsNative?.skipInitialScan,
|
||||||
skipCountsFile: gcsNative?.skipCountsFile,
|
skipCountsFile: gcsNative?.skipCountsFile,
|
||||||
|
initMode: gcsNative?.initMode,
|
||||||
cacheConfig: options.cacheConfig
|
cacheConfig: options.cacheConfig
|
||||||
})
|
})
|
||||||
configureCOW(storage, options)
|
configureCOW(storage, options)
|
||||||
|
|
@ -753,6 +801,7 @@ export async function createStorage(
|
||||||
accountKey: options.azureStorage.accountKey,
|
accountKey: options.azureStorage.accountKey,
|
||||||
connectionString: options.azureStorage.connectionString,
|
connectionString: options.azureStorage.connectionString,
|
||||||
sasToken: options.azureStorage.sasToken,
|
sasToken: options.azureStorage.sasToken,
|
||||||
|
initMode: options.azureStorage.initMode,
|
||||||
cacheConfig: options.cacheConfig
|
cacheConfig: options.cacheConfig
|
||||||
})
|
})
|
||||||
configureCOW(storage, options)
|
configureCOW(storage, options)
|
||||||
|
|
|
||||||
|
|
@ -17,6 +17,28 @@ export interface BrainyInterface<T = unknown> {
|
||||||
*/
|
*/
|
||||||
init(): Promise<void>
|
init(): Promise<void>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Promise that resolves when initialization is complete (v7.3.0+)
|
||||||
|
* Can be awaited multiple times safely.
|
||||||
|
*/
|
||||||
|
readonly ready: Promise<void>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if basic initialization is complete
|
||||||
|
*/
|
||||||
|
readonly isInitialized: boolean
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if all initialization including background tasks is complete (v7.3.0+)
|
||||||
|
*/
|
||||||
|
isFullyInitialized(): boolean
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Wait for all background initialization tasks to complete (v7.3.0+)
|
||||||
|
* For cloud storage adapters, this waits for bucket validation and count sync.
|
||||||
|
*/
|
||||||
|
awaitBackgroundInit(): Promise<void>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Modern add method - unified entity creation
|
* Modern add method - unified entity creation
|
||||||
* @param params Parameters for adding entities
|
* @param params Parameters for adding entities
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue