feat: add native Google Cloud Storage adapter with ADC support
Implement native @google-cloud/storage adapter for better performance
and easier authentication in Cloud Run/GCE environments.
Features:
- Application Default Credentials (ADC) for zero-config auth
- Service account authentication (keyFilename, credentials)
- HMAC fallback for backward compatibility
- Full UUID-based sharding preservation
- Write buffers for high-volume mode
- Multi-level caching and adaptive backpressure
- Complete feature parity with S3-compatible adapter
Benefits over S3-compatible GCS:
- No HMAC key management required
- Native SDK performance optimizations
- Automatic authentication in Cloud Run/GCE
- Simpler configuration
Configuration:
- type: 'gcs-native'
- gcsNativeStorage: { bucketName, keyFilename?, credentials? }
- Zero data migration required (same path structure)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
8a9cf1bd51
commit
e2aa8e3253
6 changed files with 2824 additions and 35 deletions
|
|
@ -548,6 +548,185 @@ const valid = await security.verify(data, signature, secret)
|
|||
|
||||
---
|
||||
|
||||
## Storage Configuration
|
||||
|
||||
Brainy supports multiple storage backends for different deployment scenarios.
|
||||
|
||||
### Storage Types
|
||||
|
||||
#### Memory Storage (Default)
|
||||
Fast in-memory storage, ideal for testing and development.
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: { type: 'memory' }
|
||||
})
|
||||
```
|
||||
|
||||
#### File System Storage
|
||||
Persistent local storage using the filesystem.
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './brainy-data'
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
#### Amazon S3 Storage
|
||||
Scalable cloud storage with S3.
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
s3Storage: {
|
||||
bucketName: 'my-bucket',
|
||||
region: 'us-east-1',
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
#### Cloudflare R2 Storage
|
||||
Scalable cloud storage with Cloudflare R2 (S3-compatible).
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'r2',
|
||||
r2Storage: {
|
||||
bucketName: 'my-bucket',
|
||||
accountId: process.env.CF_ACCOUNT_ID,
|
||||
accessKeyId: process.env.R2_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
#### Google Cloud Storage (S3-Compatible)
|
||||
GCS using HMAC keys for S3-compatible access.
|
||||
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsStorage: {
|
||||
bucketName: 'my-bucket',
|
||||
region: 'us-central1',
|
||||
accessKeyId: process.env.GCS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY,
|
||||
endpoint: 'https://storage.googleapis.com'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
#### Google Cloud Storage (Native SDK) 🆕
|
||||
**Recommended for GCS deployments.** Uses native `@google-cloud/storage` SDK with automatic authentication.
|
||||
|
||||
**Key Benefits:**
|
||||
- ✅ Application Default Credentials (ADC) - Zero config in Cloud Run/GCE
|
||||
- ✅ Better performance with native GCS optimizations
|
||||
- ✅ No HMAC key management required
|
||||
- ✅ Automatic service account integration
|
||||
|
||||
**With Application Default Credentials (Cloud Run/GCE):**
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs-native',
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-bucket'
|
||||
// No credentials needed - ADC automatic!
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**With Service Account Key File:**
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs-native',
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-bucket',
|
||||
keyFilename: '/path/to/service-account.json'
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**With Service Account Credentials Object:**
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs-native',
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-bucket',
|
||||
credentials: {
|
||||
client_email: 'service@project.iam.gserviceaccount.com',
|
||||
private_key: process.env.GCS_PRIVATE_KEY
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Storage Features
|
||||
|
||||
All storage adapters support:
|
||||
- ✅ **UUID-based sharding** - 256 buckets (00-ff) for scalability
|
||||
- ✅ **Pagination** - Efficient cursor-based pagination across shards
|
||||
- ✅ **Statistics** - O(1) count operations
|
||||
- ✅ **Caching** - Multi-level cache for performance
|
||||
- ✅ **Backpressure** - Automatic flow control
|
||||
- ✅ **Throttling detection** - Adaptive retry on rate limits
|
||||
|
||||
### Migration from HMAC to Native GCS
|
||||
|
||||
If you're currently using `type: 'gcs'` with HMAC keys, migrating to `type: 'gcs-native'` is straightforward:
|
||||
|
||||
**Before (HMAC):**
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
gcsStorage: {
|
||||
bucketName: 'my-bucket',
|
||||
accessKeyId: process.env.GCS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**After (Native with ADC):**
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs-native',
|
||||
gcsNativeStorage: {
|
||||
bucketName: 'my-bucket'
|
||||
// ADC handles authentication automatically
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Data Migration:**
|
||||
No data migration required! Both adapters use the same path structure:
|
||||
- `entities/nouns/vectors/{shard}/{id}.json`
|
||||
- `entities/nouns/metadata/{shard}/{id}.json`
|
||||
- `entities/verbs/vectors/{shard}/{id}.json`
|
||||
|
||||
---
|
||||
|
||||
## Configuration API
|
||||
|
||||
Access via: `const config = await brain.config()`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue