docs(8.0): Phase F — deep clean across 21 docs
Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.
Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
sections replaced with single-node vector tuning + filesystem framing.
Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md
Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md
MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
parent
2626ab8d62
commit
adda1570f3
22 changed files with 570 additions and 2658 deletions
|
|
@ -1,46 +1,46 @@
|
|||
# Storage Architecture
|
||||
|
||||
> **Updated**: Metadata/vector separation, UUID-based sharding, lifecycle management
|
||||
> **Updated**: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup
|
||||
|
||||
## Storage Structure
|
||||
|
||||
### Architecture: Metadata/Vector Separation
|
||||
|
||||
entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
|
||||
Entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
├── _system/ # System metadata (not sharded)
|
||||
│ ├── statistics.json # Performance metrics
|
||||
│ ├── __metadata_field_index__*.json # Field indexes
|
||||
│ └── __metadata_sorted_index__*.json # Sorted indexes
|
||||
├── _system/ # System metadata (not sharded)
|
||||
│ ├── statistics.json # Performance metrics
|
||||
│ ├── __metadata_field_index__*.json # Field indexes
|
||||
│ └── __metadata_sorted_index__*.json # Sorted indexes
|
||||
│
|
||||
├── entities/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # HNSW graph data (sharded by UUID)
|
||||
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
|
||||
│ │ │ │ ├── 00123456-....json # Vector + HNSW connections
|
||||
│ │ │ │ └── 00abcdef-....json
|
||||
│ │ │ ├── 01/ ... ff/ # 256 shards total
|
||||
│ │ │
|
||||
│ │ └── metadata/ # Business data (sharded by UUID)
|
||||
│ │ ├── 00/
|
||||
│ │ │ ├── 00123456-....json # Entity metadata only
|
||||
│ │ │ └── 00abcdef-....json
|
||||
│ │ ├── 01/ ... ff/
|
||||
│ │
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (sharded)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ │
|
||||
│ └── metadata/ # Relationship data (sharded)
|
||||
│ ├── 00/ ... ff/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # Vector graph data (sharded by UUID)
|
||||
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
|
||||
│ │ │ │ ├── 00123456-....json # Vector + graph connections
|
||||
│ │ │ │ └── 00abcdef-....json
|
||||
│ │ │ ├── 01/ ... ff/ # 256 shards total
|
||||
│ │ │
|
||||
│ │ └── metadata/ # Business data (sharded by UUID)
|
||||
│ │ ├── 00/
|
||||
│ │ │ ├── 00123456-....json # Entity metadata only
|
||||
│ │ │ └── 00abcdef-....json
|
||||
│ │ ├── 01/ ... ff/
|
||||
│ │
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (sharded)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ │
|
||||
│ └── metadata/ # Relationship data (sharded)
|
||||
│ ├── 00/ ... ff/
|
||||
```
|
||||
|
||||
### Why Split Metadata and Vectors?
|
||||
|
||||
**Performance at scale:**
|
||||
- **HNSW operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
|
||||
- **Vector search operations**: Only load vectors (4KB) during search, not metadata (2-10KB)
|
||||
- **Filtering**: Only load metadata during filtering, not vectors
|
||||
- **Pagination**: Load metadata IDs first, fetch vectors/metadata on-demand
|
||||
- **Result**: 60-70% reduction in I/O for typical queries at million-entity scale
|
||||
|
|
@ -52,115 +52,69 @@ brainy-data/
|
|||
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
|
||||
const shard = uuid.substring(0, 2) // "3f"
|
||||
|
||||
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
|
||||
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
|
||||
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- **Uniform distribution**: ~3,900 entities per shard (at 1M scale)
|
||||
- **Cloud storage optimization**: 200x faster than unsharded (30s → 150ms)
|
||||
- **Parallel operations**: Load 256 shards in parallel
|
||||
- **Filesystem optimization**: avoids huge flat directories that bog down `readdir`
|
||||
- **Parallel operations**: walk 256 shards in parallel
|
||||
- **Predictable**: Deterministic shard assignment
|
||||
|
||||
## Storage Adapters
|
||||
|
||||
Brainy provides multiple storage adapters with identical APIs and production features:
|
||||
Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` interface:
|
||||
|
||||
### FileSystem Storage (Node.js)
|
||||
### FileSystem Storage (Node.js, default)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // Gzip compression (60-80% space savings)
|
||||
}
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './data'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Server applications, CLI tools
|
||||
- **Performance**: Direct file I/O with optional compression
|
||||
- **Use case**: Server applications, CLI tools, single-node deployments
|
||||
- **Performance**: Direct file I/O
|
||||
- **Persistence**: Permanent on disk
|
||||
- **Features**:
|
||||
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
|
||||
- **Batch Delete**: Efficient bulk deletion with retries
|
||||
- **UUID Sharding**: Automatic 256-shard distribution
|
||||
- **Batch Delete**: Efficient bulk deletion with retries
|
||||
- **UUID Sharding**: Automatic 256-shard distribution
|
||||
|
||||
### S3 Compatible Storage (AWS, MinIO, R2)
|
||||
### Memory Storage
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucket: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
credentials: {
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
storage: {
|
||||
type: 'memory'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Distributed applications, cloud deployments
|
||||
- **Performance**: Network dependent, with intelligent caching
|
||||
- **Persistence**: Cloud storage durability (99.999999999%)
|
||||
- **Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
|
||||
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
|
||||
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
|
||||
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
|
||||
- **Use case**: Tests, ephemeral workloads, single-process caches
|
||||
- **Performance**: No I/O — all data lives in process memory
|
||||
- **Persistence**: None — data is lost when the process exits
|
||||
|
||||
### Google Cloud Storage (GCS)
|
||||
### Auto
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
bucketName: 'my-brainy-data',
|
||||
keyFilename: './service-account.json' // Or use ADC
|
||||
}
|
||||
storage: {
|
||||
type: 'auto',
|
||||
rootDirectory: './data'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Google Cloud deployments
|
||||
- **Performance**: Global CDN with edge caching
|
||||
- **Persistence**: 99.999999999% durability
|
||||
- **Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
|
||||
- **Autoclass**: Intelligent automatic tier optimization
|
||||
- **Batch Delete**: Efficient bulk operations
|
||||
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
|
||||
`'auto'` picks `'filesystem'` when running on Node.js with a writable `rootDirectory`, and falls back to `'memory'` otherwise.
|
||||
|
||||
### Azure Blob Storage
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'azure',
|
||||
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
|
||||
containerName: 'brainy-data'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Azure cloud deployments
|
||||
- **Performance**: Global replication with CDN
|
||||
- **Persistence**: LRS, ZRS, GRS, RA-GRS options
|
||||
- **Features**:
|
||||
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
|
||||
- **Lifecycle Policies**: Automatic tier transitions and deletions
|
||||
- **Batch Delete**: BlobBatchClient for efficient bulk operations
|
||||
- **Batch Tier Changes**: Move thousands of blobs efficiently
|
||||
- **Archive Rehydration**: Smart rehydration with priority options
|
||||
## Backup and Off-Site Replication
|
||||
|
||||
### Origin Private File System (Browser)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'opfs'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Browser applications, PWAs
|
||||
- **Performance**: Near-native file system speed
|
||||
- **Persistence**: Permanent in browser (with quota limits)
|
||||
- **Features**:
|
||||
- **Quota Monitoring**: Real-time quota tracking and warnings
|
||||
- **Batch Delete**: Efficient bulk deletion
|
||||
- **Storage Status**: Detailed usage/available reporting
|
||||
Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `rootDirectory` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns:
|
||||
|
||||
- `gsutil rsync -r ./data gs://my-bucket/brainy-data`
|
||||
- `aws s3 sync ./data s3://my-bucket/brainy-data`
|
||||
- `rclone sync ./data remote:brainy-data`
|
||||
- Periodic `tar` snapshots to any object store
|
||||
|
||||
Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory.
|
||||
|
||||
## Metadata Indexing System
|
||||
|
||||
|
|
@ -170,12 +124,12 @@ Tracks all unique values for each field:
|
|||
```json
|
||||
// __metadata_field_index__field_category.json
|
||||
{
|
||||
"values": {
|
||||
"technology": 45,
|
||||
"science": 32,
|
||||
"business": 28
|
||||
},
|
||||
"lastUpdated": 1699564234567
|
||||
"values": {
|
||||
"technology": 45,
|
||||
"science": 32,
|
||||
"business": 28
|
||||
},
|
||||
"lastUpdated": 1699564234567
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -185,11 +139,11 @@ Maps field+value combinations to entity IDs:
|
|||
```json
|
||||
// __metadata_index__category_technology_chunk0.json
|
||||
{
|
||||
"field": "category",
|
||||
"value": "technology",
|
||||
"ids": ["uuid1", "uuid2", "uuid3", ...],
|
||||
"chunk": 0,
|
||||
"total": 45
|
||||
"field": "category",
|
||||
"value": "technology",
|
||||
"ids": ["uuid1", "uuid2", "uuid3", ...],
|
||||
"chunk": 0,
|
||||
"total": 45
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -207,14 +161,14 @@ High-performance deduplication system for streaming data:
|
|||
```json
|
||||
// __entity_registry__.json
|
||||
{
|
||||
"mappings": {
|
||||
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
|
||||
},
|
||||
"stats": {
|
||||
"totalMappings": 10000,
|
||||
"lastSync": 1699564234567
|
||||
}
|
||||
"mappings": {
|
||||
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
|
||||
},
|
||||
"stats": {
|
||||
"totalMappings": 10000,
|
||||
"lastSync": 1699564234567
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -224,216 +178,46 @@ High-performance deduplication system for streaming data:
|
|||
- **Cache**: LRU with configurable TTL
|
||||
- **Sync**: Periodic or on-demand
|
||||
|
||||
## Durability
|
||||
|
||||
Ensures durability and enables recovery:
|
||||
|
||||
```json
|
||||
{
|
||||
"timestamp": 1699564234567,
|
||||
"operation": "add",
|
||||
"data": {
|
||||
"id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"content": "...",
|
||||
"metadata": {}
|
||||
},
|
||||
"checksum": "sha256:..."
|
||||
}
|
||||
```
|
||||
|
||||
### Recovery Process
|
||||
2. Replay operations from last checkpoint
|
||||
3. Verify checksums for integrity
|
||||
Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `rootDirectory` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)).
|
||||
|
||||
## Storage Optimization
|
||||
|
||||
### 1. Lifecycle Policies (Cloud Storage)
|
||||
|
||||
**Automatic cost optimization through tier transitions:**
|
||||
|
||||
```typescript
|
||||
// S3: Set lifecycle policy for automatic archival
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-data',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
|
||||
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
|
||||
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// GCS: Set lifecycle policy
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
condition: { age: 30 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||||
}, {
|
||||
condition: { age: 90 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||||
}, {
|
||||
condition: { age: 365 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||||
}]
|
||||
})
|
||||
|
||||
// Azure: Set lifecycle policy
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
name: 'archiveOldData',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: { blobTypes: ['blockBlob'] },
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 30 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 90 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
**Cost Impact (500TB dataset):**
|
||||
| Storage | Before | After | Savings |
|
||||
|---------|--------|-------|---------|
|
||||
| **AWS S3** | $138,000/yr | $5,940/yr | **96%** |
|
||||
| **GCS** | $138,000/yr | $8,300/yr | **94%** |
|
||||
| **Azure** | $107,520/yr | $5,016/yr | **95%** |
|
||||
|
||||
### 2. Intelligent-Tiering (S3)
|
||||
|
||||
**Automatic optimization without retrieval fees:**
|
||||
|
||||
```typescript
|
||||
// Enable S3 Intelligent-Tiering
|
||||
await storage.enableIntelligentTiering('entities/', 'auto-optimize')
|
||||
|
||||
// Benefits:
|
||||
// - Automatic tier transitions based on access patterns
|
||||
// - No retrieval fees (unlike Glacier)
|
||||
// - Up to 95% cost savings
|
||||
// - No performance impact on frequently accessed data
|
||||
```
|
||||
|
||||
### 3. Autoclass (GCS)
|
||||
|
||||
**Google Cloud's intelligent automatic optimization:**
|
||||
|
||||
```typescript
|
||||
// Enable GCS Autoclass
|
||||
await storage.enableAutoclass({
|
||||
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
|
||||
})
|
||||
|
||||
// Benefits:
|
||||
// - Automatic optimization based on access patterns
|
||||
// - No data retrieval delays
|
||||
// - Transparent tier transitions
|
||||
// - Up to 94% cost savings
|
||||
```
|
||||
|
||||
### 4. Compression (FileSystem)
|
||||
|
||||
```typescript
|
||||
// Enable gzip compression for local storage
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // 60-80% space savings
|
||||
}
|
||||
})
|
||||
|
||||
// Performance impact:
|
||||
// - Write: +10-20ms per file (gzip compression)
|
||||
// - Read: +5-10ms per file (gzip decompression)
|
||||
// - Space savings: 60-80% for typical JSON data
|
||||
// - CPU overhead: Minimal (~5% CPU)
|
||||
```
|
||||
|
||||
### 5. Batch Operations
|
||||
### 1. Batch Operations
|
||||
|
||||
```typescript
|
||||
// Efficient batch delete
|
||||
await storage.batchDelete([
|
||||
'entities/nouns/vectors/00/00123456-....json',
|
||||
'entities/nouns/metadata/00/00123456-....json',
|
||||
// ... up to 1000 objects
|
||||
'entities/nouns/vectors/00/00123456-....json',
|
||||
'entities/nouns/metadata/00/00123456-....json'
|
||||
// ...
|
||||
])
|
||||
|
||||
// Benefits:
|
||||
// - S3: 1000 objects per request (vs 1 per request)
|
||||
// - GCS: 100 objects per request
|
||||
// - Azure: 256 objects per batch
|
||||
// - Automatic retry logic with exponential backoff
|
||||
// - Throttling protection
|
||||
|
||||
// Batch writes for performance
|
||||
await brain.addBatch([
|
||||
{ content: "item1", metadata: {} },
|
||||
{ content: "item2", metadata: {} },
|
||||
{ content: "item3", metadata: {} }
|
||||
{ content: "item1", metadata: {} },
|
||||
{ content: "item2", metadata: {} },
|
||||
{ content: "item3", metadata: {} }
|
||||
])
|
||||
// Single transaction, optimized I/O
|
||||
```
|
||||
|
||||
### 6. Quota Monitoring (OPFS)
|
||||
### 2. Caching Strategy
|
||||
|
||||
```typescript
|
||||
// Get quota status for browser storage
|
||||
const status = await storage.getStorageStatus()
|
||||
|
||||
console.log(status)
|
||||
// {
|
||||
// type: 'opfs',
|
||||
// available: true,
|
||||
// details: {
|
||||
// usage: 45829120, // 43.7 MB used
|
||||
// quota: 536870912, // 512 MB available
|
||||
// usagePercent: 8.5,
|
||||
// quotaExceeded: false
|
||||
// }
|
||||
// }
|
||||
|
||||
// Proactive quota management:
|
||||
// - Monitor usage before writes
|
||||
// - Warn users when approaching quota
|
||||
// - Automatically clean up old data
|
||||
```
|
||||
|
||||
### 7. Tier Management (Azure)
|
||||
|
||||
```typescript
|
||||
// Change blob tier for cost optimization
|
||||
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
|
||||
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
|
||||
|
||||
// Batch tier changes (efficient)
|
||||
await storage.batchChangeTier([blob1, blob2, blob3], 'Cool')
|
||||
|
||||
// Rehydrate from Archive when needed
|
||||
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
|
||||
```
|
||||
|
||||
### 8. Caching Strategy
|
||||
|
||||
```typescript
|
||||
// Configure caching per storage type
|
||||
// Configure caching
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
ttl: 300000, // 5 minutes
|
||||
strategy: 'lru' // Least recently used
|
||||
}
|
||||
}
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
rootDirectory: './data',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
ttl: 300000, // 5 minutes
|
||||
strategy: 'lru' // Least recently used
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -443,8 +227,8 @@ const brain = new Brainy({
|
|||
```typescript
|
||||
// Automatic locking for write operations
|
||||
await brain.storage.withLock('resource-id', async () => {
|
||||
// Exclusive access to resource
|
||||
await brain.storage.saveNoun(id, data)
|
||||
// Exclusive access to resource
|
||||
await brain.storage.saveNoun(id, data)
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -459,9 +243,9 @@ await brain.storage.withLock('resource-id', async () => {
|
|||
```typescript
|
||||
// Export entire database
|
||||
const backup = await brain.export({
|
||||
format: 'json',
|
||||
includeVectors: true,
|
||||
includeIndexes: false
|
||||
format: 'json',
|
||||
includeVectors: true,
|
||||
includeIndexes: false
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -469,16 +253,16 @@ const backup = await brain.export({
|
|||
```typescript
|
||||
// Import from backup
|
||||
await brain.import(backup, {
|
||||
mode: 'merge', // or 'replace'
|
||||
validateSchema: true
|
||||
mode: 'merge', // or 'replace'
|
||||
validateSchema: true
|
||||
})
|
||||
```
|
||||
|
||||
### Storage Migration
|
||||
```typescript
|
||||
// Migrate between storage types
|
||||
const oldBrain = new Brainy({ storage: { type: 'filesystem' } })
|
||||
const newBrain = new Brainy({ storage: { type: 's3' } })
|
||||
const oldBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './old' } })
|
||||
const newBrain = new Brainy({ storage: { type: 'filesystem', rootDirectory: './new' } })
|
||||
|
||||
await oldBrain.init()
|
||||
await newBrain.init()
|
||||
|
|
@ -490,23 +274,11 @@ await newBrain.import(data)
|
|||
|
||||
## Performance Tuning
|
||||
|
||||
### Storage-Specific Optimizations
|
||||
|
||||
#### FileSystem
|
||||
- **Directory sharding**: Split files across subdirectories
|
||||
### FileSystem Optimizations
|
||||
- **Directory sharding**: 256 shards spread files across subdirectories
|
||||
- **Async I/O**: Non-blocking file operations
|
||||
- **Buffer pooling**: Reuse buffers for efficiency
|
||||
|
||||
#### S3
|
||||
- **Multipart uploads**: For large objects
|
||||
- **Request batching**: Combine small operations
|
||||
- **CDN integration**: Edge caching for reads
|
||||
|
||||
#### OPFS
|
||||
- **Quota management**: Monitor and request increases
|
||||
- **Worker offloading**: Heavy operations in workers
|
||||
- **Transaction batching**: Group operations
|
||||
|
||||
### Monitoring
|
||||
|
||||
```typescript
|
||||
|
|
@ -514,58 +286,34 @@ await newBrain.import(data)
|
|||
const stats = await brain.storage.getStatistics()
|
||||
console.log(stats)
|
||||
// {
|
||||
// totalSize: 1048576,
|
||||
// entityCount: 1000,
|
||||
// indexSize: 204800,
|
||||
// walSize: 10240,
|
||||
// cacheHitRate: 0.85
|
||||
// totalSize: 1048576,
|
||||
// entityCount: 1000,
|
||||
// indexSize: 204800,
|
||||
// walSize: 10240,
|
||||
// cacheHitRate: 0.85
|
||||
// }
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Choose the Right Adapter
|
||||
1. **Development**: FileSystem with compression (local persistence, small storage footprint)
|
||||
2. **Production Server**: FileSystem with compression or cloud storage with lifecycle policies
|
||||
3. **Browser Apps**: OPFS with quota monitoring
|
||||
4. **Distributed**: S3/GCS/Azure with Intelligent-Tiering/Autoclass
|
||||
1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence
|
||||
2. **Single-node production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone`
|
||||
3. **Multi-node production**: Run Brainy behind a service layer; replicate the on-disk artifact via your operator tooling
|
||||
|
||||
### Optimize for Your Use Case
|
||||
1. **Read-heavy**: Enable aggressive caching + cloud CDN
|
||||
2. **Write-heavy**: Batch operations + async writes
|
||||
1. **Read-heavy**: Enable caching and let the OS page cache do its job
|
||||
2. **Write-heavy**: Batch operations and tune the cache `maxSize`
|
||||
3. **Real-time**: FileSystem with periodic snapshots
|
||||
4. **Archival**: Cloud storage with lifecycle policies (96% cost savings!)
|
||||
5. **Large-scale**: Metadata/vector separation + UUID sharding + lifecycle policies
|
||||
|
||||
### Cost Optimization
|
||||
1. **Enable lifecycle policies** for cloud storage (automated cost reduction)
|
||||
2. **Use Intelligent-Tiering (S3)** or Autoclass (GCS) for automatic optimization
|
||||
3. **Enable compression** for FileSystem storage (60-80% space savings)
|
||||
4. **Monitor quota** for OPFS (prevent quota exceeded errors)
|
||||
5. **Use batch operations** for bulk deletions (efficient API usage)
|
||||
6. **Consider tier management** for Azure (Hot/Cool/Archive tiers)
|
||||
|
||||
**Example Cost Savings (500TB dataset):**
|
||||
- Without lifecycle policies: **$138,000/year**
|
||||
- With lifecycle policies: **$5,940/year**
|
||||
- **Savings: $132,060/year (96%)**
|
||||
4. **Archival**: Snapshot `rootDirectory` to cold object storage on a schedule
|
||||
5. **Large-scale**: Rely on metadata/vector separation + UUID sharding
|
||||
|
||||
### Monitor and Maintain
|
||||
1. Regular statistics collection
|
||||
2. Monitor lifecycle policy effectiveness
|
||||
2. Watch disk usage and shard balance
|
||||
3. Index optimization
|
||||
4. Cache tuning based on hit rates
|
||||
5. Track storage costs and tier distribution
|
||||
6. Review quota usage (OPFS) and storage growth patterns
|
||||
|
||||
### Production Deployment Checklist
|
||||
- ✅ Enable lifecycle policies on cloud storage
|
||||
- ✅ Configure batch delete for cleanup operations
|
||||
- ✅ Enable compression for FileSystem storage
|
||||
- ✅ Set up quota monitoring for OPFS
|
||||
- ✅ Configure appropriate tier transitions
|
||||
- ✅ Enable Intelligent-Tiering (S3) or Autoclass (GCS)
|
||||
- ✅ Monitor storage costs and optimize regularly
|
||||
5. Verify backup runs (test restore quarterly)
|
||||
|
||||
## API Reference
|
||||
|
||||
|
|
@ -573,6 +321,5 @@ See the [Storage API](../api/storage.md) for complete method documentation.
|
|||
|
||||
---
|
||||
|
||||
**Version**: 4.0.0
|
||||
**Last Updated**: 2025-10-17
|
||||
**Key Features**: Metadata/vector separation, UUID sharding, lifecycle management, tier optimization
|
||||
**Last Updated**: 2026
|
||||
**Key Features**: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue