fix: exclude __words__ keyword index from corruption detection and getStats()
The __words__ keyword index stores 50-5000 entries per entity (one per word), which inflated avg entries/entity well above the corruption threshold of 100. This caused: 1. validateConsistency() to falsely detect corruption on every startup, triggering unnecessary clearAllIndexData() + rebuild() cycles 2. getStats() to log false "Metadata index may be corrupted" warnings and report inflated totalEntries/totalIds stats Both methods now skip __words__ when counting, so stats and health checks reflect metadata fields only (noun, type, createdAt, etc.). Keyword search is unaffected since the __words__ field index itself is not modified.
This commit is contained in:
parent
32dbdcec61
commit
364360d447
128 changed files with 5637 additions and 5682 deletions
|
|
@ -1,4 +1,4 @@
|
|||
# Brainy Data Storage Architecture (v5.11.0)
|
||||
# Brainy Data Storage Architecture
|
||||
|
||||
**Complete file structure reference for all storage backends**
|
||||
|
||||
|
|
@ -13,7 +13,7 @@ This document explains how Brainy stores, indexes, and scales data across all st
|
|||
3. [The 4 Indexes](#3-the-4-indexes)
|
||||
4. [Sharding Strategy](#4-sharding-strategy)
|
||||
5. [COW (Copy-on-Write) Architecture](#5-cow-copy-on-write-architecture)
|
||||
6. [ID-First Storage Architecture (v6.0.0+)](#6-id-first-storage-architecture-v600)
|
||||
6. [ID-First Storage Architecture](#6-id-first-storage-architecture-v600)
|
||||
7. [VFS (Virtual File System)](#7-vfs-virtual-file-system)
|
||||
8. [Storage Backend Mapping](#8-storage-backend-mapping)
|
||||
9. [Performance Characteristics](#9-performance-characteristics)
|
||||
|
|
@ -438,7 +438,7 @@ Unlike entities and relationships, system metadata consists of **index files** t
|
|||
|
||||
Understanding how Brainy constructs storage paths is critical for debugging and optimization.
|
||||
|
||||
### Path Construction Steps (v6.0.0+)
|
||||
### Path Construction Steps
|
||||
|
||||
**For an entity (noun)**:
|
||||
```typescript
|
||||
|
|
@ -498,7 +498,7 @@ const fieldIndexPath = `_system/metadata_indexes/__metadata_field_index__${field
|
|||
const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
|
||||
```
|
||||
|
||||
### Path Patterns Summary (v6.0.0+)
|
||||
### Path Patterns Summary
|
||||
|
||||
| Data Type | Path Pattern | Sharded? | Branched? |
|
||||
|-----------|--------------|----------|-----------|
|
||||
|
|
@ -516,7 +516,7 @@ const hnswNodePath = `_system/hnsw/nodes/${shard}/${entityId}.json`
|
|||
| **HNSW node** | `_system/hnsw/nodes/{shard}/{uuid}.json` | ✅ Yes (UUID) | ❌ No |
|
||||
| **Field index** | `_system/metadata_indexes/__metadata_field_index__{field}.json` | ❌ No | ❌ No |
|
||||
|
||||
### Key Principles (v6.0.0+)
|
||||
### Key Principles
|
||||
|
||||
1. **Shard Extraction**: Always use first 2 hex characters of UUID/SHA-256
|
||||
2. **ID-First**: Shard + ID come BEFORE type (type is in metadata)
|
||||
|
|
@ -549,7 +549,7 @@ Brainy uses four complementary index systems for different query patterns.
|
|||
- Memory (standard): ~200MB per 100K entities
|
||||
- Memory (lazy): ~15-33MB per 100K entities (5-10x less!)
|
||||
|
||||
**Automatic Lazy Mode** (v3.36.0+): Enables automatically when vectors don't fit in UnifiedCache
|
||||
**Automatic Lazy Mode**: Enables automatically when vectors don't fit in UnifiedCache
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -559,7 +559,7 @@ Brainy uses four complementary index systems for different query patterns.
|
|||
**Location**: MetadataIndexManager index on `noun` field
|
||||
**Data Structure**: RoaringBitmap32 per type value
|
||||
|
||||
**How It Works** (v6.0.0+):
|
||||
**How It Works**:
|
||||
```typescript
|
||||
// Find all Person entities
|
||||
const people = await brain.getNouns({ type: 'person' })
|
||||
|
|
@ -721,7 +721,7 @@ COW is Brainy's **git-like versioning system** that enables:
|
|||
- ✅ **Deduplication** (identical data stored only once)
|
||||
- ✅ **Version history** (full audit trail of all changes)
|
||||
|
||||
**Status**: ALWAYS ENABLED (v5.11.0+) - cannot be disabled
|
||||
**Status**: ALWAYS ENABLED - cannot be disabled
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -782,19 +782,19 @@ _cow/blobs/ab/abc123...sha256.bin (used by both entities)
|
|||
|
||||
---
|
||||
|
||||
## 6. ID-First Storage Architecture (v6.0.0+)
|
||||
## 6. ID-First Storage Architecture
|
||||
|
||||
### 6.1 What is ID-First?
|
||||
|
||||
**ID-first storage** organizes entities by **ID shard only** - no type directories! This eliminates 42-type sequential searches that caused 20-21 second delays on cloud storage.
|
||||
|
||||
**Old type-first structure** (v5.4.0-v5.12.0):
|
||||
**Old type-first structure** (v5.12.0):
|
||||
```
|
||||
branches/main/entities/nouns/{TYPE}/metadata/00/001234...uuid.json
|
||||
# Problem: Requires knowing type OR searching 42 type directories!
|
||||
```
|
||||
|
||||
**NEW ID-first structure** (v6.0.0+):
|
||||
**NEW ID-first structure**:
|
||||
```
|
||||
branches/main/entities/nouns/00/001234...uuid/metadata.json
|
||||
# Direct O(1) lookup - no type needed!
|
||||
|
|
@ -809,7 +809,7 @@ branches/main/entities/nouns/00/001234...uuid/metadata.json
|
|||
// v5.x: Had to search 42 types if type unknown
|
||||
// Result: 21 seconds on GCS (42 types × 500ms)
|
||||
|
||||
// v6.0.0: Direct path from ID
|
||||
// Direct path from ID
|
||||
const id = '001234...'
|
||||
const shard = id.substring(0, 2) // '00'
|
||||
const path = `branches/main/entities/nouns/${shard}/${id}/metadata.json`
|
||||
|
|
@ -1165,7 +1165,7 @@ opfs://root/brainy/
|
|||
|
||||
### 10.2 clear() Operation
|
||||
|
||||
**What clear() deletes** (v5.11.0+):
|
||||
**What clear() deletes**:
|
||||
|
||||
✅ Deletes:
|
||||
- `branches/` → ALL entity data (all types, all shards, all branches, all forks)
|
||||
|
|
@ -1184,7 +1184,7 @@ opfs://root/brainy/
|
|||
|
||||
**Example**:
|
||||
```typescript
|
||||
await brain.storage.clear() // ✅ Deletes ALL data correctly (v5.11.0+)
|
||||
await brain.storage.clear() // ✅ Deletes ALL data correctly
|
||||
await brain.add({ data: 'Alice', type: 'person' }) // ✅ COW reinitializes automatically
|
||||
```
|
||||
|
||||
|
|
@ -1353,7 +1353,7 @@ const historicalData = await yesterday.getNouns({ type: 'Character' })
|
|||
await brain.storage.clear()
|
||||
```
|
||||
|
||||
**What happens in storage** (v5.11.0+):
|
||||
**What happens in storage**:
|
||||
```
|
||||
1. Delete all entity data:
|
||||
→ Remove: branches/ (entire directory)
|
||||
|
|
@ -1452,7 +1452,7 @@ await brain.addBatch([
|
|||
|
||||
## 11. Summary
|
||||
|
||||
**Complete Storage Structure (v6.0.0)**:
|
||||
**Complete Storage Structure**:
|
||||
- **3 storage layers**: branches/ (data), _cow/ (versions), _system/ (indexes)
|
||||
- **2 files per entity**: metadata.json + vector.json (optimized I/O)
|
||||
- **4 indexes**: HNSW (semantic), Type Index (metadata-based), Graph (relationships), Metadata (fields)
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ Brainy has **3 main indexes** at the top level, each with multiple sub-indexes m
|
|||
- **FieldTypeInference** - DuckDB-inspired value-based field type detection
|
||||
- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count)
|
||||
- **Sorted Indexes** - Support orderBy queries (automatically maintained)
|
||||
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes (v7.7.0)
|
||||
- **Word Index (`__words__`)** - Text search via FNV-1a word hashes
|
||||
|
||||
**GraphAdjacencyIndex contains:**
|
||||
- **lsmTreeSource** - Source → Targets (outgoing edges)
|
||||
|
|
@ -39,7 +39,7 @@ All indexes share a **UnifiedCache** for coordinated memory management, ensuring
|
|||
|
||||
**Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing.
|
||||
|
||||
### Internal Architecture (v3.42.0)
|
||||
### Internal Architecture
|
||||
|
||||
```typescript
|
||||
class MetadataIndexManager {
|
||||
|
|
@ -64,7 +64,7 @@ class MetadataIndexManager {
|
|||
|
||||
### Key Data Structures
|
||||
|
||||
#### Chunked Sparse Index (NEW in v3.42.0)
|
||||
#### Chunked Sparse Index
|
||||
```typescript
|
||||
// SparseIndex: Directory of chunks for a field
|
||||
// Example: field="status"
|
||||
|
|
@ -87,7 +87,7 @@ interface ChunkDescriptor {
|
|||
class ChunkData {
|
||||
chunkId: number
|
||||
field: string
|
||||
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (v3.43.0: roaring bitmaps!)
|
||||
entries: Map<value, RoaringBitmap32> // ~50 values per chunk (roaring bitmaps!)
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -96,7 +96,7 @@ class ChunkData {
|
|||
- O(log n) range queries with zone maps
|
||||
- 630x file reduction (560k flat files → 89 chunk files)
|
||||
|
||||
#### Roaring Bitmap Optimization (NEW in v3.43.0)
|
||||
#### Roaring Bitmap Optimization
|
||||
|
||||
**Problem Solved**: JavaScript `Set<string>` for storing entity IDs was inefficient:
|
||||
- Memory overhead: ~40 bytes per UUID string (36 chars + overhead)
|
||||
|
|
@ -150,7 +150,7 @@ class ChunkData {
|
|||
|
||||
**Multi-Field Intersection (THE BIG WIN!)**:
|
||||
```typescript
|
||||
// Before (v3.42.0): JavaScript array filtering
|
||||
// Before: JavaScript array filtering
|
||||
async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string[]> {
|
||||
// 1. Fetch UUID arrays for each field
|
||||
const statusIds = await this.getIds('status', 'active') // ["uuid1", "uuid2", ...]
|
||||
|
|
@ -160,7 +160,7 @@ async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise<string
|
|||
return statusIds.filter(id => roleIds.includes(id)) // O(n*m) array filtering
|
||||
}
|
||||
|
||||
// After (v3.43.0): Roaring bitmap intersection
|
||||
// After: Roaring bitmap intersection
|
||||
async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
|
||||
// 1. Fetch roaring bitmaps (integers, not UUIDs)
|
||||
const bitmaps: RoaringBitmap32[] = []
|
||||
|
|
@ -229,7 +229,7 @@ interface ZoneMap {
|
|||
|
||||
**Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field
|
||||
|
||||
#### Word Index (`__words__`) - v7.7.0
|
||||
#### Word Index (`__words__`) -
|
||||
```typescript
|
||||
// Special field for text/keyword search
|
||||
// Entity text content is tokenized and indexed as word hashes
|
||||
|
|
@ -253,14 +253,14 @@ interface ZoneMap {
|
|||
- **Lowercase normalization**: Case-insensitive matching
|
||||
- **Automatic integration**: Words extracted via `extractIndexableFields()`
|
||||
|
||||
**Hybrid Search** (v7.7.0): Text results combined with vector results using Reciprocal Rank Fusion (RRF):
|
||||
**Hybrid Search**: Text results combined with vector results using Reciprocal Rank Fusion (RRF):
|
||||
```typescript
|
||||
// RRF formula: score(d) = sum(1 / (k + rank(d)))
|
||||
// where k = 60 (standard constant)
|
||||
// alpha = weight for semantic (0 = text only, 1 = semantic only)
|
||||
```
|
||||
|
||||
### Query Algorithm (v3.42.0)
|
||||
### Query Algorithm
|
||||
|
||||
**Exact Match Query**:
|
||||
```typescript
|
||||
|
|
@ -317,7 +317,7 @@ async getIdsForRange(field: string, min: any, max: any): Promise<string[]> {
|
|||
- Adaptive chunking: ~50 values per chunk optimizes I/O
|
||||
- Immediate flushing: No need for dirty tracking or batch writes
|
||||
|
||||
### Temporal Bucketing (v3.41.0)
|
||||
### Temporal Bucketing
|
||||
|
||||
**Problem Solved**: High-cardinality timestamp fields created massive file pollution.
|
||||
- Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!)
|
||||
|
|
@ -396,7 +396,7 @@ const DEFAULT_EXCLUDE_FIELDS = [
|
|||
]
|
||||
```
|
||||
|
||||
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of v3.41.0 - they are indexed with automatic bucketing.
|
||||
**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing.
|
||||
|
||||
## 2. HNSWIndex - Vector Similarity Search
|
||||
|
||||
|
|
@ -735,7 +735,7 @@ async stats(): Promise<Statistics> {
|
|||
}
|
||||
```
|
||||
|
||||
### 5. Index Rebuilding (v5.7.7: Lazy Loading Support)
|
||||
### 5. Index Rebuilding (Lazy Loading Support)
|
||||
|
||||
**Two modes of index loading:**
|
||||
|
||||
|
|
@ -762,7 +762,7 @@ async init(): Promise<void> {
|
|||
}
|
||||
```
|
||||
|
||||
#### Mode 2: Lazy Loading on First Query (v5.7.7+)
|
||||
#### Mode 2: Lazy Loading on First Query
|
||||
|
||||
```typescript
|
||||
// When disableAutoRebuild: true
|
||||
|
|
@ -918,7 +918,7 @@ All indexes scale gracefully:
|
|||
- [Performance Guide](../PERFORMANCE.md) - Performance tuning
|
||||
- [Overview](./overview.md) - High-level architecture
|
||||
|
||||
## Summary: Index Hierarchy (v5.7.7)
|
||||
## Summary: Index Hierarchy
|
||||
|
||||
### Level 1: Main Indexes (3)
|
||||
All have rebuild() methods and are covered by lazy loading:
|
||||
|
|
@ -933,7 +933,7 @@ Automatically managed by parent rebuild():
|
|||
- **4 LSM-trees** (lsmTreeSource, lsmTreeTarget, lsmTreeVerbsBySource, lsmTreeVerbsByTarget)
|
||||
- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex)
|
||||
|
||||
### Lazy Loading (v5.7.7)
|
||||
### Lazy Loading
|
||||
- **Mode 1**: Auto-rebuild on init() (default)
|
||||
- **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`)
|
||||
- **Concurrency-safe**: Mutex prevents duplicate rebuilds
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, Grap
|
|||
| **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 |
|
||||
| **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 |
|
||||
|
||||
#### MetadataIndex Persistence Details (v4.2.1+)
|
||||
#### MetadataIndex Persistence Details
|
||||
|
||||
The MetadataIndex now persists two components:
|
||||
|
||||
|
|
@ -129,7 +129,7 @@ async init(): Promise<void> {
|
|||
- 100-2000ms: One-time rebuild to create indices
|
||||
- Total: ~1-3 seconds (one time only)
|
||||
|
||||
#### Mode 2: Lazy Loading on First Query (v5.7.7+)
|
||||
#### Mode 2: Lazy Loading on First Query
|
||||
|
||||
When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query:
|
||||
|
||||
|
|
@ -350,7 +350,7 @@ public async rebuild(options?: {
|
|||
|
||||
**Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities)
|
||||
|
||||
**Correct Pattern** (v3.45.0):
|
||||
**Correct Pattern**:
|
||||
```typescript
|
||||
// Load ALL nouns ONCE (not 31 times!)
|
||||
while (hasMore) {
|
||||
|
|
@ -392,7 +392,7 @@ while (hasMore) {
|
|||
```typescript
|
||||
// src/utils/metadataIndex.ts (lines 202-216)
|
||||
async init(): Promise<void> {
|
||||
// STEP 1: Load field registry to discover persisted indices (v4.2.1)
|
||||
// STEP 1: Load field registry to discover persisted indices
|
||||
// This is THE KEY FIX - O(1) discovery of existing indices
|
||||
await this.loadFieldRegistry()
|
||||
|
||||
|
|
|
|||
|
|
@ -1,40 +1,40 @@
|
|||
# Storage Architecture (v4.0.0)
|
||||
# Storage Architecture
|
||||
|
||||
> **Updated for v4.0.0**: Metadata/vector separation, UUID-based sharding, lifecycle management
|
||||
> **Updated**: Metadata/vector separation, UUID-based sharding, lifecycle management
|
||||
|
||||
## Storage Structure
|
||||
|
||||
### v4.0.0 Architecture: Metadata/Vector Separation
|
||||
### Architecture: Metadata/Vector Separation
|
||||
|
||||
In v4.0.0, entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
|
||||
entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale:
|
||||
|
||||
```
|
||||
brainy-data/
|
||||
├── _system/ # System metadata (not sharded)
|
||||
│ ├── statistics.json # Performance metrics
|
||||
│ ├── __metadata_field_index__*.json # Field indexes
|
||||
│ └── __metadata_sorted_index__*.json # Sorted indexes
|
||||
├── _system/ # System metadata (not sharded)
|
||||
│ ├── statistics.json # Performance metrics
|
||||
│ ├── __metadata_field_index__*.json # Field indexes
|
||||
│ └── __metadata_sorted_index__*.json # Sorted indexes
|
||||
│
|
||||
├── entities/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # HNSW graph data (sharded by UUID)
|
||||
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
|
||||
│ │ │ │ ├── 00123456-....json # Vector + HNSW connections
|
||||
│ │ │ │ └── 00abcdef-....json
|
||||
│ │ │ ├── 01/ ... ff/ # 256 shards total
|
||||
│ │ │
|
||||
│ │ └── metadata/ # Business data (sharded by UUID)
|
||||
│ │ ├── 00/
|
||||
│ │ │ ├── 00123456-....json # Entity metadata only
|
||||
│ │ │ └── 00abcdef-....json
|
||||
│ │ ├── 01/ ... ff/
|
||||
│ │
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (sharded)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ │
|
||||
│ └── metadata/ # Relationship data (sharded)
|
||||
│ ├── 00/ ... ff/
|
||||
│ ├── nouns/
|
||||
│ │ ├── vectors/ # HNSW graph data (sharded by UUID)
|
||||
│ │ │ ├── 00/ # Shard 00 (first 2 hex digits)
|
||||
│ │ │ │ ├── 00123456-....json # Vector + HNSW connections
|
||||
│ │ │ │ └── 00abcdef-....json
|
||||
│ │ │ ├── 01/ ... ff/ # 256 shards total
|
||||
│ │ │
|
||||
│ │ └── metadata/ # Business data (sharded by UUID)
|
||||
│ │ ├── 00/
|
||||
│ │ │ ├── 00123456-....json # Entity metadata only
|
||||
│ │ │ └── 00abcdef-....json
|
||||
│ │ ├── 01/ ... ff/
|
||||
│ │
|
||||
│ └── verbs/
|
||||
│ ├── vectors/ # Relationship vectors (sharded)
|
||||
│ │ ├── 00/ ... ff/
|
||||
│ │
|
||||
│ └── metadata/ # Relationship data (sharded)
|
||||
│ ├── 00/ ... ff/
|
||||
```
|
||||
|
||||
### Why Split Metadata and Vectors?
|
||||
|
|
@ -50,9 +50,9 @@ brainy-data/
|
|||
**How it works:**
|
||||
```typescript
|
||||
const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6"
|
||||
const shard = uuid.substring(0, 2) // "3f"
|
||||
const shard = uuid.substring(0, 2) // "3f"
|
||||
|
||||
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
|
||||
// Vector path: entities/nouns/vectors/3f/3fa85f64-....json
|
||||
// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json
|
||||
```
|
||||
|
||||
|
|
@ -64,103 +64,103 @@ const shard = uuid.substring(0, 2) // "3f"
|
|||
|
||||
## Storage Adapters
|
||||
|
||||
Brainy provides multiple storage adapters with identical APIs and v4.0.0 production features:
|
||||
Brainy provides multiple storage adapters with identical APIs and production features:
|
||||
|
||||
### FileSystem Storage (Node.js)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // v4.0.0: Gzip compression (60-80% space savings)
|
||||
}
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // Gzip compression (60-80% space savings)
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Server applications, CLI tools
|
||||
- **Performance**: Direct file I/O with optional compression
|
||||
- **Persistence**: Permanent on disk
|
||||
- **v4.0.0 Features**:
|
||||
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
|
||||
- **Batch Delete**: Efficient bulk deletion with retries
|
||||
- **UUID Sharding**: Automatic 256-shard distribution
|
||||
- **Features**:
|
||||
- **Gzip Compression**: 60-80% storage savings with minimal CPU overhead
|
||||
- **Batch Delete**: Efficient bulk deletion with retries
|
||||
- **UUID Sharding**: Automatic 256-shard distribution
|
||||
|
||||
### S3 Compatible Storage (AWS, MinIO, R2)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucket: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
credentials: {
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
storage: {
|
||||
type: 's3',
|
||||
bucket: 'my-brainy-data',
|
||||
region: 'us-east-1',
|
||||
credentials: {
|
||||
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
|
||||
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Distributed applications, cloud deployments
|
||||
- **Performance**: Network dependent, with intelligent caching
|
||||
- **Persistence**: Cloud storage durability (99.999999999%)
|
||||
- **v4.0.0 Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
|
||||
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
|
||||
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
|
||||
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
|
||||
- **Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → IA → Glacier → Deep Archive)
|
||||
- **Intelligent-Tiering**: Automatic optimization based on access patterns (up to 95% savings)
|
||||
- **Batch Delete**: Efficient bulk deletion (1000 objects per request)
|
||||
- **Cost Impact**: $138k/year → $5.9k/year at 500TB (96% savings!)
|
||||
|
||||
### Google Cloud Storage (GCS)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
bucketName: 'my-brainy-data',
|
||||
keyFilename: './service-account.json' // Or use ADC
|
||||
}
|
||||
storage: {
|
||||
type: 'gcs',
|
||||
bucketName: 'my-brainy-data',
|
||||
keyFilename: './service-account.json' // Or use ADC
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Google Cloud deployments
|
||||
- **Performance**: Global CDN with edge caching
|
||||
- **Persistence**: 99.999999999% durability
|
||||
- **v4.0.0 Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
|
||||
- **Autoclass**: Intelligent automatic tier optimization
|
||||
- **Batch Delete**: Efficient bulk operations
|
||||
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
|
||||
- **Features**:
|
||||
- **Lifecycle Policies**: Automatic tier transitions (Standard → Nearline → Coldline → Archive)
|
||||
- **Autoclass**: Intelligent automatic tier optimization
|
||||
- **Batch Delete**: Efficient bulk operations
|
||||
- **Cost Impact**: $138k/year → $8.3k/year at 500TB (94% savings!)
|
||||
|
||||
### Azure Blob Storage
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'azure',
|
||||
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
|
||||
containerName: 'brainy-data'
|
||||
}
|
||||
storage: {
|
||||
type: 'azure',
|
||||
connectionString: process.env.AZURE_STORAGE_CONNECTION_STRING,
|
||||
containerName: 'brainy-data'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Azure cloud deployments
|
||||
- **Performance**: Global replication with CDN
|
||||
- **Persistence**: LRS, ZRS, GRS, RA-GRS options
|
||||
- **v4.0.0 Features**:
|
||||
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
|
||||
- **Lifecycle Policies**: Automatic tier transitions and deletions
|
||||
- **Batch Delete**: BlobBatchClient for efficient bulk operations
|
||||
- **Batch Tier Changes**: Move thousands of blobs efficiently
|
||||
- **Archive Rehydration**: Smart rehydration with priority options
|
||||
- **Features**:
|
||||
- **Blob Tier Management**: Hot/Cool/Archive tiers (99% cost savings)
|
||||
- **Lifecycle Policies**: Automatic tier transitions and deletions
|
||||
- **Batch Delete**: BlobBatchClient for efficient bulk operations
|
||||
- **Batch Tier Changes**: Move thousands of blobs efficiently
|
||||
- **Archive Rehydration**: Smart rehydration with priority options
|
||||
|
||||
### Origin Private File System (Browser)
|
||||
```typescript
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'opfs'
|
||||
}
|
||||
storage: {
|
||||
type: 'opfs'
|
||||
}
|
||||
})
|
||||
```
|
||||
- **Use case**: Browser applications, PWAs
|
||||
- **Performance**: Near-native file system speed
|
||||
- **Persistence**: Permanent in browser (with quota limits)
|
||||
- **v4.0.0 Features**:
|
||||
- **Quota Monitoring**: Real-time quota tracking and warnings
|
||||
- **Batch Delete**: Efficient bulk deletion
|
||||
- **Storage Status**: Detailed usage/available reporting
|
||||
- **Features**:
|
||||
- **Quota Monitoring**: Real-time quota tracking and warnings
|
||||
- **Batch Delete**: Efficient bulk deletion
|
||||
- **Storage Status**: Detailed usage/available reporting
|
||||
|
||||
## Metadata Indexing System
|
||||
|
||||
|
|
@ -170,12 +170,12 @@ Tracks all unique values for each field:
|
|||
```json
|
||||
// __metadata_field_index__field_category.json
|
||||
{
|
||||
"values": {
|
||||
"technology": 45,
|
||||
"science": 32,
|
||||
"business": 28
|
||||
},
|
||||
"lastUpdated": 1699564234567
|
||||
"values": {
|
||||
"technology": 45,
|
||||
"science": 32,
|
||||
"business": 28
|
||||
},
|
||||
"lastUpdated": 1699564234567
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -185,11 +185,11 @@ Maps field+value combinations to entity IDs:
|
|||
```json
|
||||
// __metadata_index__category_technology_chunk0.json
|
||||
{
|
||||
"field": "category",
|
||||
"value": "technology",
|
||||
"ids": ["uuid1", "uuid2", "uuid3", ...],
|
||||
"chunk": 0,
|
||||
"total": 45
|
||||
"field": "category",
|
||||
"value": "technology",
|
||||
"ids": ["uuid1", "uuid2", "uuid3", ...],
|
||||
"chunk": 0,
|
||||
"total": 45
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -207,14 +207,14 @@ High-performance deduplication system for streaming data:
|
|||
```json
|
||||
// __entity_registry__.json
|
||||
{
|
||||
"mappings": {
|
||||
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
|
||||
},
|
||||
"stats": {
|
||||
"totalMappings": 10000,
|
||||
"lastSync": 1699564234567
|
||||
}
|
||||
"mappings": {
|
||||
"did:plc:alice123": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"handle:alice.bsky.social": "550e8400-e29b-41d4-a716-446655440000"
|
||||
},
|
||||
"stats": {
|
||||
"totalMappings": 10000,
|
||||
"lastSync": 1699564234567
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -229,14 +229,14 @@ Ensures durability and enables recovery:
|
|||
|
||||
```json
|
||||
{
|
||||
"timestamp": 1699564234567,
|
||||
"operation": "add",
|
||||
"data": {
|
||||
"id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"content": "...",
|
||||
"metadata": {}
|
||||
},
|
||||
"checksum": "sha256:..."
|
||||
"timestamp": 1699564234567,
|
||||
"operation": "add",
|
||||
"data": {
|
||||
"id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"content": "...",
|
||||
"metadata": {}
|
||||
},
|
||||
"checksum": "sha256:..."
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -244,7 +244,7 @@ Ensures durability and enables recovery:
|
|||
2. Replay operations from last checkpoint
|
||||
3. Verify checksums for integrity
|
||||
|
||||
## Storage Optimization (v4.0.0)
|
||||
## Storage Optimization
|
||||
|
||||
### 1. Lifecycle Policies (Cloud Storage)
|
||||
|
||||
|
|
@ -253,48 +253,48 @@ Ensures durability and enables recovery:
|
|||
```typescript
|
||||
// S3: Set lifecycle policy for automatic archival
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
id: 'archive-old-data',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
|
||||
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
|
||||
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
|
||||
]
|
||||
}]
|
||||
rules: [{
|
||||
id: 'archive-old-data',
|
||||
prefix: 'entities/',
|
||||
status: 'Enabled',
|
||||
transitions: [
|
||||
{ days: 30, storageClass: 'STANDARD_IA' }, // Move to IA after 30 days
|
||||
{ days: 90, storageClass: 'GLACIER' }, // Archive after 90 days
|
||||
{ days: 365, storageClass: 'DEEP_ARCHIVE' } // Deep archive after 1 year
|
||||
]
|
||||
}]
|
||||
})
|
||||
|
||||
// GCS: Set lifecycle policy
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
condition: { age: 30 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||||
}, {
|
||||
condition: { age: 90 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||||
}, {
|
||||
condition: { age: 365 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||||
}]
|
||||
rules: [{
|
||||
condition: { age: 30 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'NEARLINE' }
|
||||
}, {
|
||||
condition: { age: 90 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'COLDLINE' }
|
||||
}, {
|
||||
condition: { age: 365 },
|
||||
action: { type: 'SetStorageClass', storageClass: 'ARCHIVE' }
|
||||
}]
|
||||
})
|
||||
|
||||
// Azure: Set lifecycle policy
|
||||
await storage.setLifecyclePolicy({
|
||||
rules: [{
|
||||
name: 'archiveOldData',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: { blobTypes: ['blockBlob'] },
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 30 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 90 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
rules: [{
|
||||
name: 'archiveOldData',
|
||||
enabled: true,
|
||||
type: 'Lifecycle',
|
||||
definition: {
|
||||
filters: { blobTypes: ['blockBlob'] },
|
||||
actions: {
|
||||
baseBlob: {
|
||||
tierToCool: { daysAfterModificationGreaterThan: 30 },
|
||||
tierToArchive: { daysAfterModificationGreaterThan: 90 }
|
||||
}
|
||||
}
|
||||
}
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -327,7 +327,7 @@ await storage.enableIntelligentTiering('entities/', 'auto-optimize')
|
|||
```typescript
|
||||
// Enable GCS Autoclass
|
||||
await storage.enableAutoclass({
|
||||
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
|
||||
terminalStorageClass: 'ARCHIVE' // Optional: Set lowest tier
|
||||
})
|
||||
|
||||
// Benefits:
|
||||
|
|
@ -342,11 +342,11 @@ await storage.enableAutoclass({
|
|||
```typescript
|
||||
// Enable gzip compression for local storage
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // 60-80% space savings
|
||||
}
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
path: './data',
|
||||
compression: true // 60-80% space savings
|
||||
}
|
||||
})
|
||||
|
||||
// Performance impact:
|
||||
|
|
@ -359,11 +359,11 @@ const brain = new Brainy({
|
|||
### 5. Batch Operations
|
||||
|
||||
```typescript
|
||||
// v4.0.0: Efficient batch delete
|
||||
// Efficient batch delete
|
||||
await storage.batchDelete([
|
||||
'entities/nouns/vectors/00/00123456-....json',
|
||||
'entities/nouns/metadata/00/00123456-....json',
|
||||
// ... up to 1000 objects
|
||||
'entities/nouns/vectors/00/00123456-....json',
|
||||
'entities/nouns/metadata/00/00123456-....json',
|
||||
// ... up to 1000 objects
|
||||
])
|
||||
|
||||
// Benefits:
|
||||
|
|
@ -375,9 +375,9 @@ await storage.batchDelete([
|
|||
|
||||
// Batch writes for performance
|
||||
await brain.addBatch([
|
||||
{ content: "item1", metadata: {} },
|
||||
{ content: "item2", metadata: {} },
|
||||
{ content: "item3", metadata: {} }
|
||||
{ content: "item1", metadata: {} },
|
||||
{ content: "item2", metadata: {} },
|
||||
{ content: "item3", metadata: {} }
|
||||
])
|
||||
// Single transaction, optimized I/O
|
||||
```
|
||||
|
|
@ -390,14 +390,14 @@ const status = await storage.getStorageStatus()
|
|||
|
||||
console.log(status)
|
||||
// {
|
||||
// type: 'opfs',
|
||||
// available: true,
|
||||
// details: {
|
||||
// usage: 45829120, // 43.7 MB used
|
||||
// quota: 536870912, // 512 MB available
|
||||
// usagePercent: 8.5,
|
||||
// quotaExceeded: false
|
||||
// }
|
||||
// type: 'opfs',
|
||||
// available: true,
|
||||
// details: {
|
||||
// usage: 45829120, // 43.7 MB used
|
||||
// quota: 536870912, // 512 MB available
|
||||
// usagePercent: 8.5,
|
||||
// quotaExceeded: false
|
||||
// }
|
||||
// }
|
||||
|
||||
// Proactive quota management:
|
||||
|
|
@ -410,14 +410,14 @@ console.log(status)
|
|||
|
||||
```typescript
|
||||
// Change blob tier for cost optimization
|
||||
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
|
||||
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
|
||||
await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings)
|
||||
await storage.changeBlobTier(blobPath, 'Archive') // Cool → Archive (99% savings)
|
||||
|
||||
// Batch tier changes (efficient)
|
||||
await storage.batchChangeTier([blob1, blob2, blob3], 'Cool')
|
||||
|
||||
// Rehydrate from Archive when needed
|
||||
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
|
||||
await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
|
||||
```
|
||||
|
||||
### 8. Caching Strategy
|
||||
|
|
@ -425,15 +425,15 @@ await storage.rehydrateBlob(blobPath, 'Standard') // Standard or High priority
|
|||
```typescript
|
||||
// Configure caching per storage type
|
||||
const brain = new Brainy({
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
ttl: 300000, // 5 minutes
|
||||
strategy: 'lru' // Least recently used
|
||||
}
|
||||
}
|
||||
storage: {
|
||||
type: 'filesystem',
|
||||
cache: {
|
||||
enabled: true,
|
||||
maxSize: 1000, // Maximum cached items
|
||||
ttl: 300000, // 5 minutes
|
||||
strategy: 'lru' // Least recently used
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -443,8 +443,8 @@ const brain = new Brainy({
|
|||
```typescript
|
||||
// Automatic locking for write operations
|
||||
await brain.storage.withLock('resource-id', async () => {
|
||||
// Exclusive access to resource
|
||||
await brain.storage.saveNoun(id, data)
|
||||
// Exclusive access to resource
|
||||
await brain.storage.saveNoun(id, data)
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -459,9 +459,9 @@ await brain.storage.withLock('resource-id', async () => {
|
|||
```typescript
|
||||
// Export entire database
|
||||
const backup = await brain.export({
|
||||
format: 'json',
|
||||
includeVectors: true,
|
||||
includeIndexes: false
|
||||
format: 'json',
|
||||
includeVectors: true,
|
||||
includeIndexes: false
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -469,8 +469,8 @@ const backup = await brain.export({
|
|||
```typescript
|
||||
// Import from backup
|
||||
await brain.import(backup, {
|
||||
mode: 'merge', // or 'replace'
|
||||
validateSchema: true
|
||||
mode: 'merge', // or 'replace'
|
||||
validateSchema: true
|
||||
})
|
||||
```
|
||||
|
||||
|
|
@ -514,15 +514,15 @@ await newBrain.import(data)
|
|||
const stats = await brain.storage.getStatistics()
|
||||
console.log(stats)
|
||||
// {
|
||||
// totalSize: 1048576,
|
||||
// entityCount: 1000,
|
||||
// indexSize: 204800,
|
||||
// walSize: 10240,
|
||||
// cacheHitRate: 0.85
|
||||
// totalSize: 1048576,
|
||||
// entityCount: 1000,
|
||||
// indexSize: 204800,
|
||||
// walSize: 10240,
|
||||
// cacheHitRate: 0.85
|
||||
// }
|
||||
```
|
||||
|
||||
## Best Practices (v4.0.0)
|
||||
## Best Practices
|
||||
|
||||
### Choose the Right Adapter
|
||||
1. **Development**: FileSystem with compression (local persistence, small storage footprint)
|
||||
|
|
@ -537,7 +537,7 @@ console.log(stats)
|
|||
4. **Archival**: Cloud storage with lifecycle policies (96% cost savings!)
|
||||
5. **Large-scale**: Metadata/vector separation + UUID sharding + lifecycle policies
|
||||
|
||||
### v4.0.0 Cost Optimization
|
||||
### Cost Optimization
|
||||
1. **Enable lifecycle policies** for cloud storage (automated cost reduction)
|
||||
2. **Use Intelligent-Tiering (S3)** or Autoclass (GCS) for automatic optimization
|
||||
3. **Enable compression** for FileSystem storage (60-80% space savings)
|
||||
|
|
@ -547,7 +547,7 @@ console.log(stats)
|
|||
|
||||
**Example Cost Savings (500TB dataset):**
|
||||
- Without lifecycle policies: **$138,000/year**
|
||||
- With v4.0.0 lifecycle policies: **$5,940/year**
|
||||
- With lifecycle policies: **$5,940/year**
|
||||
- **Savings: $132,060/year (96%)**
|
||||
|
||||
### Monitor and Maintain
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue