From 994276f09fcb9e0b667378d440c7ef7a4174af37 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Mon, 25 Aug 2025 10:15:38 -0700 Subject: [PATCH] docs: consolidate and archive redundant documentation - Archived 13 API design iterations to docs/api-design-archive/ - Consolidated augmentation docs to docs/augmentations-archive/ - Maintained ONE definitive API doc at docs/api/README.md - Cleaned up documentation structure for 2.0 release - Preserved all historical documents for reference --- .../API-CRITICAL-FIXES.md | 0 .../API-QUICK-REFERENCE.md | 0 .../API-REFERENCE.md | 4 +- .../BRAINY-2.0-COMPLETE-API.md | 6 +- .../BRAINY-2.0-CORRECT-API.md | 4 +- .../BRAINY-2.0-DETAILED-API.md | 2 +- .../BRAINY-2.0-FINAL-API.md | 2 +- .../BRAINY-2.0-FINAL-CORRECT-API.md | 4 +- .../BRAINY-2.0-SIMPLIFIED-API.md | 8 +- .../BRAINY-2.0-UNIFIED-API.md | 4 +- .../COMPLETE-PUBLIC-API.md | 2 +- .../CRITICAL-API-AUDIT.md | 0 .../FINAL-2.0-PUBLIC-API.md | 6 +- docs/api-design-archive/README.md | 28 + docs/api/README.md | 555 +++++++++++------- docs/architecture/noun-verb-taxonomy.md | 2 +- docs/architecture/overview.md | 4 +- docs/architecture/triple-intelligence.md | 8 +- .../API-EXPOSURE-ARCHITECTURE.md | 0 .../AUGMENTATION-EXAMPLES.md | 0 .../AUGMENTATION-PIPELINE-ARCHITECTURE.md | 0 .../AUGMENTATION-TYPES-SIMPLE-GUIDE.md | 0 .../COMPLETE-ARCHITECTURE-VISION.md | 0 docs/augmentations-archive/README.md | 18 + .../STORAGE-AUGMENTATIONS-GUIDE.md | 0 .../UNIFIED-AUGMENTATION-SYSTEM.md | 0 docs/features/complete-feature-list.md | 2 +- docs/guides/getting-started.md | 4 +- docs/guides/natural-language.md | 2 +- docs/planning/FINAL_RELEASE_ASSESSMENT.md | 8 +- docs/planning/IMPLEMENTATION_STATUS.md | 2 +- .../planning/IMPLEMENTATION_STATUS_UPDATED.md | 2 +- docs/planning/RELEASE_READINESS.md | 2 +- docs/planning/TEST_COVERAGE_ANALYSIS.md | 2 +- docs/troubleshooting.md | 2 +- 35 files changed, 444 insertions(+), 239 deletions(-) rename docs/{api-design => api-design-archive}/API-CRITICAL-FIXES.md (100%) rename docs/{api-design => api-design-archive}/API-QUICK-REFERENCE.md (100%) rename docs/{api-design => api-design-archive}/API-REFERENCE.md (98%) rename docs/{api-design => api-design-archive}/BRAINY-2.0-COMPLETE-API.md (97%) rename docs/{api-design => api-design-archive}/BRAINY-2.0-CORRECT-API.md (98%) rename docs/{api-design => api-design-archive}/BRAINY-2.0-DETAILED-API.md (99%) rename docs/{api-design => api-design-archive}/BRAINY-2.0-FINAL-API.md (99%) rename docs/{api-design => api-design-archive}/BRAINY-2.0-FINAL-CORRECT-API.md (98%) rename docs/{api-design => api-design-archive}/BRAINY-2.0-SIMPLIFIED-API.md (96%) rename docs/{api-design => api-design-archive}/BRAINY-2.0-UNIFIED-API.md (98%) rename docs/{api-design => api-design-archive}/COMPLETE-PUBLIC-API.md (99%) rename docs/{api-design => api-design-archive}/CRITICAL-API-AUDIT.md (100%) rename docs/{api-design => api-design-archive}/FINAL-2.0-PUBLIC-API.md (97%) create mode 100644 docs/api-design-archive/README.md rename docs/{ => augmentations-archive}/API-EXPOSURE-ARCHITECTURE.md (100%) rename docs/{ => augmentations-archive}/AUGMENTATION-EXAMPLES.md (100%) rename docs/{ => augmentations-archive}/AUGMENTATION-PIPELINE-ARCHITECTURE.md (100%) rename docs/{ => augmentations-archive}/AUGMENTATION-TYPES-SIMPLE-GUIDE.md (100%) rename docs/{ => augmentations-archive}/COMPLETE-ARCHITECTURE-VISION.md (100%) create mode 100644 docs/augmentations-archive/README.md rename docs/{ => augmentations-archive}/STORAGE-AUGMENTATIONS-GUIDE.md (100%) rename docs/{ => augmentations-archive}/UNIFIED-AUGMENTATION-SYSTEM.md (100%) diff --git a/docs/api-design/API-CRITICAL-FIXES.md b/docs/api-design-archive/API-CRITICAL-FIXES.md similarity index 100% rename from docs/api-design/API-CRITICAL-FIXES.md rename to docs/api-design-archive/API-CRITICAL-FIXES.md diff --git a/docs/api-design/API-QUICK-REFERENCE.md b/docs/api-design-archive/API-QUICK-REFERENCE.md similarity index 100% rename from docs/api-design/API-QUICK-REFERENCE.md rename to docs/api-design-archive/API-QUICK-REFERENCE.md diff --git a/docs/api-design/API-REFERENCE.md b/docs/api-design-archive/API-REFERENCE.md similarity index 98% rename from docs/api-design/API-REFERENCE.md rename to docs/api-design-archive/API-REFERENCE.md index 8141beab..bbb3f3c1 100644 --- a/docs/api-design/API-REFERENCE.md +++ b/docs/api-design-archive/API-REFERENCE.md @@ -60,11 +60,11 @@ await brain.searchWithinNouns(ids[], query) // Search within specific nouns ### Triple Intelligence 🧠 ```typescript -await brain.find(query) // Unified Vector + Graph + Field search +await brain.find(query) // Unified Vector + Graph + Metadata search // Examples: await brain.find('documents about AI') // Natural language await brain.find({ like: 'sample-id' }) // Similar to ID -await brain.find({ where: { type: 'doc' }}) // Field filter +await brain.find({ where: { type: 'doc' }}) // Metadata filter await brain.find({ connected: { to: id }}) // Graph traversal ``` diff --git a/docs/api-design/BRAINY-2.0-COMPLETE-API.md b/docs/api-design-archive/BRAINY-2.0-COMPLETE-API.md similarity index 97% rename from docs/api-design/BRAINY-2.0-COMPLETE-API.md rename to docs/api-design-archive/BRAINY-2.0-COMPLETE-API.md index e5077176..43c5fd8b 100644 --- a/docs/api-design/BRAINY-2.0-COMPLETE-API.md +++ b/docs/api-design-archive/BRAINY-2.0-COMPLETE-API.md @@ -47,14 +47,14 @@ deleteVerbs(ids[]) // Delete multiple verbs // Core Search search(query, k?, options?) // Primary vector search searchText(text, k?, options?) // Natural language search -find(query) // Triple Intelligence (Vector+Graph+Field) 🧠 +find(query) // Triple Intelligence (Vector+Graph+Metadata) 🧠 findSimilar(id, k?, options?) // Find similar to existing noun // Advanced Search searchByNounTypes(types[], query, k?) // Filter by noun types searchWithinItems(query, ids[], k?) // Search within specific nouns searchWithCursor(query, cursor) // Paginated search -searchByStandardField(field, value, k?) // Field-based search +searchByStandardField(field, value, k?) // Metadata-based search // Graph Search searchVerbs(query, options?) // Search relationships @@ -206,7 +206,7 @@ These are now private (use new methods above): - `addItem()`, `addToBoth()`, `addBatch()`, `getBatch()` ### ✅ Triple Intelligence -- New `find()` method unifies Vector + Graph + Field search +- New `find()` method unifies Vector + Graph + Metadata search - Most powerful search capability in one simple method ### ✅ Zero-Configuration diff --git a/docs/api-design/BRAINY-2.0-CORRECT-API.md b/docs/api-design-archive/BRAINY-2.0-CORRECT-API.md similarity index 98% rename from docs/api-design/BRAINY-2.0-CORRECT-API.md rename to docs/api-design-archive/BRAINY-2.0-CORRECT-API.md index 44a5842d..078692db 100644 --- a/docs/api-design/BRAINY-2.0-CORRECT-API.md +++ b/docs/api-design-archive/BRAINY-2.0-CORRECT-API.md @@ -48,7 +48,7 @@ find({ // Vector search like: 'text' | vector | {id: 'noun-id'}, - // Field filtering with BRAINY OPERATORS (NOT MongoDB!) + // Metadata filtering with BRAINY OPERATORS (NOT MongoDB!) where: { // Direct equality field: value, @@ -261,7 +261,7 @@ brain.generateRandomGraph(nodes, edges) // Test data 3. **Simple neuralImport** - One method, smart detection 4. **Visualization exports** - For D3, Cytoscape, GraphML 5. **search() is just convenience** - Not a complete alias -6. **find() has Triple Intelligence** - Vector + Graph + Field +6. **find() has Triple Intelligence** - Vector + Graph + Metadata 7. **Proper operator names** - greaterThan not $gt 8. **Complete clustering API** - Fast, large, streaming options 9. **Synapses and Conduits** - External and internal sync diff --git a/docs/api-design/BRAINY-2.0-DETAILED-API.md b/docs/api-design-archive/BRAINY-2.0-DETAILED-API.md similarity index 99% rename from docs/api-design/BRAINY-2.0-DETAILED-API.md rename to docs/api-design-archive/BRAINY-2.0-DETAILED-API.md index 29c26099..1a5a701e 100644 --- a/docs/api-design/BRAINY-2.0-DETAILED-API.md +++ b/docs/api-design-archive/BRAINY-2.0-DETAILED-API.md @@ -126,7 +126,7 @@ Simple vector similarity search (convenience wrapper) // Vector similarity like?: string | number[] | {id: string}, // Text, vector, or noun ID - // Field filtering + // Metadata filtering where?: { field: value, // Exact match field: {$in: [values]}, // In array diff --git a/docs/api-design/BRAINY-2.0-FINAL-API.md b/docs/api-design-archive/BRAINY-2.0-FINAL-API.md similarity index 99% rename from docs/api-design/BRAINY-2.0-FINAL-API.md rename to docs/api-design-archive/BRAINY-2.0-FINAL-API.md index fc7c8390..ff6cdeab 100644 --- a/docs/api-design/BRAINY-2.0-FINAL-API.md +++ b/docs/api-design-archive/BRAINY-2.0-FINAL-API.md @@ -242,7 +242,7 @@ mode // Current mode (readonly) ✅ **Neural Import** - Smart AI-powered data import ✅ **Clustering** - Automatic and manual clustering -✅ **Triple Intelligence** - Vector + Graph + Field combined +✅ **Triple Intelligence** - Vector + Graph + Metadata combined ✅ **Verb Scoring** - Intelligent relationship scoring ✅ **Synapses** - External platform connectors ✅ **Conduits** - Brainy-to-Brainy sync diff --git a/docs/api-design/BRAINY-2.0-FINAL-CORRECT-API.md b/docs/api-design-archive/BRAINY-2.0-FINAL-CORRECT-API.md similarity index 98% rename from docs/api-design/BRAINY-2.0-FINAL-CORRECT-API.md rename to docs/api-design-archive/BRAINY-2.0-FINAL-CORRECT-API.md index eb9b6bf2..43a13d40 100644 --- a/docs/api-design/BRAINY-2.0-FINAL-CORRECT-API.md +++ b/docs/api-design-archive/BRAINY-2.0-FINAL-CORRECT-API.md @@ -48,7 +48,7 @@ Just TWO methods - simple and powerful: ```typescript search(query, k?) // Convenience: same as find({like: query, limit: k}) -find(query) // TRIPLE INTELLIGENCE: Vector + Graph + Field +find(query) // TRIPLE INTELLIGENCE: Vector + Graph + Metadata ``` ### Find Query (with CORRECT Brainy Operators): @@ -229,7 +229,7 @@ BrainyData.warmup() // Warmup 1. **Zero-Config** - Works instantly, no setup 2. **Auto-Embedding** - Text automatically becomes vectors -3. **Triple Intelligence** - Vector + Graph + Field combined +3. **Triple Intelligence** - Vector + Graph + Metadata combined 4. **Brainy Operators** - Clean, legal, no MongoDB style 5. **Complete Neural API** - All clustering/viz features 6. **Simple Import** - One method, auto-detects everything diff --git a/docs/api-design/BRAINY-2.0-SIMPLIFIED-API.md b/docs/api-design-archive/BRAINY-2.0-SIMPLIFIED-API.md similarity index 96% rename from docs/api-design/BRAINY-2.0-SIMPLIFIED-API.md rename to docs/api-design-archive/BRAINY-2.0-SIMPLIFIED-API.md index fa108fe6..8b283082 100644 --- a/docs/api-design/BRAINY-2.0-SIMPLIFIED-API.md +++ b/docs/api-design-archive/BRAINY-2.0-SIMPLIFIED-API.md @@ -53,7 +53,7 @@ find('documents about AI') // Similar to existing noun find({ like: 'noun-id-123' }) -// Field filtering +// Metadata filtering find({ where: { type: 'article' }}) // Graph traversal @@ -62,7 +62,7 @@ find({ connected: { to: 'id', via: 'references' }}) // Combined queries (Triple Intelligence!) find({ like: 'sample-doc', // Vector similarity - where: { status: 'published' }, // Field filter + where: { status: 'published' }, // Metadata filter connected: { via: 'cites' }, // Graph relationships limit: 10 // Pagination }) @@ -127,7 +127,7 @@ shutdown() // Cleanup ### Why So Simple? 1. **`addNoun()` handles everything** - Text? Auto-embeds. Vector? Uses directly. -2. **`find()` is the ultimate search** - Combines vector, graph, and field search +2. **`find()` is the ultimate search** - Combines vector, graph, and metadata search 3. **`search()` is just convenience** - Simple alias to `find()` for basic queries 4. **No duplicate methods** - One way to do each thing @@ -136,7 +136,7 @@ shutdown() // Cleanup The `find()` method is your Swiss Army knife: - Text search → Auto-embeds and searches - Vector search → `{ like: 'id' }` or `{ like: vector }` -- Field search → `{ where: { field: value }}` +- Metadata search → `{ where: { field: value }}` - Graph search → `{ connected: { to/from: 'id' }}` - Combine them all → Triple Intelligence! diff --git a/docs/api-design/BRAINY-2.0-UNIFIED-API.md b/docs/api-design-archive/BRAINY-2.0-UNIFIED-API.md similarity index 98% rename from docs/api-design/BRAINY-2.0-UNIFIED-API.md rename to docs/api-design-archive/BRAINY-2.0-UNIFIED-API.md index 7c0271cb..0adc2450 100644 --- a/docs/api-design/BRAINY-2.0-UNIFIED-API.md +++ b/docs/api-design-archive/BRAINY-2.0-UNIFIED-API.md @@ -54,7 +54,7 @@ search(query, k?) // Simple vector search // Equivalent to: find({like: query, limit: k}) find(query) // TRIPLE INTELLIGENCE 🧠 - // Combines Vector + Graph + Field search + // Combines Vector + Graph + Metadata search ``` ### Find Query Structure @@ -329,7 +329,7 @@ brain.initialized // Is initialized? 1. **Brainy Operators** - NOT MongoDB style ($gt, $lt) 2. **Neural API** - Complete with visualization export 3. **Simple Search** - Just `search()` and `find()` -4. **Triple Intelligence** - Vector + Graph + Field in `find()` +4. **Triple Intelligence** - Vector + Graph + Metadata in `find()` 5. **Auto-embedding** - `addNoun()` accepts text directly 6. **Unified Methods** - `getNouns()` handles all plural queries 7. **Clean Architecture** - Augmentation system for extensibility diff --git a/docs/api-design/COMPLETE-PUBLIC-API.md b/docs/api-design-archive/COMPLETE-PUBLIC-API.md similarity index 99% rename from docs/api-design/COMPLETE-PUBLIC-API.md rename to docs/api-design-archive/COMPLETE-PUBLIC-API.md index cb4bed61..ac94b2d4 100644 --- a/docs/api-design/COMPLETE-PUBLIC-API.md +++ b/docs/api-design-archive/COMPLETE-PUBLIC-API.md @@ -60,7 +60,7 @@ searchByNounTypes(types[], query) // Filter by noun types searchWithinItems(query, ids[], k?) // Search within specific nouns searchVerbs(query) // Search relationships searchNounsByVerbs(conditions) // Graph-based noun search -searchByStandardField(field, value) // Field-based search +searchByStandardField(field, value) // Metadata-based search ``` ### Distributed Search diff --git a/docs/api-design/CRITICAL-API-AUDIT.md b/docs/api-design-archive/CRITICAL-API-AUDIT.md similarity index 100% rename from docs/api-design/CRITICAL-API-AUDIT.md rename to docs/api-design-archive/CRITICAL-API-AUDIT.md diff --git a/docs/api-design/FINAL-2.0-PUBLIC-API.md b/docs/api-design-archive/FINAL-2.0-PUBLIC-API.md similarity index 97% rename from docs/api-design/FINAL-2.0-PUBLIC-API.md rename to docs/api-design-archive/FINAL-2.0-PUBLIC-API.md index 4789ebe8..7717a7a1 100644 --- a/docs/api-design/FINAL-2.0-PUBLIC-API.md +++ b/docs/api-design-archive/FINAL-2.0-PUBLIC-API.md @@ -47,14 +47,14 @@ deleteVerbs(ids[]) // Delete multiple verbs // Core Search search(query, k?, options?) // Primary vector search searchText(text, k?, options?) // Natural language search -find(query) // Triple Intelligence (Vector+Graph+Field) 🧠 +find(query) // Triple Intelligence (Vector+Graph+Metadata) 🧠 findSimilar(id, k?, options?) // Find similar to existing noun // Advanced Search searchByNounTypes(types[], query, k?) // Filter by noun types searchWithinItems(query, ids[], k?) // Search within specific nouns searchWithCursor(query, cursor) // Paginated search -searchByStandardField(field, value, k?) // Field-based search +searchByStandardField(field, value, k?) // Metadata-based search // Graph Search searchVerbs(query, options?) // Search relationships @@ -206,5 +206,5 @@ These are now private (use new methods above): - `addItem()`, `addToBoth()`, `addBatch()`, `getBatch()` ### ✅ Triple Intelligence -- New `find()` method unifies Vector + Graph + Field search +- New `find()` method unifies Vector + Graph + Metadata search - Most powerful search capability in one simple method \ No newline at end of file diff --git a/docs/api-design-archive/README.md b/docs/api-design-archive/README.md new file mode 100644 index 00000000..018828bf --- /dev/null +++ b/docs/api-design-archive/README.md @@ -0,0 +1,28 @@ +# API Design Archive + +This directory contains historical API design documents from the Brainy 2.0 development process. + +## Purpose +These documents represent the iterative refinement of the Brainy 2.0 API during development. They are preserved here for historical reference and to document the design decisions made along the way. + +## Current API Documentation +The definitive API documentation is now located at: +- **`/docs/api/README.md`** - The ONE official API reference + +## Archived Documents +These documents show the evolution of the API design: +- Various iterations of API structure +- Exploration of different naming conventions +- Refinement of the Triple Intelligence concept +- Transition from MongoDB operators to Brainy operators +- Evolution of the Neural API + +## Key Decisions Made +1. **Terminology**: Vector + Graph + Metadata (not Field) +2. **Operators**: Brainy operators (greaterThan, lessThan) not MongoDB ($gt, $lt) +3. **Methods**: Specific noun/verb naming (addNoun, getNoun) +4. **Unification**: Single getNouns() method instead of multiple variants +5. **Neural API**: Complete clustering and visualization features + +## Note +These documents are archived, not deleted, to preserve the development history and rationale behind API decisions. \ No newline at end of file diff --git a/docs/api/README.md b/docs/api/README.md index 1bc397d5..e3b7ae43 100644 --- a/docs/api/README.md +++ b/docs/api/README.md @@ -1,236 +1,395 @@ -# API Reference +# 🧠 Brainy 2.0 API Reference -Complete API documentation for Brainy's multi-dimensional AI database. +> **The definitive API documentation for Brainy 2.0** +> Clean • Powerful • Zero-Configuration -## Core APIs +## Quick Start -### [BrainyData](./brainy-data.md) -The main entry point for all operations. - -### [Triple Intelligence](./triple-intelligence.md) -Unified query system for vector, graph, and field search. - -### [Storage](./storage.md) -Storage adapter interfaces and implementations. - -### [Entity Registry](./entity-registry.md) -High-performance entity deduplication system. - -### [Neural API](./neural-api.md) -Natural language processing and similarity operations. - -## Quick Reference - -### Initialization ```typescript -import { BrainyData } from 'brainy' - -const brain = new BrainyData({ - storage: { type: 'filesystem', path: './data' }, - vectors: { dimensions: 384 } -}) +import { BrainyData } from '@soulcraft/brainy' +const brain = new BrainyData() // Zero config! await brain.init() -``` -### Basic Operations +// Add data (text auto-embeds!) +await brain.addNoun('The future of AI is here') -#### Add Entities (Nouns) and Relationships (Verbs) -```typescript -// Add entities (nouns) with automatic embedding -const id = await brain.addNoun("Machine learning is fascinating", { - category: "technology", - timestamp: Date.now() -}) - -// Add relationships (verbs) between entities -const sourceId = await brain.addNoun("Research Paper") -const targetId = await brain.addNoun("Neural Networks") -await brain.addVerb(sourceId, targetId, "discusses", { - confidence: 0.95, - section: "methodology" -}) - -// Batch operations -const entities = ["Entity 1", "Entity 2", "Entity 3"] -for (const entity of entities) { - await brain.addNoun(entity, { type: "batch" }) -} -``` - -#### Search and Find -```typescript -// Simple semantic search -const results = await brain.search("AI and machine learning") - -// Natural language queries with find() -const nlpResults = await brain.find("research papers about neural networks from 2024") -// Automatically interprets: document type, topic, and time range - -// Advanced triple intelligence search with structured query -const structured = await brain.find({ - like: "neural networks", - where: { category: "research" }, - connected: { to: "team-id", depth: 2 }, - limit: 20 -}) - -// Complex natural language with multiple conditions -const complex = await brain.find("highly cited papers on deep learning with over 100 citations published in Nature") -// Automatically extracts: citation count, topic, publication venue -``` - -#### Get and Update -```typescript -// Get noun by ID -const noun = await brain.getNoun("noun-id") - -// Get verb (relationship) by ID -const verb = await brain.getVerb("verb-id") - -// Update noun metadata -await brain.updateNounMetadata("noun-id", { - verified: true, - lastModified: Date.now() -}) - -// Delete noun (soft delete by default) -await brain.deleteNoun("noun-id") - -// Delete verb (relationship) -await brain.deleteVerb("verb-id") -``` - -## Advanced Features - -### Augmentations -```typescript -import { - WALAugmentation, - EntityRegistryAugmentation, - BatchProcessingAugmentation -} from 'brainy' - -const brain = new BrainyData({ - augmentations: [ - new WALAugmentation(), - new EntityRegistryAugmentation({ maxCacheSize: 100000 }), - new BatchProcessingAugmentation({ batchSize: 100 }) - ] +// Search with Triple Intelligence +const results = await brain.find({ + like: 'artificial intelligence', + where: { year: { greaterThan: 2020 } }, + connected: { via: 'references' } }) ``` -### Event System +## Core Concepts + +### 🧬 Nouns +Vectors with metadata - the fundamental data unit in Brainy. + +### 🔗 Verbs +Relationships between nouns - the connections that create knowledge graphs. + +### 🧠 Triple Intelligence +Vector search + Graph traversal + Metadata filtering in one unified query. + +--- + +## API Reference + +### Data Operations + +#### Nouns (Vectors with Metadata) + +##### `addNoun(dataOrVector, metadata?)` +Add a single noun to the database. +- **dataOrVector**: `string | number[]` - Text (auto-embeds) or pre-computed vector +- **metadata**: `object` - Associated metadata +- **Returns**: `Promise` - The noun's ID + +##### `getNoun(id)` +Retrieve a noun by ID. +- **id**: `string` - The noun's ID +- **Returns**: `Promise` + +##### `updateNoun(id, dataOrVector?, metadata?)` +Update an existing noun. +- **id**: `string` - The noun's ID +- **dataOrVector**: `string | number[]` - New data/vector (optional) +- **metadata**: `object` - New metadata (optional) +- **Returns**: `Promise` + +##### `deleteNoun(id)` +Delete a noun. +- **id**: `string` - The noun's ID +- **Returns**: `Promise` + +##### `getNouns(options)` +Get multiple nouns (unified method). +- **options**: Can be: + - `string[]` - Array of IDs + - `{filter: object}` - Metadata filter + - `{limit: number, offset: number}` - Pagination +- **Returns**: `Promise` + +#### Verbs (Relationships) + +##### `addVerb(source, target, type, metadata?)` +Create a relationship between nouns. +- **source**: `string` - Source noun ID +- **target**: `string` - Target noun ID +- **type**: `string` - Relationship type +- **metadata**: `object` - Relationship metadata (optional) +- **Returns**: `Promise` - The verb's ID + +##### `getVerbsBySource(sourceId)` +Get all outgoing relationships. +- **sourceId**: `string` - Source noun ID +- **Returns**: `Promise` + +##### `getVerbsByTarget(targetId)` +Get all incoming relationships. +- **targetId**: `string` - Target noun ID +- **Returns**: `Promise` + +--- + +### Search Operations + +#### `search(query, k?)` +Simple vector similarity search. +- **query**: `string | number[]` - Text or vector +- **k**: `number` - Number of results (default: 10) +- **Returns**: `Promise` + +> 💡 This is equivalent to: `find({like: query, limit: k})` + +#### `find(query)` - Triple Intelligence 🧠 +The ultimate search method combining vector, graph, and metadata search. + ```typescript -brain.on('addNoun', (noun) => { - console.log('Noun added:', noun.id) -}) - -brain.on('addVerb', (verb) => { - console.log('Relationship created:', verb.type) -}) - -brain.on('search', (query, results) => { - console.log(`Search for "${query}" returned ${results.length} results`) -}) - -brain.on('error', (error) => { - console.error('Error occurred:', error) +find({ + // Vector similarity + like: 'text query' | vector | {id: 'noun-id'}, + + // Metadata filtering (Brainy operators) + where: { + field: value, // Exact match + field: { + equals: value, + greaterThan: value, + lessThan: value, + greaterEqual: value, + lessEqual: value, + oneOf: [val1, val2], // In array + notOneOf: [val1, val2], // Not in array + contains: value, // Array/string contains + startsWith: value, + endsWith: value, + matches: /pattern/, // Pattern match + between: [min, max] + } + }, + + // Graph traversal + connected: { + to: 'noun-id', // Target noun + from: 'noun-id', // Source noun + via: 'relationship-type', // Relationship type + depth: 2 // Traversal depth + }, + + // Control + limit: 10, // Max results + offset: 0, // Skip results + explain: false // Include explanation }) ``` -### Statistics +--- + +### Neural API + +Access advanced AI features via `brain.neural`: + +#### `brain.neural.similar(a, b)` +Calculate semantic similarity between two items. +- **Returns**: `Promise` - Similarity score (0-1) + +#### `brain.neural.clusters(options?)` +Automatically cluster nouns. +- **Returns**: `Promise` - Generated clusters + +#### `brain.neural.hierarchy(id)` +Build semantic hierarchy from a noun. +- **Returns**: `Promise` - Hierarchy structure + +#### `brain.neural.neighbors(id, k?)` +Find k-nearest neighbors. +- **Returns**: `Promise` - Nearest neighbors + +#### `brain.neural.outliers(threshold?)` +Detect outlier nouns. +- **Returns**: `Promise` - Outlier IDs + +#### `brain.neural.visualize(options?)` +Generate visualization data for external tools. ```typescript -const stats = await brain.statistics() -console.log(` - Total items: ${stats.totalItems} - Index size: ${stats.indexSize} - Average query time: ${stats.avgQueryTime}ms -`) +visualize({ + maxNodes: 100, + dimensions: 2 | 3, + algorithm: 'force' | 'hierarchical' | 'radial', + includeEdges: true +}) +// Returns format for D3, Cytoscape, or GraphML ``` -## Type Definitions +--- -### Core Types -```typescript -interface SearchResult { - id: string - score: number - content?: string - metadata?: Record -} +### Import & Export -interface TripleQuery { - like?: string | Vector | any - where?: Record - connected?: ConnectionQuery - limit?: number - threshold?: number -} +#### `neuralImport(data, options?)` +AI-powered smart import that auto-detects format. +- **data**: `any` - Data to import +- **options**: Import configuration + - `confidenceThreshold`: Minimum confidence (0-1) + - `autoApply`: Automatically add to database + - `skipDuplicates`: Skip existing entities +- **Returns**: Detected entities and relationships -interface Vector { - values: number[] - dimensions: number -} -``` +#### `backup()` +Create a full backup. +- **Returns**: `Promise` -## Error Handling +#### `restore(backup)` +Restore from backup. +- **backup**: `BackupData` - Previous backup +- **Returns**: `Promise` -All methods follow consistent error handling: +--- -```typescript -try { - await brain.addNoun("content", metadata) -} catch (error) { - if (error.code === 'STORAGE_ERROR') { - // Handle storage issues - } else if (error.code === 'VALIDATION_ERROR') { - // Handle validation issues - } -} -``` +### Intelligence Features -## Performance Guidelines +#### Verb Scoring +Train the relationship scoring model: +- `provideFeedbackForVerbScoring(feedback)` - Train model +- `getVerbScoringStats()` - Get statistics +- `exportVerbScoringLearningData()` - Export training +- `importVerbScoringLearningData(data)` - Import training -### Batching -Always use batch operations for bulk data: -```typescript -// Good - efficient batch processing -const items = ["item1", "item2", "item3"] -for (const item of items) { - await brain.addNoun(item, { batch: true }) -} +#### Embeddings +- `embed(text)` - Generate embedding vector +- `calculateSimilarity(a, b, metric?)` - Calculate similarity -// For relationships -const relationships = [ - { source: id1, target: id2, type: "related" }, - { source: id2, target: id3, type: "similar" } -] -for (const rel of relationships) { - await brain.addVerb(rel.source, rel.target, rel.type) -} -``` +--- -### Caching -Configure caching for your use case: +### Configuration & Management + +#### Operational Modes +- `setReadOnly(bool)` - Toggle read-only mode +- `setWriteOnly(bool)` - Toggle write-only mode +- `setFrozen(bool)` - Freeze all modifications + +#### Cache & Performance +- `getCacheStats()` - Get cache statistics +- `clearCache()` - Clear search cache +- `size()` - Get total noun count +- `getStatistics()` - Get full statistics + +#### Data Management +- `clear(options?)` - Clear all data +- `clearNouns()` - Clear nouns only +- `clearVerbs()` - Clear verbs only +- `rebuildMetadataIndex()` - Rebuild index + +--- + +### Lifecycle + +#### Initialization ```typescript const brain = new BrainyData({ - cache: { - search: { maxSize: 100, ttl: 60000 }, - metadata: { maxSize: 1000, ttl: 300000 } - } + storage: 'auto', // auto | memory | filesystem | s3 + dimensions: 384, // Vector dimensions + cache: true, // Enable caching + index: true // Enable indexing }) + +await brain.init() // Required before use! ``` -### Indexing -Ensure fields used in queries are indexed: +#### Cleanup ```typescript -// Configure indexed fields -const brain = new BrainyData({ - indexedFields: ['category', 'author', 'timestamp'] +await brain.shutdown() // Graceful shutdown +``` + +#### Static Methods +- `BrainyData.preloadModel()` - Preload ML model +- `BrainyData.warmup()` - Warmup system + +--- + +## Query Operators Reference + +Brainy uses its own clean, readable operators: + +| Brainy Operator | Description | Example | +|-----------------|-------------|---------| +| `equals` | Exact match | `{age: {equals: 25}}` | +| `greaterThan` | Greater than | `{age: {greaterThan: 18}}` | +| `lessThan` | Less than | `{price: {lessThan: 100}}` | +| `greaterEqual` | Greater or equal | `{score: {greaterEqual: 90}}` | +| `lessEqual` | Less or equal | `{rating: {lessEqual: 3}}` | +| `oneOf` | In array | `{color: {oneOf: ['red', 'blue']}}` | +| `notOneOf` | Not in array | `{status: {notOneOf: ['deleted']}}` | +| `contains` | Contains value | `{tags: {contains: 'ai'}}` | +| `startsWith` | String prefix | `{name: {startsWith: 'John'}}` | +| `endsWith` | String suffix | `{email: {endsWith: '@gmail.com'}}` | +| `matches` | Pattern match | `{text: {matches: /^[A-Z]/}}` | +| `between` | Range | `{year: {between: [2020, 2024]}}` | + +--- + +## Examples + +### Basic Usage +```typescript +// Add data +const id = await brain.addNoun('Quantum computing breakthrough', { + category: 'technology', + year: 2024, + importance: 'high' +}) + +// Simple search +const results = await brain.search('quantum physics', 5) + +// Complex query with Triple Intelligence +const articles = await brain.find({ + like: 'quantum computing', + where: { + year: { greaterThan: 2022 }, + importance: { oneOf: ['high', 'critical'] } + }, + connected: { + via: 'references', + depth: 2 + }, + limit: 10 }) ``` -## Migration from v1.x +### Creating Knowledge Graphs +```typescript +// Add entities +const ai = await brain.addNoun('Artificial Intelligence') +const ml = await brain.addNoun('Machine Learning') +const dl = await brain.addNoun('Deep Learning') -See the [Migration Guide](../MIGRATION.md) for upgrading from Brainy 1.x to 2.0. \ No newline at end of file +// Create relationships +await brain.addVerb(ml, ai, 'subset_of') +await brain.addVerb(dl, ml, 'subset_of') +await brain.addVerb(dl, ai, 'enables') + +// Traverse the graph +const aiEcosystem = await brain.find({ + connected: { from: ai, depth: 3 } +}) +``` + +### Using Neural Features +```typescript +// Find similar concepts +const similarity = await brain.neural.similar( + 'renewable energy', + 'sustainable power' +) + +// Auto-cluster documents +const clusters = await brain.neural.clusters({ + method: 'kmeans', + k: 5 +}) + +// Generate visualization +const vizData = await brain.neural.visualize({ + maxNodes: 200, + algorithm: 'force', + dimensions: 3 +}) +// Use vizData with D3.js, Cytoscape, etc. +``` + +--- + +## Key Features + +### ✨ Zero Configuration +Works instantly with sensible defaults. No setup required. + +### 🧠 Triple Intelligence +Combines vector search, graph traversal, and metadata filtering in one query. + +### 🚀 Auto-Embedding +Text automatically converts to vectors - no manual embedding needed. + +### 📊 Built-in Visualization +Export data formatted for popular visualization libraries. + +### 🔒 Clean Operators +Readable, intuitive operators - no cryptic symbols. + +### 🎯 Everything Included +All features in the MIT licensed package - no premium tiers. + +--- + +## Support + +- **GitHub**: [github.com/soulcraft/brainy](https://github.com/soulcraft/brainy) +- **Documentation**: [docs.soulcraft.com/brainy](https://docs.soulcraft.com/brainy) +- **License**: MIT + +--- + +*Brainy 2.0 - Intelligence for Everyone* \ No newline at end of file diff --git a/docs/architecture/noun-verb-taxonomy.md b/docs/architecture/noun-verb-taxonomy.md index 21692502..e2f96d4e 100644 --- a/docs/architecture/noun-verb-taxonomy.md +++ b/docs/architecture/noun-verb-taxonomy.md @@ -462,7 +462,7 @@ for (const order of orders) { // Similar to graph databases but with added benefits: // 1. Automatic vector embeddings for similarity // 2. Natural language querying -// 3. Unified with field filtering +// 3. Unified with metadata filtering // Enhanced graph queries const results = await brain.find("similar users who purchased similar products") diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index f0ecbae1..29fb0961 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,6 +1,6 @@ # Architecture Overview -Brainy is a multi-dimensional AI database that combines vector similarity, graph relationships, and field filtering into a unified query system. This document provides a comprehensive overview of the system architecture. +Brainy is a multi-dimensional AI database that combines vector similarity, graph relationships, and metadata filtering into a unified query system. This document provides a comprehensive overview of the system architecture. ## Core Components @@ -23,7 +23,7 @@ Brainy's revolutionary feature that unifies three types of search: const results = await brain.find({ like: "machine learning papers", // Vector similarity connected: { to: "research-team", depth: 2 }, // Graph traversal - where: { published: { $gte: "2024-01-01" } } // Field filtering + where: { published: { $gte: "2024-01-01" } } // Metadata filtering }) ``` diff --git a/docs/architecture/triple-intelligence.md b/docs/architecture/triple-intelligence.md index 7bdfac3a..fe6f8d19 100644 --- a/docs/architecture/triple-intelligence.md +++ b/docs/architecture/triple-intelligence.md @@ -1,10 +1,10 @@ # Triple Intelligence System -The Triple Intelligence System is Brainy's revolutionary query engine that unifies vector similarity, graph relationships, and field filtering into a single, optimized query interface. +The Triple Intelligence System is Brainy's revolutionary query engine that unifies vector similarity, graph relationships, and metadata filtering into a single, optimized query interface. ## Overview -Traditional databases force you to choose between vector search, graph traversal, OR field filtering. Brainy combines all three intelligences into one magical API that automatically optimizes execution for maximum performance. +Traditional databases force you to choose between vector search, graph traversal, OR metadata filtering. Brainy combines all three intelligences into one magical API that automatically optimizes execution for maximum performance. ## Query Interface @@ -116,7 +116,7 @@ const results = await brain.find({ like: "recent posts", // Applied to filtered set limit: 5 }) -// Field filter first, then vector search on results +// Metadata filter first, then vector search on results ``` ## Fusion Ranking @@ -193,7 +193,7 @@ Brainy uses itself to optimize queries: Triple Intelligence leverages all available indexes: - **HNSW Index**: For vector similarity -- **Metadata Index**: For field filtering +- **Metadata Index**: For metadata filtering - **Graph Index**: For relationship traversal ## Advanced Features diff --git a/docs/API-EXPOSURE-ARCHITECTURE.md b/docs/augmentations-archive/API-EXPOSURE-ARCHITECTURE.md similarity index 100% rename from docs/API-EXPOSURE-ARCHITECTURE.md rename to docs/augmentations-archive/API-EXPOSURE-ARCHITECTURE.md diff --git a/docs/AUGMENTATION-EXAMPLES.md b/docs/augmentations-archive/AUGMENTATION-EXAMPLES.md similarity index 100% rename from docs/AUGMENTATION-EXAMPLES.md rename to docs/augmentations-archive/AUGMENTATION-EXAMPLES.md diff --git a/docs/AUGMENTATION-PIPELINE-ARCHITECTURE.md b/docs/augmentations-archive/AUGMENTATION-PIPELINE-ARCHITECTURE.md similarity index 100% rename from docs/AUGMENTATION-PIPELINE-ARCHITECTURE.md rename to docs/augmentations-archive/AUGMENTATION-PIPELINE-ARCHITECTURE.md diff --git a/docs/AUGMENTATION-TYPES-SIMPLE-GUIDE.md b/docs/augmentations-archive/AUGMENTATION-TYPES-SIMPLE-GUIDE.md similarity index 100% rename from docs/AUGMENTATION-TYPES-SIMPLE-GUIDE.md rename to docs/augmentations-archive/AUGMENTATION-TYPES-SIMPLE-GUIDE.md diff --git a/docs/COMPLETE-ARCHITECTURE-VISION.md b/docs/augmentations-archive/COMPLETE-ARCHITECTURE-VISION.md similarity index 100% rename from docs/COMPLETE-ARCHITECTURE-VISION.md rename to docs/augmentations-archive/COMPLETE-ARCHITECTURE-VISION.md diff --git a/docs/augmentations-archive/README.md b/docs/augmentations-archive/README.md new file mode 100644 index 00000000..cd05fb63 --- /dev/null +++ b/docs/augmentations-archive/README.md @@ -0,0 +1,18 @@ +# Augmentations Documentation Archive + +This directory contains historical augmentation documentation from the Brainy 2.0 development process. + +## Current Documentation +The definitive augmentation documentation is now located at: +- **`/docs/augmentations/README.md`** - Main augmentation guide +- **`/docs/architecture/augmentations.md`** - Architecture details + +## Archived Documents +These documents represent the evolution of the augmentation system design: +- Various implementation approaches +- Pipeline architecture exploration +- Storage augmentation patterns +- Example implementations + +## Note +These documents are archived to preserve development history while maintaining a clean documentation structure. \ No newline at end of file diff --git a/docs/STORAGE-AUGMENTATIONS-GUIDE.md b/docs/augmentations-archive/STORAGE-AUGMENTATIONS-GUIDE.md similarity index 100% rename from docs/STORAGE-AUGMENTATIONS-GUIDE.md rename to docs/augmentations-archive/STORAGE-AUGMENTATIONS-GUIDE.md diff --git a/docs/UNIFIED-AUGMENTATION-SYSTEM.md b/docs/augmentations-archive/UNIFIED-AUGMENTATION-SYSTEM.md similarity index 100% rename from docs/UNIFIED-AUGMENTATION-SYSTEM.md rename to docs/augmentations-archive/UNIFIED-AUGMENTATION-SYSTEM.md diff --git a/docs/features/complete-feature-list.md b/docs/features/complete-feature-list.md index c53c06b0..8ea51421 100644 --- a/docs/features/complete-feature-list.md +++ b/docs/features/complete-feature-list.md @@ -15,7 +15,7 @@ Unified query system that automatically combines: // All three intelligences work together automatically const results = await brain.find({ like: 'AI research', // Vector search - where: { year: 2024 }, // Field filtering + where: { year: 2024 }, // Metadata filtering connected: { to: authorId } // Graph traversal }) ``` diff --git a/docs/guides/getting-started.md b/docs/guides/getting-started.md index 623719b3..705d684a 100644 --- a/docs/guides/getting-started.md +++ b/docs/guides/getting-started.md @@ -1,6 +1,6 @@ # Getting Started with Brainy -This guide will help you get up and running with Brainy, the multi-dimensional AI database that combines vector similarity, graph relationships, and field filtering. +This guide will help you get up and running with Brainy, the multi-dimensional AI database that combines vector similarity, graph relationships, and metadata filtering. ## Installation @@ -90,7 +90,7 @@ results.forEach(result => { const results = await brain.find("show me technology articles about AI from 2023") // Automatically interprets: topic, category, and time range -// Structured queries with vector similarity and field filtering +// Structured queries with vector similarity and metadata filtering const structured = await brain.find({ like: "artificial intelligence", where: { diff --git a/docs/guides/natural-language.md b/docs/guides/natural-language.md index 48aa3822..47fdef7e 100644 --- a/docs/guides/natural-language.md +++ b/docs/guides/natural-language.md @@ -162,7 +162,7 @@ The natural language is converted to a structured Triple Intelligence query: ### 5. Execution The structured query is executed using Triple Intelligence, combining: - Vector similarity search -- Field filtering +- Metadata filtering - Graph traversal ## Advanced Features diff --git a/docs/planning/FINAL_RELEASE_ASSESSMENT.md b/docs/planning/FINAL_RELEASE_ASSESSMENT.md index 17c0e400..e34c0861 100644 --- a/docs/planning/FINAL_RELEASE_ASSESSMENT.md +++ b/docs/planning/FINAL_RELEASE_ASSESSMENT.md @@ -20,7 +20,7 @@ - ✅ Natural language queries ("find developers") - ✅ Vector similarity search (`similar: 'text'`) - ✅ Graph traversal (`connected: { to: 'node' }`) -- ✅ Field filtering (`where: { field: 'value' }`) +- ✅ Metadata filtering (`where: { field: 'value' }`) - ✅ Combined intelligence with fusion scoring - ✅ Performance optimized for complex queries - **Status: PRODUCTION READY** @@ -94,7 +94,7 @@ const viz = await neural.visualize() ``` ### Core Innovation Validated -- ✅ **Triple Intelligence**: Vector + Graph + Field search unified +- ✅ **Triple Intelligence**: Vector + Graph + Metadata search unified - ✅ **Intelligent Verb Scoring**: Smart relationship weights - ✅ **Neural APIs**: Ready for external visualization libraries - ✅ **Zero Config**: Works perfectly out of the box @@ -118,7 +118,7 @@ const viz = await neural.visualize() - **Scalability**: Tested with complex datasets ### Innovation Leadership ✅ -- **First True Triple Intelligence**: Vector + Graph + Field unified +- **First True Triple Intelligence**: Vector + Graph + Metadata unified - **Smart by Default**: No configuration required - **Revolutionary Data Model**: Noun-verb taxonomy - **Neural API**: Ready for external clustering/visualization libraries @@ -133,7 +133,7 @@ const viz = await neural.visualize() Brainy 2.0 represents a fundamental leap forward in vector database technology: -1. **Triple Intelligence** solves the problem of having to choose between vector, graph, or field search +1. **Triple Intelligence** solves the problem of having to choose between vector, graph, or metadata search 2. **Intelligent Verb Scoring** automatically computes optimal relationship weights 3. **Neural APIs** enable external libraries to build advanced visualizations 4. **Zero Configuration** makes it accessible to all developers diff --git a/docs/planning/IMPLEMENTATION_STATUS.md b/docs/planning/IMPLEMENTATION_STATUS.md index d7cf7c56..4d04ccc5 100644 --- a/docs/planning/IMPLEMENTATION_STATUS.md +++ b/docs/planning/IMPLEMENTATION_STATUS.md @@ -4,7 +4,7 @@ ### Core Features - ✅ **Noun-Verb Taxonomy** - Complete implementation with addNoun() and addVerb() -- ✅ **Triple Intelligence Engine** - Vector + Graph + Field unified queries +- ✅ **Triple Intelligence Engine** - Vector + Graph + Metadata unified queries - ✅ **Natural Language find()** - Basic NLP with 220+ embedded patterns - ✅ **HNSW Vector Search** - O(log n) similarity search - ✅ **Field Indexing** - O(1) metadata lookups via FieldIndex class diff --git a/docs/planning/IMPLEMENTATION_STATUS_UPDATED.md b/docs/planning/IMPLEMENTATION_STATUS_UPDATED.md index 5b02f3fe..1f1b77f9 100644 --- a/docs/planning/IMPLEMENTATION_STATUS_UPDATED.md +++ b/docs/planning/IMPLEMENTATION_STATUS_UPDATED.md @@ -6,7 +6,7 @@ After thorough investigation of the codebase, here's what's ACTUALLY implemented ### Core Features - ✅ **Noun-Verb Taxonomy** - Complete with addNoun() and addVerb() -- ✅ **Triple Intelligence Engine** - Vector + Graph + Field unified queries +- ✅ **Triple Intelligence Engine** - Vector + Graph + Metadata unified queries - ✅ **Natural Language find()** - Basic NLP with 220+ embedded patterns - ✅ **HNSW Vector Search** - O(log n) similarity search with partitioning support - ✅ **Field Indexing** - O(1) metadata lookups via FieldIndex class diff --git a/docs/planning/RELEASE_READINESS.md b/docs/planning/RELEASE_READINESS.md index be738938..9b6bf17a 100644 --- a/docs/planning/RELEASE_READINESS.md +++ b/docs/planning/RELEASE_READINESS.md @@ -6,7 +6,7 @@ ### Core Features (100% Complete) - ✅ **Noun-Verb Taxonomy**: Revolutionary data model -- ✅ **Triple Intelligence**: Vector + Graph + Field unified queries +- ✅ **Triple Intelligence**: Vector + Graph + Metadata unified queries - ✅ **HNSW Indexing**: O(log n) vector search - ✅ **384 Dimensions**: Fixed with all-MiniLM-L6-v2 - ✅ **Zero-Config**: Works out of the box diff --git a/docs/planning/TEST_COVERAGE_ANALYSIS.md b/docs/planning/TEST_COVERAGE_ANALYSIS.md index 60551ebc..1dfbe62a 100644 --- a/docs/planning/TEST_COVERAGE_ANALYSIS.md +++ b/docs/planning/TEST_COVERAGE_ANALYSIS.md @@ -32,7 +32,7 @@ - Real transformer models (no mocking) ### 6. Natural Language ✅ -- `tests/triple-intelligence.test.ts` - Vector + Graph + Field queries +- `tests/triple-intelligence.test.ts` - Vector + Graph + Metadata queries - Natural language query understanding ### 7. Error Handling ✅ diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 90b00cda..538c8430 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -225,7 +225,7 @@ const brain = new BrainyData({ const results = await brain.search("query", { limit: 10 }) ``` -3. **Consider field filtering first** +3. **Consider metadata filtering first** ```typescript // Filter by metadata first, then semantic search const results = await brain.search("query", {