feat: Universal Import with intelligent type matching (v2.1.0)
✨ ONE universal import method for everything - Auto-detects files, URLs, and raw data - Intelligent noun/verb type matching using embeddings - Support for JSON, CSV, YAML, and text formats - Zero configuration required 🧠 Intelligent Type Matching - Uses semantic embeddings to match 31 noun types - Automatically detects 40 verb relationship types - Confidence scores for type predictions - Caching for improved performance 📦 Import Manager - Centralized import logic with lazy loading - Integrates NeuralImportAugmentation for AI processing - Proper CSV parsing with quote handling - Basic YAML support 🎯 Simplified API - brain.import() - ONE method that handles everything - Auto-detection of URLs and file paths - Backwards compatible with existing code - Clean, modern, delightful developer experience 📚 Documentation - Comprehensive import guide in docs/guides/import-anything.md - Examples for every format and use case - Philosophy of simplicity and zero config ✅ Tests - Full unit test coverage for import functionality - Type matching tests for all 31 nouns and 40 verbs - Tests for CSV, YAML, JSON, and text formats BREAKING CHANGES: None - fully backward compatible
This commit is contained in:
parent
492267b509
commit
1ef01f394a
10 changed files with 1833 additions and 102 deletions
|
|
@ -8,7 +8,7 @@
|
|||
|
||||
Brainy's **Noun-Verb Taxonomy** achieves **universal coverage** of all human knowledge through **infinite expressiveness**:
|
||||
|
||||
- **24 Noun Types × 40 Verb Types = 960 Base Combinations**
|
||||
- **31 Noun Types × 40 Verb Types = 1,240 Base Combinations**
|
||||
- **Unlimited Metadata Fields = ∞ Domain Specificity**
|
||||
- **Multi-hop Graph Traversals = ∞ Relationship Complexity**
|
||||
- **Result: Can Model ANY Data in ANY Industry**
|
||||
|
|
@ -95,7 +95,7 @@ await brain.sync.jira({
|
|||
|
||||
Like **HTTP** became the protocol for the web and **TCP/IP** for the internet, Brainy's noun-verb taxonomy is becoming the **Universal Knowledge Protocol**:
|
||||
|
||||
- **Learn Once**: Developers learn 24 nouns + 40 verbs, not 1000s of schemas
|
||||
- **Learn Once**: Developers learn 31 nouns + 40 verbs, not 1000s of schemas
|
||||
- **Build Anywhere**: Tools built for one domain work in others
|
||||
- **Share Everything**: Knowledge graphs are universally shareable
|
||||
- **Compose Freely**: Augmentations compose without conflicts
|
||||
|
|
@ -574,7 +574,7 @@ const results = await brain.find("similar users who purchased similar products")
|
|||
|
||||
The Noun-Verb taxonomy is designed to represent **all human knowledge** through a comprehensive set of types that can be combined infinitely.
|
||||
|
||||
### Complete Noun Types (24 Types)
|
||||
### Complete Noun Types (31 Types)
|
||||
|
||||
#### Core Entity Types (6)
|
||||
|
||||
|
|
@ -746,7 +746,7 @@ await brain.addNoun("Website Redesign", {
|
|||
})
|
||||
```
|
||||
|
||||
#### Descriptive Types (6)
|
||||
#### Descriptive Types (7)
|
||||
|
||||
##### 19. **Process** - Workflows and procedures
|
||||
```typescript
|
||||
|
|
@ -775,30 +775,99 @@ await brain.addNoun("Admin Role", {
|
|||
})
|
||||
```
|
||||
|
||||
##### 22. **Relationship** - Abstract relationships
|
||||
##### 22. **Topic** - Subjects or themes
|
||||
```typescript
|
||||
await brain.addNoun("Partnership Agreement", {
|
||||
type: "relationship",
|
||||
parties: ["CompanyA", "CompanyB"],
|
||||
terms: "5 years"
|
||||
await brain.addNoun("Machine Learning", {
|
||||
type: "topic",
|
||||
field: "AI",
|
||||
popularity: "high"
|
||||
})
|
||||
```
|
||||
|
||||
##### 23. **Property** - Attributes or characteristics
|
||||
##### 23. **Language** - Languages or linguistic entities
|
||||
```typescript
|
||||
await brain.addNoun("Color Scheme", {
|
||||
type: "property",
|
||||
values: ["#000", "#FFF"],
|
||||
applies_to: "theme"
|
||||
await brain.addNoun("English", {
|
||||
type: "language",
|
||||
iso_code: "en",
|
||||
speakers_millions: 1500
|
||||
})
|
||||
```
|
||||
|
||||
##### 24. **Custom** - User-defined types
|
||||
##### 24. **Currency** - Monetary units
|
||||
```typescript
|
||||
await brain.addNoun("Industry-Specific Entity", {
|
||||
type: "custom",
|
||||
subtype: "medical_device",
|
||||
fda_approved: true
|
||||
await brain.addNoun("US Dollar", {
|
||||
type: "currency",
|
||||
symbol: "$",
|
||||
code: "USD"
|
||||
})
|
||||
```
|
||||
|
||||
##### 25. **Measurement** - Metrics or quantities
|
||||
```typescript
|
||||
await brain.addNoun("Temperature Reading", {
|
||||
type: "measurement",
|
||||
value: 23.5,
|
||||
unit: "celsius"
|
||||
})
|
||||
```
|
||||
|
||||
#### Scientific/Research Types (2)
|
||||
|
||||
##### 26. **Hypothesis** - Scientific theories and propositions
|
||||
```typescript
|
||||
await brain.addNoun("String Theory", {
|
||||
type: "hypothesis",
|
||||
field: "physics",
|
||||
status: "unproven"
|
||||
})
|
||||
```
|
||||
|
||||
##### 27. **Experiment** - Studies and research trials
|
||||
```typescript
|
||||
await brain.addNoun("Clinical Trial XYZ", {
|
||||
type: "experiment",
|
||||
phase: 3,
|
||||
participants: 1000
|
||||
})
|
||||
```
|
||||
|
||||
#### Legal/Regulatory Types (2)
|
||||
|
||||
##### 28. **Contract** - Legal agreements and terms
|
||||
```typescript
|
||||
await brain.addNoun("Service Agreement", {
|
||||
type: "contract",
|
||||
duration: "2 years",
|
||||
value: 100000
|
||||
})
|
||||
```
|
||||
|
||||
##### 29. **Regulation** - Laws and compliance requirements
|
||||
```typescript
|
||||
await brain.addNoun("GDPR", {
|
||||
type: "regulation",
|
||||
jurisdiction: "EU",
|
||||
category: "data protection"
|
||||
})
|
||||
```
|
||||
|
||||
#### Technical Infrastructure Types (2)
|
||||
|
||||
##### 30. **Interface** - APIs and protocols
|
||||
```typescript
|
||||
await brain.addNoun("REST API", {
|
||||
type: "interface",
|
||||
version: "v2",
|
||||
endpoints: 45
|
||||
})
|
||||
```
|
||||
|
||||
##### 31. **Resource** - Infrastructure and compute assets
|
||||
```typescript
|
||||
await brain.addNoun("Database Server", {
|
||||
type: "resource",
|
||||
capacity: "1TB",
|
||||
availability: "99.9%"
|
||||
})
|
||||
```
|
||||
|
||||
|
|
|
|||
250
docs/guides/import-anything.md
Normal file
250
docs/guides/import-anything.md
Normal file
|
|
@ -0,0 +1,250 @@
|
|||
# Import Anything - ONE Method, Infinite Intelligence 🚀
|
||||
|
||||
Brainy's import is **ONE magical method** that understands EVERYTHING:
|
||||
- 📊 Data (objects, arrays, strings)
|
||||
- 📁 Files (auto-detects by path)
|
||||
- 🌐 URLs (auto-fetches)
|
||||
- 📄 Formats (JSON, CSV, YAML, text - all auto-detected)
|
||||
|
||||
## The Ultimate Simplicity
|
||||
|
||||
```javascript
|
||||
import { BrainyData } from '@soulcraft/brainy'
|
||||
|
||||
const brain = new BrainyData()
|
||||
await brain.init()
|
||||
|
||||
// ONE method for EVERYTHING:
|
||||
await brain.import(anything)
|
||||
```
|
||||
|
||||
## Import Examples - It Just Works™
|
||||
|
||||
### 📊 Import JSON Data
|
||||
```javascript
|
||||
// Array of objects? No problem.
|
||||
const people = [
|
||||
{ name: 'Alice', role: 'Engineer', company: 'TechCorp' },
|
||||
{ name: 'Bob', role: 'Designer', company: 'TechCorp' }
|
||||
]
|
||||
|
||||
await brain.import(people)
|
||||
// ✨ Automatically detected as Person entities with Organization relationships!
|
||||
```
|
||||
|
||||
### 📄 Import CSV - File or String
|
||||
```javascript
|
||||
// From file? Just pass the path!
|
||||
await brain.import('customers.csv')
|
||||
// ✨ Auto-detects it's a file, parses CSV, creates entities!
|
||||
|
||||
// Or pass CSV content directly
|
||||
const csv = `name,age,city
|
||||
John,30,NYC
|
||||
Jane,25,SF`
|
||||
|
||||
await brain.import(csv, { format: 'csv' })
|
||||
// ✨ Smart CSV parsing handles quotes, escapes, everything!
|
||||
```
|
||||
|
||||
### 📝 Import YAML - File or String
|
||||
```javascript
|
||||
// From file? Auto-detected!
|
||||
await brain.import('config.yaml')
|
||||
// ✨ Knows it's a file, reads it, parses YAML!
|
||||
|
||||
// Or directly:
|
||||
const yaml = `
|
||||
project: AI Assistant
|
||||
team:
|
||||
- name: Alice
|
||||
role: Lead
|
||||
- name: Bob
|
||||
role: Dev
|
||||
`
|
||||
await brain.import(yaml, { format: 'yaml' })
|
||||
// ✨ Hierarchical data becomes a connected graph!
|
||||
```
|
||||
|
||||
### 🌐 Import from URLs - Auto-Detected!
|
||||
```javascript
|
||||
// Just pass the URL - it knows!
|
||||
await brain.import('https://api.example.com/data.json')
|
||||
// ✨ Auto-detects URL, fetches, parses, processes!
|
||||
|
||||
// Works with any URL
|
||||
await brain.import('https://data.gov/census.csv')
|
||||
// ✨ Fetches CSV from web, parses, imports!
|
||||
```
|
||||
|
||||
### 📖 Import Plain Text
|
||||
```javascript
|
||||
// Even unstructured text works
|
||||
const article = `Artificial Intelligence is transforming industries.
|
||||
Machine learning enables predictive analytics.
|
||||
Natural language processing powers chatbots.`
|
||||
|
||||
await brain.import(article, { format: 'text' })
|
||||
// ✨ Extracts concepts, creates semantic connections!
|
||||
```
|
||||
|
||||
## The Magic Behind the Scenes
|
||||
|
||||
When you import data, Brainy:
|
||||
|
||||
1. **Auto-detects format** - JSON, CSV, YAML, text, or by file extension
|
||||
2. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!)
|
||||
3. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!)
|
||||
4. **Creates embeddings** - Makes everything semantically searchable
|
||||
5. **Indexes metadata** - Enables lightning-fast filtering
|
||||
|
||||
## Intelligent Type Detection
|
||||
|
||||
Brainy automatically detects what TYPE of data you're importing:
|
||||
|
||||
```javascript
|
||||
// This becomes a Person entity
|
||||
{ name: 'John', email: 'john@example.com' }
|
||||
|
||||
// This becomes an Organization
|
||||
{ companyName: 'Acme', employees: 500 }
|
||||
|
||||
// This becomes a Document
|
||||
{ title: 'Report', content: '...', author: 'Jane' }
|
||||
|
||||
// This becomes a Location
|
||||
{ latitude: 37.7, longitude: -122.4, city: 'SF' }
|
||||
```
|
||||
|
||||
**31 noun types** and **40 verb types** cover EVERYTHING!
|
||||
|
||||
## Relationship Detection
|
||||
|
||||
Brainy finds connections in your data:
|
||||
|
||||
```javascript
|
||||
const data = [
|
||||
{ id: 'u1', name: 'Alice', managerId: 'u2' },
|
||||
{ id: 'u2', name: 'Bob', departmentId: 'd1' },
|
||||
{ id: 'd1', name: 'Engineering' }
|
||||
]
|
||||
|
||||
await brain.import(data)
|
||||
// ✨ Automatically creates:
|
||||
// - Alice "reportsTo" Bob
|
||||
// - Bob "memberOf" Engineering
|
||||
```
|
||||
|
||||
## Query Your Imported Data
|
||||
|
||||
Once imported, use Triple Intelligence to query:
|
||||
|
||||
```javascript
|
||||
// Vector search
|
||||
const similar = await brain.search('engineers')
|
||||
|
||||
// Natural language
|
||||
const results = await brain.find('people in engineering who joined this year')
|
||||
|
||||
// Graph traversal + filters
|
||||
const connected = await brain.find({
|
||||
like: 'Alice',
|
||||
connected: { depth: 2 },
|
||||
where: { department: 'Engineering' }
|
||||
})
|
||||
```
|
||||
|
||||
## Import Options (Optional!)
|
||||
|
||||
Everything works with zero config, but you can customize:
|
||||
|
||||
```javascript
|
||||
await brain.import(data, {
|
||||
format: 'auto', // 'json' | 'csv' | 'yaml' | 'text' | 'auto'
|
||||
batchSize: 50, // Process in batches
|
||||
relationships: true // Extract relationships (default: true)
|
||||
})
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
Import continues even if some items fail:
|
||||
|
||||
```javascript
|
||||
const results = await brain.import(problematicData)
|
||||
// Returns IDs of successful imports
|
||||
// Logs warnings for failures
|
||||
// Never crashes your app!
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
- **Parallel processing** - Fast imports
|
||||
- **Batch operations** - Memory efficient
|
||||
- **Lazy loading** - ImportManager loads only when needed
|
||||
- **Smart caching** - Type detection results are cached
|
||||
|
||||
## Use Cases
|
||||
|
||||
### 🏢 Business Data
|
||||
```javascript
|
||||
// Import ANY source - ONE method!
|
||||
await brain.import('customers.csv') // File
|
||||
await brain.import('https://api.co/orders') // URL
|
||||
await brain.import(productsArray) // Data
|
||||
|
||||
// Now query across all of it!
|
||||
await brain.find('customers who bought products in Q4')
|
||||
```
|
||||
|
||||
### 🔬 Research Data
|
||||
```javascript
|
||||
// Import research papers
|
||||
await brain.import(papers)
|
||||
|
||||
// Import citations
|
||||
await brain.import(citations)
|
||||
|
||||
// Find connections
|
||||
await brain.find('papers citing machine learning from 2024')
|
||||
```
|
||||
|
||||
### 📱 Application Data
|
||||
```javascript
|
||||
// Import users
|
||||
await brain.import(users)
|
||||
|
||||
// Import posts
|
||||
await brain.import(posts)
|
||||
|
||||
// Import comments
|
||||
await brain.import(comments)
|
||||
|
||||
// Query the social graph
|
||||
await brain.find('posts by users following Alice with >10 comments')
|
||||
```
|
||||
|
||||
## The Philosophy
|
||||
|
||||
**Zero Configuration**: Works perfectly out of the box
|
||||
**Maximum Intelligence**: AI understands your data's meaning
|
||||
**Universal Protocol**: 31 nouns × 40 verbs = ANY data model
|
||||
**Delightful DX**: Simple, clean, modern API
|
||||
|
||||
## The ONE Method Philosophy
|
||||
|
||||
```javascript
|
||||
// ONE method that understands EVERYTHING:
|
||||
await brain.import(data) // Objects, arrays, strings
|
||||
await brain.import('file.csv') // Files (auto-detected)
|
||||
await brain.import('http://..') // URLs (auto-fetched)
|
||||
|
||||
// It ALWAYS knows what to do! ✨
|
||||
```
|
||||
|
||||
**Why ONE method?**
|
||||
- 🎯 **Simpler** - No need to remember different methods
|
||||
- 🧠 **Smarter** - Auto-detects what you're importing
|
||||
- ✨ **Magical** - It just works, every time
|
||||
|
||||
That's the power of the Universal Knowledge Protocol™ - infinite intelligence, zero complexity!
|
||||
Loading…
Add table
Add a link
Reference in a new issue