brainy/docs/NEURAL_API_PATTERNS.md

🧠 Neural API Patterns: AI-Powered Intelligence

Learn the correct patterns for Brainy's Neural API. Avoid performance pitfalls and use AI features effectively.

🚨 Critical: Access Neural APIs Correctly

❌ WRONG - Outdated Access Patterns

// DON'T DO THIS - Outdated documentation patterns
import { Brainy } from '@soulcraft/brainy'  // ❌ Wrong import
const brain = new Brainy()                  // ❌ Old class name

// These may not work as expected:
const neural = brain.neural                     // ❌ May be undefined

✅ CORRECT - Modern Neural Access

// ✅ Use modern Brainy class
import { Brainy } from '@soulcraft/brainy'

const brain = new Brainy()
await brain.init()

// ✅ Neural API is available after initialization
const clusters = await brain.neural.clusters()
const similarity = await brain.neural.similar('item1', 'item2')

🔍 Similarity Analysis Patterns

❌ WRONG - Inefficient Similarity Checks

// DON'T DO THIS - N² comparisons
const items = await brain.find({ limit: 1000 })
const similarities = []

for (const item1 of items) {
  for (const item2 of items) {
    if (item1.id !== item2.id) {
      const sim = await brain.neural.similar(item1.id, item2.id)  // ❌ Millions of calls
      similarities.push({ from: item1.id, to: item2.id, score: sim })
    }
  }
}

✅ CORRECT - Efficient Similarity Patterns

// ✅ Pattern 1: Find neighbors (much more efficient)
const item = await brain.get('target-item-id')
const neighbors = await brain.neural.neighbors(item.id, {
  limit: 10,                     // Top 10 most similar
  threshold: 0.7,               // Minimum similarity
  includeScores: true           // Include similarity scores
})

console.log(`Found ${neighbors.length} similar items`)

// ✅ Pattern 2: Batch similarity for specific pairs
const itemPairs = [
  ['item1', 'item2'],
  ['item1', 'item3'],
  ['item2', 'item3']
]

const similarities = await Promise.all(
  itemPairs.map(async ([a, b]) => ({
    from: a,
    to: b,
    score: await brain.neural.similar(a, b)
  }))
)

// ✅ Pattern 3: Text-to-text similarity (no need for IDs)
const textSimilarity = await brain.neural.similar(
  "Machine learning is fascinating",
  "AI and deep learning are interesting",
  { detailed: true }  // Get explanation of similarity
)

console.log(`Similarity: ${textSimilarity.score}`)
console.log(`Explanation: ${textSimilarity.explanation}`)

// ✅ Pattern 4: Vector-level similarity for optimization
const vector1 = await brain.embed("First concept")
const vector2 = await brain.embed("Second concept")
const vectorSimilarity = await brain.neural.similar(vector1, vector2)

🎯 Clustering Patterns

❌ WRONG - Uncontrolled Clustering

// DON'T DO THIS - Clustering everything without limits
const everything = await brain.find({ limit: 100000 })  // ❌ Too much data
const clusters = await brain.neural.clusters()          // ❌ May crash or timeout

✅ CORRECT - Smart Clustering Patterns

// ✅ Pattern 1: Controlled clustering with limits
const recentItems = await brain.find({
  where: {
    createdAt: { $gte: Date.now() - 30 * 24 * 60 * 60 * 1000 } // Last 30 days
  },
  limit: 1000  // Reasonable limit
})

const clusters = await brain.neural.clusters(
  recentItems.map(item => item.id),
  {
    algorithm: 'kmeans',        // Reliable algorithm
    maxClusters: 10,           // Reasonable number
    threshold: 0.75,           // High similarity required
    iterations: 50             // Convergence limit
  }
)

// ✅ Pattern 2: Domain-specific clustering
const techDocs = await brain.find({
  where: { category: 'technology', type: 'document' },
  limit: 500
})

const techClusters = await brain.neural.clusterByDomain(
  'category',  // Group by this field
  {
    items: techDocs.map(doc => doc.id),
    minClusterSize: 3,         // Minimum items per cluster
    maxClusters: 8
  }
)

// ✅ Pattern 3: Temporal clustering for time-series data
const timebasedClusters = await brain.neural.clusterByTime(
  'createdAt',  // Time field
  'week',       // Time window (hour, day, week, month)
  {
    items: recentItems.map(item => item.id),
    overlap: 0.2,              // 20% overlap between windows
    minPerWindow: 5            // Minimum items per time window
  }
)

// ✅ Pattern 4: Streaming clustering for large datasets
async function clusterLargeDataset() {
  const clusterStream = brain.neural.clusterStream({
    batchSize: 100,            // Process 100 items at a time
    updateInterval: 1000,      // Update clusters every 1000 items
    maxMemory: 512 * 1024 * 1024  // 512MB memory limit
  })

  const allClusters = []
  for await (const batch of clusterStream) {
    console.log(`Processed ${batch.processed} items, found ${batch.clusters.length} clusters`)
    allClusters.push(...batch.clusters)
  }

  return allClusters
}

🔍 Neighbor Discovery Patterns

❌ WRONG - Manual Similarity Searches

// DON'T DO THIS - Reinventing neighbor search
async function findSimilarManually(targetId: string) {
  const allItems = await brain.find({ limit: 10000 })  // ❌ Load everything
  const similarities = []

  for (const item of allItems) {
    if (item.id !== targetId) {
      const score = await brain.neural.similar(targetId, item.id)  // ❌ Slow
      if (score > 0.7) {
        similarities.push({ id: item.id, score })
      }
    }
  }

  return similarities.sort((a, b) => b.score - a.score).slice(0, 10)  // ❌ Inefficient
}

✅ CORRECT - Optimized Neighbor Patterns

// ✅ Pattern 1: Basic neighbor search
const neighbors = await brain.neural.neighbors('target-item-id', {
  limit: 20,                   // Top 20 neighbors
  threshold: 0.6,             // Minimum similarity
  includeMetadata: true,      // Include item metadata
  includeDistances: true      // Include exact similarity scores
})

// ✅ Pattern 2: Filtered neighbor search
const filteredNeighbors = await brain.neural.neighbors('article-id', {
  limit: 10,
  filter: {
    type: 'document',          // Only find similar documents
    status: 'published',       // Only published content
    language: 'en'            // Only English content
  },
  excludeIds: ['self-id', 'duplicate-id']  // Exclude specific items
})

// ✅ Pattern 3: Multi-level neighbor discovery
async function discoverNeighborNetwork(rootId: string, maxDepth = 2) {
  const network = new Map()
  const visited = new Set()
  const queue = [{ id: rootId, depth: 0 }]

  while (queue.length > 0) {
    const { id, depth } = queue.shift()!

    if (visited.has(id) || depth >= maxDepth) continue
    visited.add(id)

    const neighbors = await brain.neural.neighbors(id, {
      limit: 5,
      threshold: 0.8
    })

    network.set(id, neighbors)

    // Add neighbors to queue for next depth level
    if (depth < maxDepth - 1) {
      neighbors.forEach(neighbor => {
        queue.push({ id: neighbor.id, depth: depth + 1 })
      })
    }
  }

  return network
}

// ✅ Pattern 4: Recommendation engine
async function getRecommendations(userId: string) {
  // Get user's liked items
  const userItems = await brain.find({
    connected: { to: userId, via: 'liked-by' }
  })

  // Find neighbors for each liked item
  const allNeighbors = await Promise.all(
    userItems.map(item =>
      brain.neural.neighbors(item.id, {
        limit: 10,
        threshold: 0.7,
        excludeConnected: { to: userId, via: 'liked-by' }  // Exclude already liked
      })
    )
  )

  // Aggregate and rank recommendations
  const recommendations = new Map()
  allNeighbors.flat().forEach(neighbor => {
    const current = recommendations.get(neighbor.id) || { score: 0, count: 0 }
    current.score += neighbor.score
    current.count += 1
    recommendations.set(neighbor.id, current)
  })

  // Return top recommendations by average score
  return Array.from(recommendations.entries())
    .map(([id, stats]) => ({
      id,
      avgScore: stats.score / stats.count,
      mentions: stats.count
    }))
    .sort((a, b) => b.avgScore - a.avgScore)
    .slice(0, 10)
}

🏗️ Hierarchy & Structure Patterns

❌ WRONG - Manual Hierarchy Building

// DON'T DO THIS - Building hierarchies manually
async function buildHierarchyManually(rootId: string) {
  const root = await brain.get(rootId)
  const allItems = await brain.find({ limit: 1000 })  // ❌ Load everything

  // Manual tree building with nested loops
  const hierarchy = { root, children: [] }
  // ... complex manual logic
}

✅ CORRECT - Semantic Hierarchy Patterns

// ✅ Pattern 1: Automatic semantic hierarchy
const hierarchy = await brain.neural.hierarchy('root-concept-id', {
  maxDepth: 4,                 // Maximum tree depth
  minSimilarity: 0.6,         // Minimum similarity for inclusion
  branchingFactor: 5,         // Maximum children per node
  algorithm: 'semantic'       // Use semantic clustering
})

// ✅ Pattern 2: Domain-specific hierarchy
const techHierarchy = await brain.neural.hierarchy('technology-id', {
  filter: { category: 'technology' },
  weights: {
    semantic: 0.7,            // 70% based on content similarity
    metadata: 0.3            // 30% based on metadata similarity
  },
  includeMetrics: true        // Include hierarchy quality metrics
})

// ✅ Pattern 3: Multi-root hierarchy for complex domains
async function buildMultiRootHierarchy(rootIds: string[]) {
  const hierarchies = await Promise.all(
    rootIds.map(rootId =>
      brain.neural.hierarchy(rootId, {
        maxDepth: 3,
        crossReference: true    // Allow cross-hierarchy connections
      })
    )
  )

  // Merge hierarchies and find connections
  const merged = {
    roots: hierarchies,
    connections: await findCrossHierarchyConnections(hierarchies)
  }

  return merged
}

async function findCrossHierarchyConnections(hierarchies: any[]) {
  const connections = []

  for (let i = 0; i < hierarchies.length; i++) {
    for (let j = i + 1; j < hierarchies.length; j++) {
      const leafNodes1 = extractLeafNodes(hierarchies[i])
      const leafNodes2 = extractLeafNodes(hierarchies[j])

      // Find connections between leaf nodes of different hierarchies
      for (const leaf1 of leafNodes1) {
        const neighbors = await brain.neural.neighbors(leaf1.id, {
          limit: 5,
          threshold: 0.8,
          filter: { id: { $in: leafNodes2.map(l => l.id) } }
        })

        connections.push(...neighbors.map(n => ({
          from: leaf1.id,
          to: n.id,
          hierarchyPair: [i, j],
          similarity: n.score
        })))
      }
    }
  }

  return connections
}

🚨 Outlier Detection Patterns

❌ WRONG - Manual Outlier Detection

// DON'T DO THIS - Manual statistical outlier detection
async function findOutliersManually() {
  const items = await brain.find({ limit: 1000 })
  const similarities = []

  // Calculate average similarity for each item (expensive)
  for (const item of items) {
    let totalSim = 0
    let count = 0
    for (const other of items) {
      if (item.id !== other.id) {
        totalSim += await brain.neural.similar(item.id, other.id)  // ❌ N² operations
        count++
      }
    }
    similarities.push({ id: item.id, avgSimilarity: totalSim / count })
  }

  // Manual outlier calculation
  const threshold = calculateManualThreshold(similarities)  // ❌ Complex statistics
  return similarities.filter(s => s.avgSimilarity < threshold)
}

✅ CORRECT - AI-Powered Outlier Detection

// ✅ Pattern 1: Automatic outlier detection
const outliers = await brain.neural.outliers({
  threshold: 0.3,              // Items with < 30% avg similarity to others
  method: 'isolation-forest',  // AI-based outlier detection
  contamination: 0.1,         // Expect ~10% outliers
  includeReasons: true        // Explain why each item is an outlier
})

console.log(`Found ${outliers.length} outliers`)
outliers.forEach(outlier => {
  console.log(`Outlier: ${outlier.id}, Score: ${outlier.score}`)
  console.log(`Reason: ${outlier.reason}`)
})

// ✅ Pattern 2: Domain-specific outlier detection
const techOutliers = await brain.neural.outliers({
  filter: { category: 'technology' },
  features: ['content', 'metadata.tags', 'metadata.complexity'],
  method: 'local-outlier-factor',
  neighbors: 20               // Consider 20 nearest neighbors
})

// ✅ Pattern 3: Temporal outlier detection
const recentOutliers = await brain.neural.outliers({
  timeWindow: '7days',        // Look at last 7 days
  baseline: '30days',         // Compare to 30-day baseline
  method: 'statistical',      // Use statistical methods
  autoThreshold: true         // Automatically determine threshold
})

// ✅ Pattern 4: Streaming outlier detection
async function detectOutliersInStream() {
  const outlierStream = brain.neural.outlierStream({
    batchSize: 50,
    updateInterval: 100,       // Check every 100 new items
    adaptiveThreshold: true    // Threshold adapts as data changes
  })

  for await (const batch of outlierStream) {
    console.log(`Batch ${batch.batchNumber}: ${batch.outliers.length} outliers detected`)

    // Process outliers immediately
    for (const outlier of batch.outliers) {
      await handleOutlier(outlier)
    }
  }
}

async function handleOutlier(outlier: any) {
  // Flag for manual review
  await brain.update(outlier.id, {
    metadata: {
      flagged: true,
      outlierScore: outlier.score,
      outlierReason: outlier.reason,
      flaggedAt: Date.now()
    }
  })
}

📊 Visualization Patterns

❌ WRONG - Manual Visualization Data Preparation

// DON'T DO THIS - Manual coordinate calculation
async function prepareVisualizationManually() {
  const items = await brain.find({ limit: 500 })
  const coordinates = []

  // Manual dimensionality reduction (complex math)
  for (const item of items) {
    const vector = await brain.embed(item.data)
    // Complex PCA/t-SNE calculations manually
    const x = complexMathFunction(vector)  // ❌ Error-prone
    const y = anotherComplexFunction(vector)
    coordinates.push({ id: item.id, x, y })
  }

  return coordinates
}

✅ CORRECT - AI-Powered Visualization

// ✅ Pattern 1: Automatic 2D visualization
const visualization = await brain.neural.visualize({
  dimensions: 2,               // 2D plot
  algorithm: 'umap',          // UMAP for better clustering preservation
  maxItems: 1000,             // Performance limit
  includeMetadata: true,      // Include item metadata in output
  colorBy: 'cluster'          // Color points by cluster membership
})

// Result format:
// {
//   points: [{ id, x, y, cluster, metadata }, ...],
//   clusters: [{ id, centroid: [x, y], members: [...] }, ...],
//   stats: { stress, kruskalStress, trustworthiness }
// }

// ✅ Pattern 2: 3D visualization for complex data
const viz3D = await brain.neural.visualize({
  dimensions: 3,
  algorithm: 'tsne',
  perplexity: 30,             // t-SNE parameter
  learningRate: 200,          // t-SNE learning rate
  iterations: 1000            // Number of optimization steps
})

// ✅ Pattern 3: Interactive visualization with filtering
const interactiveViz = await brain.neural.visualize({
  filter: {
    type: 'document',
    createdAt: { $gte: Date.now() - 7 * 24 * 60 * 60 * 1000 }
  },
  groupBy: 'category',        // Group points by metadata field
  showLabels: true,           // Include text labels
  labelField: 'title',        // Field to use for labels
  includeEdges: true,         // Show connections between similar items
  edgeThreshold: 0.8          // Only show high-similarity connections
})

// ✅ Pattern 4: Real-time visualization updates
class LiveVisualization {
  private visualization: any = null
  private updateInterval: NodeJS.Timeout | null = null

  async start() {
    // Initial visualization
    this.visualization = await brain.neural.visualize({
      dimensions: 2,
      algorithm: 'umap',
      maxItems: 500,
      includeMetadata: true
    })

    // Update every 30 seconds
    this.updateInterval = setInterval(async () => {
      await this.update()
    }, 30000)
  }

  async update() {
    // Get recent items
    const recentItems = await brain.find({
      where: {
        createdAt: { $gte: Date.now() - 30000 }  // Last 30 seconds
      },
      limit: 50
    })

    if (recentItems.length > 0) {
      // Incrementally update visualization
      const updates = await brain.neural.updateVisualization(
        this.visualization.id,
        {
          newItems: recentItems.map(item => item.id),
          algorithm: 'incremental'  // Faster incremental updates
        }
      )

      this.visualization = { ...this.visualization, ...updates }
      this.onUpdate(updates)
    }
  }

  onUpdate(updates: any) {
    // Emit updates to frontend
    console.log(`Visualization updated: ${updates.newPoints.length} new points`)
  }

  stop() {
    if (this.updateInterval) {
      clearInterval(this.updateInterval)
    }
  }
}

🚀 Performance Optimization Patterns

✅ High-Performance Neural Operations

// ✅ Pattern 1: Batch processing for similarity
async function batchSimilarityCalculation(itemPairs: Array<[string, string]>) {
  const batchSize = 100
  const results = []

  for (let i = 0; i < itemPairs.length; i += batchSize) {
    const batch = itemPairs.slice(i, i + batchSize)

    const batchResults = await Promise.all(
      batch.map(async ([a, b]) => ({
        from: a,
        to: b,
        similarity: await brain.neural.similar(a, b)
      }))
    )

    results.push(...batchResults)

    // Progress reporting
    console.log(`Processed ${Math.min(i + batchSize, itemPairs.length)}/${itemPairs.length} pairs`)
  }

  return results
}

// ✅ Pattern 2: Caching expensive operations
class NeuralCache {
  private clusterCache = new Map()
  private similarityCache = new Map()
  private readonly TTL = 5 * 60 * 1000  // 5 minutes

  async getClusters(options: any) {
    const key = JSON.stringify(options)
    const cached = this.clusterCache.get(key)

    if (cached && Date.now() - cached.timestamp < this.TTL) {
      return cached.data
    }

    const clusters = await brain.neural.clusters(undefined, options)
    this.clusterCache.set(key, {
      data: clusters,
      timestamp: Date.now()
    })

    return clusters
  }

  async getSimilarity(id1: string, id2: string) {
    // Create consistent cache key regardless of order
    const key = [id1, id2].sort().join('-')
    const cached = this.similarityCache.get(key)

    if (cached && Date.now() - cached.timestamp < this.TTL) {
      return cached.data
    }

    const similarity = await brain.neural.similar(id1, id2)
    this.similarityCache.set(key, {
      data: similarity,
      timestamp: Date.now()
    })

    return similarity
  }
}

// ✅ Pattern 3: Memory-efficient streaming
async function processLargeDatasetEfficiently() {
  const stream = brain.neural.clusterStream({
    batchSize: 50,              // Small batches for memory efficiency
    maxMemoryMB: 256,          // Memory limit
    diskCache: true,           // Use disk for temporary storage
    compression: true          // Compress cached data
  })

  const results = []
  let totalProcessed = 0

  for await (const batch of stream) {
    // Process batch immediately, don't accumulate in memory
    const processedBatch = await processBatch(batch)

    // Save to disk or send to another service
    await saveBatchToDisk(processedBatch)

    totalProcessed += batch.items.length
    console.log(`Processed ${totalProcessed} items`)

    // Clear memory
    batch.items = null
  }

  return { totalProcessed }
}

// ✅ Pattern 4: Parallel processing with worker threads
async function parallelNeuralProcessing(items: string[]) {
  const numWorkers = require('os').cpus().length
  const batchSize = Math.ceil(items.length / numWorkers)

  const workers = []
  for (let i = 0; i < numWorkers; i++) {
    const batch = items.slice(i * batchSize, (i + 1) * batchSize)
    if (batch.length > 0) {
      workers.push(processWorkerBatch(batch))
    }
  }

  const results = await Promise.all(workers)
  return results.flat()
}

async function processWorkerBatch(batch: string[]) {
  // This would run in a worker thread in real implementation
  return Promise.all(
    batch.map(async itemId => {
      const neighbors = await brain.neural.neighbors(itemId, { limit: 5 })
      return { itemId, neighbors }
    })
  )
}

🎯 Summary: Neural API Best Practices

❌ Avoid These Patterns	✅ Use These Instead
Manual similarity loops	brain.neural.neighbors()
Uncontrolled clustering	Limit items and set maxClusters
Manual outlier detection	brain.neural.outliers()
Manual visualization prep	brain.neural.visualize()
Loading entire datasets	Streaming and batch processing
No caching	Cache expensive operations
Blocking operations	Parallel and async patterns

---

🎉 Following these patterns gives you:

• 🚀 Optimized performance with intelligent algorithms
• 🧠 AI-powered insights instead of manual statistics
• 📊 Rich visualizations for data exploration
• 🎯 Accurate clustering with semantic understanding
• 🚨 Smart outlier detection for quality control
• ⚡ Scalable processing for large datasets

Next: Augmentation Patterns → | Core API Patterns →