brainy/docs/guides/per-service-statistics.md

Per-Service Statistics and Tracking Guide

Brainy provides comprehensive per-service statistics tracking, allowing you to monitor and analyze data from different services sharing the same storage backend.

Table of Contents

• Overview
• Configuration
• Service Tracking
• Statistics Methods
• Service Filtering
• Use Cases
• API Reference

Overview

When multiple services (e.g., bluesky-package, github-package, scout-search) share the same Brainy storage, per-service statistics help you:

• Track individual service performance - Monitor how many nouns/verbs each service has stored
• Debug service-specific issues - Isolate problems to specific services
• Monitor service health - Track last activity timestamps and error counts
• Audit data sources - Know which service created which data
• Analyze usage patterns - Understand how different services use the database

Configuration

Setting a Default Service

When creating a Brainy instance, specify a default service name:

const brainy = new BrainyData({
  defaultService: 'github-package',
  storage: {
    // Storage configuration
  }
})

Specifying Service Per Operation

You can override the default service for individual operations:

// Add data with explicit service
await brainy.add(
  { content: 'repository data' },
  { noun: 'Repository' },
  { service: 'github-service' }
)

// Create relationship with service tracking
await brainy.relate(
  sourceId,
  targetId,
  { verb: 'stars' },
  { service: 'github-service' }
)

Service Tracking

Automatic Tracking

Brainy automatically tracks for each service:

• Total nouns created
• Total verbs created
• Total metadata entries
• First activity timestamp
• Last activity timestamp
• Operation counts (adds, updates, deletes)
• Error counts

Service Activity States

Services can have three states:

• active - Had activity within the last hour
• inactive - No recent activity
• read-only - Only performs read operations (no writes)

Statistics Methods

List All Services

Get a list of all services that have written data:

const services = await brainy.listServices()

// Returns:
[
  {
    name: 'bluesky-package',
    totalNouns: 12450,
    totalVerbs: 34820,
    totalMetadata: 12450,
    firstActivity: '2024-01-01T00:00:00Z',
    lastActivity: '2024-01-06T16:30:00Z',
    status: 'active',
    operations: {
      adds: 12450,
      updates: 230,
      deletes: 0
    }
  },
  {
    name: 'github-package',
    totalNouns: 45230,
    totalVerbs: 89420,
    totalMetadata: 45230,
    lastActivity: '2024-01-06T16:25:00Z',
    status: 'active'
  }
]

Get Service-Specific Statistics

Get detailed statistics for a single service:

const stats = await brainy.getServiceStatistics('github-package')

// Returns:
{
  name: 'github-package',
  totalNouns: 45230,
  totalVerbs: 89420,
  totalMetadata: 45230,
  firstActivity: '2024-01-01T00:00:00Z',
  lastActivity: '2024-01-06T16:25:00Z',
  status: 'active',
  operations: {
    adds: 45230,
    updates: 1250,
    deletes: 50
  },
  errorCount: 5
}

Filter Global Statistics by Service

Filter the global statistics to specific services:

// Single service
const stats = await brainy.getStatistics({ 
  service: 'bluesky-package' 
})

// Multiple services
const stats = await brainy.getStatistics({ 
  service: ['bluesky-package', 'github-package'] 
})

// Returns statistics with serviceBreakdown:
{
  nounCount: 57680,
  verbCount: 124240,
  metadataCount: 57680,
  serviceBreakdown: {
    'bluesky-package': {
      nounCount: 12450,
      verbCount: 34820,
      metadataCount: 12450
    },
    'github-package': {
      nounCount: 45230,
      verbCount: 89420,
      metadataCount: 45230
    }
  }
}

Service Filtering

Filter Search Results

Filter search results to only include data from specific services:

const results = await brainy.search('javascript', 10, {
  service: 'github-package'
})

Filter Noun Queries

Get nouns only from specific services:

const nouns = await brainy.getNouns({
  filter: {
    service: 'bluesky-package',
    nounType: 'Post'
  }
})

Filter Verb Queries

Get verbs only from specific services:

const verbs = await brainy.getVerbs({
  filter: {
    service: 'social-service',
    verbType: 'follows'
  }
})

Use Cases

Multi-Tenant Applications

When multiple applications share the same Brainy storage:

// Application A
const brainyA = new BrainyData({
  defaultService: 'app-a',
  storage: sharedStorage
})

// Application B
const brainyB = new BrainyData({
  defaultService: 'app-b',
  storage: sharedStorage
})

// Monitor usage per application
const services = await brainyA.listServices()
for (const service of services) {
  console.log(`${service.name}: ${service.totalNouns} nouns`)
}

Service Health Monitoring

Monitor the health and activity of different services:

async function monitorServiceHealth() {
  const services = await brainy.listServices()
  
  for (const service of services) {
    if (service.status === 'inactive') {
      console.warn(`Service ${service.name} is inactive`)
    }
    
    if (service.errorCount && service.errorCount > 100) {
      console.error(`Service ${service.name} has high error count: ${service.errorCount}`)
    }
    
    // Check for stale services
    if (service.lastActivity) {
      const lastActivity = new Date(service.lastActivity)
      const daysSinceActivity = (Date.now() - lastActivity.getTime()) / (1000 * 60 * 60 * 24)
      
      if (daysSinceActivity > 7) {
        console.warn(`Service ${service.name} hasn't been active for ${daysSinceActivity} days`)
      }
    }
  }
}

Data Source Auditing

Track which service created specific data:

// Add data with service tracking
const docId = await brainy.add(
  { 
    content: 'Important document',
    timestamp: new Date().toISOString()
  },
  { 
    noun: 'Document',
    createdBy: { augmentation: 'audit-service' }
  },
  { service: 'audit-service' }
)

// Later, verify the source
const stats = await brainy.getServiceStatistics('audit-service')
console.log(`Audit service created ${stats.totalNouns} documents`)

Performance Analysis

Analyze ingestion rates and patterns per service:

async function analyzePerformance() {
  const services = await brainy.listServices()
  
  for (const service of services) {
    if (!service.firstActivity || !service.lastActivity) continue
    
    const start = new Date(service.firstActivity)
    const end = new Date(service.lastActivity)
    const durationHours = (end.getTime() - start.getTime()) / (1000 * 60 * 60)
    
    const nounsPerHour = service.totalNouns / durationHours
    const verbsPerHour = service.totalVerbs / durationHours
    
    console.log(`${service.name} performance:`)
    console.log(`  Nouns/hour: ${nounsPerHour.toFixed(2)}`)
    console.log(`  Verbs/hour: ${verbsPerHour.toFixed(2)}`)
  }
}

API Reference

Configuration Options

interface BrainyDataConfig {
  /**
   * Default service name for all operations
   */
  defaultService?: string
  
  // Other configuration options...
}

Operation Options

interface OperationOptions {
  /**
   * Override the default service for this operation
   */
  service?: string
  
  // Other operation options...
}

Service Statistics Type

interface ServiceStatistics {
  /**
   * Service name
   */
  name: string
  
  /**
   * Total number of nouns created by this service
   */
  totalNouns: number
  
  /**
   * Total number of verbs created by this service
   */
  totalVerbs: number
  
  /**
   * Total number of metadata entries
   */
  totalMetadata: number
  
  /**
   * First activity timestamp
   */
  firstActivity?: string
  
  /**
   * Last activity timestamp
   */
  lastActivity?: string
  
  /**
   * Error count for this service
   */
  errorCount?: number
  
  /**
   * Operation breakdown
   */
  operations?: {
    adds: number
    updates: number
    deletes: number
  }
  
  /**
   * Service status
   */
  status?: 'active' | 'inactive' | 'read-only'
}

Best Practices

1. Always specify a defaultService when creating Brainy instances to ensure proper tracking
2. Use consistent service names across your application
3. Monitor service health regularly using listServices()
4. Use service filtering when debugging service-specific issues
5. Track error counts to identify problematic services
6. Archive inactive services periodically to maintain performance

Migration Guide

If you have existing data without service tracking:

// Set a default service for untracked data
const stats = await brainy.getStatistics()

// All existing data will be under the 'default' service
console.log(stats.serviceBreakdown['default'])

// Going forward, specify explicit services
const brainy = new BrainyData({
  defaultService: 'my-app-v2',
  // ...
})

Limitations

• Service tracking is done at the statistics level, not embedded in each noun/verb
• Service filtering in search operations depends on metadata structure
• Historical service activity may not be available for data created before enabling tracking
• Service names should be kept reasonably short for performance

Examples

Complete Example: Multi-Service Dashboard

import { BrainyData } from '@soulcraft/brainy'

async function createServiceDashboard() {
  const brainy = new BrainyData({
    defaultService: 'dashboard',
    storage: {
      // Your storage configuration
    }
  })
  
  await brainy.init()
  
  // Get all services
  const services = await brainy.listServices()
  
  // Sort by most active
  services.sort((a, b) => {
    const aTime = a.lastActivity ? new Date(a.lastActivity).getTime() : 0
    const bTime = b.lastActivity ? new Date(b.lastActivity).getTime() : 0
    return bTime - aTime
  })
  
  // Display dashboard
  console.log('=== Service Dashboard ===')
  console.log(`Total Services: ${services.length}`)
  console.log('')
  
  for (const service of services) {
    console.log(`Service: ${service.name}`)
    console.log(`  Status: ${service.status}`)
    console.log(`  Nouns: ${service.totalNouns}`)
    console.log(`  Verbs: ${service.totalVerbs}`)
    
    if (service.lastActivity) {
      const lastActivity = new Date(service.lastActivity)
      console.log(`  Last Active: ${lastActivity.toLocaleString()}`)
    }
    
    if (service.operations) {
      console.log(`  Operations:`)
      console.log(`    Adds: ${service.operations.adds}`)
      console.log(`    Updates: ${service.operations.updates}`)
      console.log(`    Deletes: ${service.operations.deletes}`)
    }
    
    console.log('')
  }
  
  // Get aggregate statistics
  const totalStats = await brainy.getStatistics()
  console.log('=== Total Statistics ===')
  console.log(`Total Nouns: ${totalStats.nounCount}`)
  console.log(`Total Verbs: ${totalStats.verbCount}`)
  console.log(`Total Metadata: ${totalStats.metadataCount}`)
}

createServiceDashboard().catch(console.error)

This comprehensive per-service statistics feature enables better observability, debugging, and management of multi-service Brainy deployments.