brainy/TESTING-STRATEGY.md

🧠 Brainy Testing Strategy

Overview

Brainy uses ONNX Runtime with transformer models, requiring 4-8GB memory for full functionality. This document explains our testing strategy based on 2024-2025 best practices.

Memory Requirements

Component	Memory Usage	Notes
ONNX Model	4-8GB	all-MiniLM-L6-v2 transformer
Node.js Heap	2-4GB	JavaScript runtime
Test Framework	1-2GB	Vitest overhead
Total	8-16GB	Recommended for full suite

Test Commands

Quick Reference

npm test                  # Standard test run
npm run test:memory       # With 16GB heap allocation
npm run test:core         # Core functionality only
npm run test:ci           # CI optimized
npm run test:shard       # Supports VITEST_SHARD env

Memory-Intensive Tests

# Allocate 16GB for transformer models
NODE_OPTIONS='--max-old-space-size=16384' npm test

# Or use our helper script
npm run test:memory

Test Sharding (CI/CD)

# Split tests across 4 machines/processes
VITEST_SHARD=1/4 npm run test:shard  # 1st quarter
VITEST_SHARD=2/4 npm run test:shard  # 2nd quarter
VITEST_SHARD=3/4 npm run test:shard  # 3rd quarter
VITEST_SHARD=4/4 npm run test:shard  # 4th quarter

Vitest Configuration

Our vitest.config.ts implements industry best practices:

Memory Optimization

• Pool: forks for better memory isolation
• Max Forks: 1 (sequential execution)
• Isolation: Enabled (prevents memory leaks)

Timeouts

• Test: 120 seconds (ONNX loading)
• Hooks: 60 seconds (model initialization)
• Teardown: 10 seconds

Performance

• Parallelism: Disabled (prevents OOM)
• Retry: Once in CI (handles flaky tests)
• Reporters: Dot for CI, verbose for local

Test Organization

Current Structure (All Tests)

tests/
├── core.test.ts              # Core functionality
├── triple-intelligence.test.ts  # AI features
├── metadata-filter.test.ts      # Brain Patterns
├── neural-api.test.ts          # Neural operations
└── ... (45+ test files)

Future Structure (Recommended)

tests/
├── unit/                    # No models, fast
│   ├── utils/
│   ├── storage/
│   └── metadata/
├── integration/             # With models, slow
│   ├── search/
│   ├── embeddings/
│   └── triple/
└── e2e/                     # Full system tests

Common Issues & Solutions

Out of Memory (OOM)

Error: FATAL ERROR: Ineffective mark-compacts near heap limit

Solutions:

1. Increase heap: NODE_OPTIONS='--max-old-space-size=16384'
2. Run fewer tests: npm test tests/core.test.ts
3. Use sharding: VITEST_SHARD=1/2

Test Timeouts

Error: Test timed out after 30000ms

Solutions:

1. Already extended to 120s in config
2. Skip model tests: npm test -- --exclude "**/neural*"
3. Mock embeddings for unit tests

ClearAll Safety

Error: clearAll requires force: true option

Solution: Always use brain.clearAll({ force: true }) ✅ Already fixed in all test files

CI/CD Recommendations

GitHub Actions

name: Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1/4, 2/4, 3/4, 4/4]
    
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 18
      
      - run: npm ci
      - run: npm run build
      
      # Run sharded tests with memory
      - run: VITEST_SHARD=${{ matrix.shard }} npm run test:ci
        env:
          NODE_OPTIONS: --max-old-space-size=16384

Docker

FROM node:18
WORKDIR /app

# Increase memory limits
ENV NODE_OPTIONS="--max-old-space-size=16384"

COPY . .
RUN npm ci
RUN npm run build

# Run tests
CMD ["npm", "run", "test:memory"]

Performance Benchmarks

With proper configuration:

• Model Load: 30-60 seconds (first time)
• Embedding: 10-50ms per text
• Test Suite: 5-10 minutes (sequential)
• Memory Usage: 4-8GB peak

Best Practices

1. Always build before testing

   npm run build && npm test
   

2. Monitor memory during tests

   watch -n 1 "ps aux | grep node | head -5"
   

3. Use appropriate test command

   • Development: npm test
   • CI: npm run test:ci
   • Debugging: npm run test:memory -- --reporter=verbose

4. Mock for unit tests

   // Mock embeddings for non-AI tests
   const mockEmbed = () => new Array(384).fill(0.1)
   

Industry Standards

We follow these 2024-2025 best practices:

1. Vitest Forks Pool: Better memory isolation than threads
2. Test Sharding: Distribute across multiple processes
3. Sequential Execution: Prevent memory competition
4. Extended Timeouts: Account for model loading
5. Memory Monitoring: Track usage during tests

Conclusion

Brainy's testing requires significant memory due to transformer models. This is not a bug - it's the cost of running state-of-the-art AI locally. Our configuration follows industry best practices to manage this requirement effectively.

For projects that cannot allocate 8-16GB for testing:

• Use mock embeddings for unit tests
• Run integration tests separately in CI
• Consider cloud-based testing environments
• Use test sharding to distribute load