dpsifr 7c493632ad **docs(statistics): add statistics-summary.md explaining tracking, storage, and usage**

- **Documentation**: Added a new `statistics-summary.md` file detailing how statistics are tracked, updated, and stored within the system. Includes examples of API usage, implementation details, and consumer use cases.
- **Purpose**: Provide clear guidance for developers and stakeholders on the statistics system's functionality, including real-time updates, service-specific breakdowns, and persistent storage mechanisms. Aids in understanding and leveraging the system effectively.

2025-07-25 09:51:15 -07:00

3.2 KiB

Raw Blame History

Brainy Statistics System Summary

How Statistics Are Updated and Stored

What Statistics Are Tracked

Brainy tracks the following statistics:

Noun Count: Number of vector data points, tracked by service
Verb Count: Number of relationships between nouns, tracked by service
Metadata Count: Number of metadata entries, tracked by service
HNSW Index Size: Total size of the HNSW index used for vector search

How Statistics Are Updated

Statistics are updated automatically as data is added or removed:

When a noun is added using add(), the noun count for the specified service is incremented
When a verb is added using addVerb() or relate(), the verb count is incremented
When metadata is added, the metadata count is incremented
The HNSW index size is updated whenever nouns are added or removed

Each operation includes a service parameter that identifies which service is adding the data.

Storage Implementation

Statistics are stored persistently with several optimizations:

Local Caching: Statistics are cached in memory to reduce storage API calls
Batched Updates: Updates are batched and flushed periodically (5-30 seconds)
Time-based Partitioning: Statistics are stored in daily files (e.g., statistics_20250724.json)
Adaptive Flush Timing: The system adjusts the flush frequency based on recent activity

How Statistics Can Be Used by Consumers

Direct API Access

Consumers can access statistics through the BrainyData API:

// Using the instance method
const stats = await db.getStatistics()

// Filter by service
const serviceStats = await db.getStatistics({
    service: "my-service"
})

The returned statistics object includes counts and service breakdown:

{
    "nounCount": 150,
    "verbCount": 75,
    "metadataCount": 150,
    "hnswIndexSize": 150,
    "serviceBreakdown": {
        "default": {
            "nounCount": 100,
            "verbCount": 50,
            "metadataCount": 100
        },
        "my-service": {
            "nounCount": 50,
            "verbCount": 25,
            "metadataCount": 50
        }
    }
}

Web Service Access

The web service provides a /api/status endpoint, but it only returns basic information about storage usage and capacity, not the detailed statistics tracked by the system. To access the full statistics through the web service, consumers would need to implement their own endpoint that calls getStatistics().

Use Cases for Consumers

Monitoring Database Growth: Track how the database grows over time
Analyzing Service Usage: Identify which services are adding the most data
Cleaning Up Service Data: Identify services with minimal data
Performance Monitoring: Track the size of the HNSW index

Consistency of Statistics Tracking

The statistics system consistently tracks:

Total counts: Overall counts of nouns, verbs, metadata, and index size
Per-service breakdown: All counts are tracked by the service that inserted the data
Real-time updates: Statistics are updated in real-time as data is added or removed
Persistent storage: Statistics are stored persistently and survive database restarts

3.2 KiB Raw Blame History