feat: add stddev/variance aggregation ops with Welford's online algorithm

- New aggregation ops: stddev and variance (incremental, O(1) per update)
- Welford's online algorithm for numerically stable running variance
- MetricState extended with m2 field for variance tracking
- AggregationProvider interface: defineAggregate, removeAggregate, restoreState, serializeState
- Export AggregateGroupState and MetricState types
- Plugin docs: aggregation provider, analytics providers (HyperLogLog, t-digest, etc.)
- Aggregation architecture and usage guide docs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-02-17 17:04:11 -08:00
parent 4602960e86
commit b7c6752388
9 changed files with 1435 additions and 792 deletions

View file

@ -192,6 +192,23 @@ context.registerProvider('graphIndex', (storage) => {
})
```
#### `aggregation`
**Type:** `(storage: StorageAdapter) => AggregationProvider-compatible`
Factory function that creates an aggregation engine for write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows. The returned object must implement the `AggregationProvider` interface.
```typescript
context.registerProvider('aggregation', (storage) => {
return new MyNativeAggregationEngine(storage)
})
```
When provided by a native plugin like `@soulcraft/cortex`, this enables:
- Compiled source filters (vs per-entity JS object traversal)
- Precise MIN/MAX via sorted data structures (vs lazy recompute)
- Parallel aggregate rebuild across CPU cores
- SIMD-accelerated timestamp bucketing
### Utility Providers
#### `cache`
@ -220,6 +237,29 @@ Replacement for the roaring bitmap implementation. Used internally by the metada
Native msgpack encode/decode for SSTable serialization.
### Analytics Providers (Native-Only)
These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when a native plugin like `@soulcraft/cortex` is installed.
Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it.
#### `analytics:hyperloglog`
Approximate distinct counts. Count unique values (e.g., unique merchants) across millions of records using ~16KB of memory with ~1% error. Each update is O(1).
#### `analytics:tdigest`
Streaming percentiles. Compute P50/P90/P95/P99 from streaming data without storing all values. Uses ~4KB per digest with ~1% accuracy at the tails.
#### `analytics:countmin`
Frequency estimation. Find the most common values (e.g., top-K merchants) using ~40KB with 0.1% error. O(1) per update.
#### `analytics:anomaly`
Real-time anomaly detection. Flag statistically unusual values at write-time using exponentially weighted moving averages. 64 bytes per group, sub-microsecond decisions.
#### `aggregation:mmap`
Persistent aggregate storage via memory-mapped files. Aggregate state survives process crashes without explicit flush. Zero serialization overhead.
---
## Storage Adapter Plugins
Plugins can register custom storage backends that users reference by name.