- AggregationIndex: defineAggregate before init no longer forces a backfill.
init reconciles instead of clobbering: the app definition wins, persisted
state is adopted on hash match, a write landing pre-adoption forces an exact
rescan, and a successful state load clears the backfill flag. New ready()
settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
store walk and every pending aggregate fills from that same walk; the old
behavior let each concurrent query wipe the others' partial state and start
its own full walk, so a store under steady aggregate traffic never converged.
queryAggregate also waits for persisted definitions before deciding an
aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
at 1000 - below the documented 10000 auto floor, reported under a stale
basis label) is removed; the cap comes from its construction-time basis or
explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
semantics, RELEASES.md 8.5.1 entry.
The 3.0 native engine ships as @soulcraft/cor (brainy 8.x <-> cor 3.x are a
version-matched pair); the old @soulcraft/cortex package stays on the 2.x/7.x
line. Public docs and .d.ts-visible comments now name the correct package.
Historical 2.x contract references (e.g. the 2.3.1 read-side fallback) keep
the old name deliberately.
- id-normalization (#18): `brain.newId()` (UUID v7) + v7 default ids; non-UUID string ids are
transparently normalized to a stable UUID v5 on creation AND every lookup — get/update/remove,
relate(from,to), related, find({connected}), getMany, removeMany, the transact op-handler, and
db.with() overlays — with the caller's original key preserved under `_originalId`. The engine only
ever sees a UUID; real UUIDs pass through untouched. universal/uuid.ts gains v5/v7/isUUID +
BRAINY_ID_NAMESPACE; new utils/idNormalization.ts (coerceNewEntityId / resolveEntityId).
- aggregation min/max delete-safety: `queryAggregate` no longer leaves a stale min/max after
deleting the current extreme — min/max now track a value multiset and recompute in-memory (no
entity scan; that scan was the prior delete-then-hang a consumer reported). Other metrics were
already exact across deletes. Regression test proves resolve-after-delete + correct new min/max.
- release.sh RC-safe: explicit version arg, prerelease detection (npm `--tag rc` + GitHub
`--prerelease`), full `test:ci` gate (unit + integration), current-branch push, npm-access
verification after publish.
- native-engine references point at `@soulcraft/cor` (8.0's partner) in user-facing strings/docs.
Unit 1461/0 · integration 599/0 · tsc clean.
distinctCount previously routed through getNumericField, so it silently returned 0 for
string/categorical fields — its primary use case (distinct categories / users / tags).
It now tracks the raw value (any type, keyed by string form) on both the add and remove
contribution paths; numeric ops (sum/avg/min/max/stddev/variance/percentile) are unchanged.
Also adds regression coverage confirming two long-standing query behaviors hold on the 8.0
engine: find({ where: { field: { missing: true } } }) matches a never-registered field, and
find({ type: [...], orderBy, limit }) returns the full set on the first call after
mutate+query+get. (percentile/median were already correct.)
Tests: tests/unit/brainy/find-agg-edge-cases.test.ts + existing aggregation suite green.
Replace localeCompare (default-locale, non-deterministic across environments —
unsafe for a persisted sorted index) with UTF-8 byte / code-point order via a
shared compareCodePoints() helper. String ordering is now deterministic and
byte-identical to the native column store / aggregation sort, so results are the
same with or without the native accelerator. Covers the aggregate orderBy sort
and all 5 column-store comparison sites (tail buffer, merge sort, binary search).
Adds compareCodePoints unit tests.
Add 'percentile' (with a 'p' fraction in [0,1]) and 'distinctCount' to the
aggregation engine. Both are exact, computed from a per-metric value multiset
(MetricState.valueCounts) maintained incrementally and delete-safe; percentile
uses numpy-linear interpolation. The multiset is JSON-serializable so results
survive persistence. 35 aggregation unit tests pass.
- groupBy now supports { field, unnest: true }: an entity contributes once per
distinct element of an array field (tag frequency / faceted counts). Duplicate
elements on one entity count once; an empty/missing array joins no group. The
incremental add/update/delete paths fan out across the unnested groups.
- extractEntities/extractConcepts batch-embed the unique candidate spans in one
embedBatch call instead of one embed() per candidate (N sequential model calls);
falls back to per-candidate embedding if the batch fails. No behavior change.
New report APIs:
- brain.queryAggregate(name, params) returns the clean AggregateResult[] shape
({ groupKey, metrics, count }) directly, instead of the search-Result wrapper that
find({ aggregate }) returns.
- HAVING: find({ aggregate, having }) and queryAggregate filter groups by computed
metric values (e.g. revenue > 1000), complementing where (which filters group keys).
Evaluated per group: O(groups), independent of entity count.
Fixes (all reproducible on Node; surfaced under Cortex+Bun by Memory):
- Aggregate backfill-on-define: defining an aggregate over a store that already holds
matching entities returned []. It now backfills from existing entities on first query
(storage-agnostic via getNouns), so it works under durable backends that reopen
pre-populated. groupBy:['noun'] resolves to the entity type; find({ aggregate }) rows
expose groupKey/metrics/count at the top level.
- Multi-hop find({ connected: { depth, via } }) honors depth and via at every hop
(was 1-hop only, with verb filtering applied to hop 1 only); BFS bounded by limit.
- extractEntities no longer bleeds a neighbour's type indicator across candidates; each
candidate is typed by its own span. Also fixes the "Dr." title pattern.
Adds real-embedding regression tests; full unit suite green.
Preventive cleanup of sites that previously used dual-lookup patterns
(metadata[field] ?? entity[field]) or hardcoded timestamp if-chains
to read fields off HNSWNounWithMetadata. These worked today but were
fragile — the same failure mode that caused the orderBy sort bug
would have recurred the next time someone reordered or dropped a
lookup.
- AggregationIndex.computeGroupKey + getNumericField now route all
dimension and metric field lookups through resolveEntityField.
- improvedNeuralAPI._getItemsByTimeWindow + _clusterItemsByField
use resolveEntityField as the primary path, with item.data as a
legacy producer fallback. The type/nounType legacy branch is
preserved as-is since it handles storage format compatibility
rather than shape contract drift.
No behavior change for correct inputs. All aggregation and orderby
regression tests pass unchanged.
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.
Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
