Commit graph

18 commits

Author SHA1 Message Date
0951fa1da0 feat(8.0): thread commit generation through the graph-write provider contract
Graph time-travel needs an edge's existence recorded per generation so
db.asOf(g) hops resolve historically correct endpoints. The metadata layer
already threads brainy's commit generation per write; the graph write path did
not, leaving a versioned verb-endpoint store unable to answer "which edges
existed at generation g".

- GraphIndexProvider.addVerb/removeVerb gain a `generation: bigint` parameter
  (the same watermark the storage layer stamps onto the record). A provider with
  a per-generation edge chain stamps the edge at that generation; the JS baseline
  has no such chain and accepts-and-ignores it — graph time-travel is a
  native-provider capability, and the open-core path serves edges as-of-now (the
  one documented graph time-travel limitation).
- The two graph transaction operations resolve the generation via a thunk at
  EXECUTE time: the generation store assigns the batch generation only once the
  commit begins executing, after the operations are planned. The same generation
  is reused for an operation's rollback half.
- All graph-write call sites pass the in-flight generation.

Adds a spy-provider test proving the threading, execute-time resolution, and
forward/rollback generation reuse. The JS index ignores the value, so behaviour
is unchanged: unit 1402/1402, db-mvcc 25/25, bigint-contract relate/unrelate 10/10.
2026-06-16 09:42:35 -07:00
1f7e365a4e chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase 2026-06-11 14:51:00 -07:00
431cd64406 feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist)
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.

Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
  transact() commit and once per single-operation write (storage hook), so
  brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
  batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
  (the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
  and forces an index rebuild; refcounted pins gate compactHistory(), which
  records a horizon (asOf below it throws GenerationCompactedError).

Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
  overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
  generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
  (GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
  {confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
  generation; index-accelerated queries at historical generations throw
  NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
  plugin.ts: feature-detected, balanced pin/release in lockstep with Db
  lifecycle, post-commit applier + replay-gap model documented.

Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.

Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
2026-06-10 14:14:07 -07:00
49e49483d1 fix(8.0): createIndex resolves the canonical 'vector' provider key — drop diskann/hnsw key lookups + legacy migration APIs
createIndex() still resolved the retired 'diskann' and 'hnsw' provider
keys and never looked up 'vector', so an 8.0-era plugin registering its
vector engine under the canonical 'vector' key silently never engaged
and Brainy always fell back to the built-in JS HNSW index. createIndex()
now resolves only getProvider('vector') (same factory call shape as the
old hnsw path) and falls back to setupIndex().

The brainy-side mode/migration machinery is obsolete in the 8.0 provider
world — the registered engine adapts internally (in-memory / hybrid /
on-disk selection is the provider's job, not Brainy's):

- delete migrateToDiskAnn() / migrateToHnsw() and the ADR-002
  index-engine migration block
- delete diskAnnAutoEngageConditionsMet() / instantiateDiskAnn()
- narrow HNSWConfig.type to 'vector' and drop the diskann tuning block
  (coreTypes.ts)
- well-known provider key lists + diagnostics now report 'vector'
  instead of the never-registered 'hnsw' key
- plugin.ts docs: 'vector' is the only vector-index key consulted; the
  pre-8.0 'hnsw'/'diskann' keys are retired

The schema-migration machinery (migrate(), checkMigrations(),
MigrationRunner, autoMigrate) serves data migrations and is untouched.
2026-06-10 11:29:05 -07:00
2427bb7960 feat(8.0): u64 BigInt graph provider contract — punch list a-d,g,h
GraphIndexProvider now speaks BigInt at the boundary (D.2 mirror):
- getNeighbors/getVerbIdsBySource/getVerbIdsByTarget take entity ints and
  return entity/verb ints as bigint[]
- new REQUIRED verbIntsToIds(bigint[]) batch reverse resolver (L.7
  identity-fingerprint design — verb ids are UUIDs by contract, so the
  provider-side interning is losslessly reversible)
- addVerb(verb, sourceInt, targetInt) returns the interned verb int;
  removeVerb(verbId) joins the contract

Coordinator (brainy.ts) owns ALL UUID <-> int conversion: getOrAssign on
writes, getInt on reads (unmapped UUID -> empty result without calling the
provider), getUuid / verbIntsToIds on returns, plus a bounded ~100k-entry
insertion-order warm cache for verb-int -> verb-id pairs fed by addVerb
returns and resolver results. GraphVerb gains derived sourceInt/targetInt
(populated at add time, never persisted). findConnectedSubtype gains a
native fast path that routes single-type single-subtype outgoing BFS
through the provider when available.

JS GraphAdjacencyIndex satisfies the contract while staying string/u32-keyed
internally: entity ints resolve through the shared entity-id mapper (threaded
in by the coordinator on init/fork/checkout), verb ints come from an
in-process append-only interning map re-derived from storage on
rebuild/cold-start. ColumnStoreProvider widens the same way: addEntity/
removeEntity take bigint, sortTopK/filteredSortTopK return bigint[].

relate() now rejects a caller-supplied id with a teaching error — verb ids
are brainy-generated UUIDs by contract in 8.0 (previously a passed id was
silently ignored). No Roaring64 provider-boundary decode site exists yet;
the JS-internal column store stays Roaring32 and the Treemap decoder lands
with the first consumer of provider-returned filter buffers.

Public brain API unchanged. 1413 tests green (+10 new BigInt contract tests).
2026-06-10 10:45:45 -07:00
62f6472fa0 chore(8.0): delete vectorStore:mmap wiring — dead in the 8.0 provider world
The mmap-vector backend was a 2.x-era accelerator for the JS HNSW read path:
a native provider registered under 'vectorStore:mmap' supplied a single-file
mmap'd vector store, and JsHnswVectorIndex tried it before per-entity storage
reads with lazy write-back migration.

In the 8.0 provider world nothing registers that key — the native plugin's
3.0 line replaces the entire vector index under the 'vector' provider key
(vectors live inside its own index file), and the open-core path never had
an mmap provider. The wiring is unreachable; per the no-unwired-code rule
it goes away:

- src/hnsw/mmapVectorBackend.ts deleted (175 LOC)
- wireMmapVectorBackend() + its init call site removed from brainy.ts
- VectorStoreMmapProvider + VectorStoreMmapInstance contracts removed
  from plugin.ts
- JsHnswVectorIndex loses the vectorBackend field, setVectorBackend(),
  and the mmap-first branches in getVectorSafe/preloadVectors — the
  storage + UnifiedCache path is now the single read path
- tests/unit/hnsw/mmap-vector-backend.test.ts deleted

The 7.x line keeps the wiring and got the capacity-NaN bugfix as 7.31.3
(see the platform handoff mmap thread). EntityIdMapperProvider stays — the
connections codec and the id mapper still implement it.

1403/1403 tests, build clean.
2026-06-10 09:40:38 -07:00
b20666e020 refactor(8.0): final cleanup — drop HnswProvider alias + config.hnsw + 'hnsw' surface (scaffold step 6)
Step 6 of the brainy 8.0 rename scaffolding. Removes every legacy 'hnsw'-
flavoured public name introduced as a compat shim in steps 1-5. The
algorithm-neutral surface is now the only surface.

REMOVED — PHASE A (type aliases)

src/plugin.ts
- Removed `export type HnswProvider = VectorIndexProvider` alias.
- Removed `export type DiskAnnProvider = VectorIndexProvider` alias.

src/index.ts
- Removed `export const HNSWIndex = JsHnswVectorIndex` alias.

src/types/brainy.types.ts
- Removed `BrainyStats.indexHealth.hnsw` field (the new `vector` field
  carries the same boolean).

src/brainy.ts
- `stats()` no longer emits the legacy `hnsw` field on `indexHealth`.

REMOVED — PHASE B (storage adapter rename, 8 adapters)

Repo-wide rename: `saveHNSWData` → `saveVectorIndexData` and
`getHNSWData` → `getVectorIndexData` across:

- src/storage/adapters/baseStorageAdapter.ts (abstract declarations)
- src/storage/adapters/fileSystemStorage.ts
- src/storage/adapters/gcsStorage.ts
- src/storage/adapters/r2Storage.ts
- src/storage/adapters/s3CompatibleStorage.ts
- src/storage/adapters/azureBlobStorage.ts
- src/storage/adapters/opfsStorage.ts
- src/storage/adapters/memoryStorage.ts
- src/storage/adapters/historicalStorageAdapter.ts
- src/brainy.ts (call sites)
- src/hnsw/hnswIndex.ts + src/hnsw/typeAwareHNSWIndex.ts (call sites)

The default-delegation wrappers added in scaffold step 4 are removed
(would have been duplicate declarations after the rename).

REMOVED — PHASE C (cache category 'hnsw')

src/utils/unifiedCache.ts
- Cache-category union narrowed from
    'hnsw' | 'vectors' | 'metadata' | 'embedding' | 'other'
  to
    'vectors' | 'metadata' | 'embedding' | 'other'
- typeAccessCounts, typeSizes, typeCounts, accessRatios, sizeRatios,
  and the fairness-check iterator all drop the 'hnsw' key.
- Per planning § 2.5: pre-8.0 'hnsw' cache entries are an in-memory
  category. Cache is rebuildable; entries naturally don't exist after
  a restart, so no migration path is needed.

src/hnsw/hnswIndex.ts
- 3 cache.set() call sites migrated from category 'hnsw' to 'vectors'.
- Cache key prefix `hnsw:vector:` → `vector:`.
- typeCounts/typeSizes/typeAccessCounts accessor renames
  `.hnsw` → `.vectors`.

tests/unit/utils/unifiedCache-eviction.test.ts
- Test fixtures updated: cache.set(..., 'hnsw', ...) → cache.set(..., 'vectors', ...).
- Stats assertion `stats.typeSizes.hnsw` → `stats.typeSizes.vectors`.

REMOVED — PHASE D (config.hnsw)

src/types/brainy.types.ts
- Removed `BrainyConfig.hnsw` field entirely. The 8.0 surface is
  `BrainyConfig.vector.{recall, quantization, vectorStorage, advanced}`.

src/brainy.ts
- normalizeConfig() no longer emits a `hnsw` field on Required<BrainyConfig>.
- setupIndex() rewired:
  - Reads from `this.config.vector` (not `this.config.hnsw`).
  - Calls `resolveJsHnswConfig(this.config.vector)` to translate the
    `recall` preset into M / efConstruction / efSearch knobs (with
    `advanced.hnsw` overrides winning when supplied).
- Imports `resolveJsHnswConfig` from './utils/recallPreset.js'.

NOT IN THIS COMMIT (deliberately)

- Persisted file path migration `_system/hnsw-*.json` →
  `_system/vector-index-*.json`. Requires dual-read logic on boot
  across 8 storage adapters. Per integration doc lines 531-539:
  "Reader accepts either spelling on load; writer emits the new spelling
  only; brains self-migrate on the next persist after upgrade." This is
  a separate body of work and lands in a follow-up commit before 8.0 GA.
- `config.hnswPersistMode` top-level field is unchanged. It's not in
  the rename inventory; a future commit can fold it into
  `config.vector.persistMode` if desired.
- strictConfig enforcement wiring. The field is accepted by
  normalizeConfig() with default 'warn'; the actual warning emission at
  knob-mismatch sites lands when the config-resolution layer is touched
  for the persisted-path migration.

VERIFICATION

- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit (one test fixture updated to use the new
  category name; assertion still passes after fixture rename)
- npm run build: clean

The 8.0 PR's user-facing surface is now algorithm-neutral end-to-end:
VectorIndexProvider, config.vector.recall, JsHnswVectorIndex,
saveVectorIndexData, 'vectors' cache category, indexHealth.vector,
strictConfig — all standalone. No legacy 'hnsw' name remains in any
public type or method signature.
2026-06-09 13:28:53 -07:00
f39d420cb4 refactor(8.0): rename HNSWIndex class → JsHnswVectorIndex (scaffold step 3)
Step 3 of the brainy 8.0 rename scaffolding. The class name now matches
its role: the JS HNSW implementation of the VectorIndexProvider contract.
Per planning § 2.3: this is NOT cosmetic — leaving HNSWIndex when the
public contract is VectorIndexProvider creates a confusing read at the
implementation layer.

CHANGES

Repo-wide mechanical rename: HNSWIndex → JsHnswVectorIndex across 52
references in src/ + 5 references in tests/ via sed. Class declaration,
imports, JSDoc, comments, and identifiers all updated.

src/index.ts
- Primary public export: JsHnswVectorIndex (the new name).
- Backwards-compat alias: `export const HNSWIndex = JsHnswVectorIndex`
  with @deprecated note. Removed in the final 8.0 cleanup commit.

NO-OP scope

No behavioural change. The class is exactly the same; only the name
moved. Tests + build green.

VERIFICATION

- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
2026-06-09 13:07:56 -07:00
076c26f6fd refactor(8.0): rename HnswProvider → VectorIndexProvider (8.0 scaffold)
Brainy 8.0 collapses the vector-index contract to algorithm-neutral names.
"Vector index" describes the role; "HNSW" was the name of one specific
implementation. The role name is what the public contract should carry;
the algorithm name belongs to the concrete class.

This is step 1 of the brainy 8.0 rename scaffolding tracked in handoff
thread BRAINY-8.0-RENAME-COORDINATION § A.1 (cortex shipped the matching
VectorIndexProvider in commit 0e4d637).

CHANGES

src/plugin.ts
- Primary interface name flipped: HnswProvider → VectorIndexProvider.
  Same byte-for-byte shape (8 methods, no signatures changed).
- HnswProvider kept as a deprecated type alias so call sites compile
  mid-migration. Removed in a later 8.0 commit.
- DiskAnnProvider was already a type alias of HnswProvider; redeclared
  as alias of VectorIndexProvider with a deprecation note explaining
  the 'diskann' provider key folds into 'vector' in 8.0.
- JSDoc updated to describe the implementation-neutral surface:
  Brainy's own JS HNSW + any native acceleration provider both
  satisfy it.

src/hnsw/hnswIndex.ts
- Internal class HNSWIndex now declares `implements VectorIndexProvider`
  instead of `implements HnswProvider`. Import + class comment updated.

NO-OP scope

No behavioural change. Every existing call site continues to work
because HnswProvider remains as a temporary alias. Tests + build green.

VERIFICATION

- npx tsc --noEmit: clean
- npm test: 1468 / 1468 unit
- npm run build: clean (verified at parent commit 89e4d81; this commit
  is type-only so doesn't change dist)

NEXT IN SCAFFOLDING

- Add 'vector' provider key registration alongside 'hnsw' (cache + cortex)
- Add config.vector top-level path alongside config.hnsw with recall preset
- Migrate brainy.ts call sites to the new names
- Rename HNSWIndex class → JsHnswVectorIndex (per planning § 2.3)
- Final cleanup commit: remove HnswProvider alias + config.hnsw + 'hnsw' key
2026-06-09 12:58:26 -07:00
f885f813fe feat(plugin): DiskAnnProvider contract + HNSWConfig.type/diskann knobs
Adds the plugin-side surface for the new billion-scale index option:

- plugin.ts: DiskAnnProvider type alias (mirrors HnswProvider's shape
  so cortex's DiskANN wrapper is a drop-in for the 'hnsw' factory).
- coreTypes.ts: HNSWConfig.type ('hnsw' | 'diskann') for explicit
  engine selection, plus HNSWConfig.diskann for the build-time tuning
  knobs (pqM, pqKsub, maxDegree, searchListSize, alpha, mmap
  adjacency).

No algorithm code here — the implementation lives in the cortex
plugin. Brainy without cortex continues to use HNSW exactly as before.
2026-05-28 14:29:12 -07:00
617c156feb feat: graph link compression — delta-varint connections (2.4.0 #3)
Cortex registers a graph:compression provider exposing `encode`/`decode`
for HNSW connection lists (delta-varint, ~4× smaller edges than the
JSON-UUID-array shape brainy has persisted since the beginning). This
wires the consumer side without changing the saveHNSWData/getHNSWData
adapter signatures.

Architecture:

- NEW ConnectionsCodec (src/hnsw/connectionsCodec.ts) — translates between
  UUID-keyed Map<level, Set<UUID>> and a single compact per-node buffer
  via the provider + stable EntityIdMapper. Custom wire format:
  [count: u8] [[level: u8] [len: u32 LE] [bytes...]]*. One blob per node
  regardless of level count, so the storage I/O is identical to the
  legacy path (one saveHNSWData call) plus a single saveBinaryBlob.

- HNSWIndex changes — a `connectionsCodec: ConnectionsCodec | null` field
  with `setConnectionsCodec()` setter. THREE save sites (deferred flush,
  immediate-mode entity persist, immediate-mode neighbor updates) now go
  through a single `persistNodeConnections(nodeId, noun)` helper: when the
  codec is wired AND the storage adapter exposes saveBinaryBlob, it
  encodes the connections, stores them at `_hnsw_conn/<nodeId>`, and
  records `connections: {}` in saveHNSWData as the marker. Otherwise the
  legacy JSON-array path is taken. TWO load sites use a matching
  `restoreNodeConnections` helper that tries loadBinaryBlob first and
  falls back to the legacy hnswData.connections field on miss / decode
  error. Format convergence is lazy: pre-2.4.0 nodes still load via the
  legacy path, then write the compressed form on next dirty save.

- brainy.ts wireConnectionsCodec — runs alongside wireMmapVectorBackend
  during init. Activates when (a) the graph:compression provider is
  registered and (b) the metadata index exposes its idMapper. Unlike the
  mmap-vector backend, this layer engages on EVERY brainy 7.25.0
  adapter — the blob primitive itself is universal; only the codec
  presence gates activation.

- Provider interface GraphCompressionProvider in plugin.ts — encode +
  decode static signatures. Brainy depends on the interface; cortex's
  registered { encode: encodeConnections, decode: decodeConnections }
  satisfies it structurally.

Tests (1437 total, +4 vs prior tip):

- tests/unit/hnsw/connections-codec.test.ts — 4 unit tests with a JSON
  mock provider: single-level round-trip, empty-Map round-trip (one-byte
  buffer), multi-level fan-out round-trip, and silent-drop of UUIDs the
  decoding idMapper no longer knows. The real cross-language byte
  format is exercised when cortex 2.4.0 wires its delta-varint
  encode/decode in.

This completes the brainy half of the 2.4.0 storage foundation: stable
ids (#23), mmap vectors (#24), graph link compression (#25), and
column-store interchange (#26). Coordinated release as brainy 7.26.0 +
cortex 2.4.0 follows.
2026-05-28 10:37:47 -07:00
d4cb26c604 feat: mmap-vector backend wiring — HNSWIndex consumes vectorStore:mmap (2.4.0 #2)
Cortex already registers the vectorStore:mmap provider (its Rust
NativeMmapVectorStore), but brainy has never consumed it — preloadVectors
and getVectorSafe still go straight to storage.getNounVector for every id,
even when an mmap layer is available. This wires the consumer end.

Architecture:

- NEW MmapVectorBackend (src/hnsw/mmapVectorBackend.ts) — bridges brainy's
  UUID-keyed vector reads to a int-slot mmap file via the
  vectorStore:mmap provider. Slots are addressed by the stable int id
  from the post-2.4.0 #1 EntityIdMapper (the foundation this depends on).
  Auto-grows the file (doubling) when a write lands beyond capacity, so
  HNSWIndex never has to think about sizing. The class never touches
  per-entity storage — it owns only the mmap layer.

- HNSWIndex changes — adds a vectorBackend field + a setVectorBackend
  setter. The vector read paths (preloadVectors, getVectorSafe) try the
  mmap layer first; on a storage fallback hit, they LAZILY write back into
  the mmap slot. An upgraded install converges to the zero-copy fast path
  under live traffic — no big-bang migration step. The legacy per-entity
  path is preserved and still used when no backend is set.

- brainy.ts wiring — a new private wireMmapVectorBackend() runs once
  during init, after plugin activation + metadataIndex setup. It activates
  the backend only when (a) the vectorStore:mmap provider is registered,
  (b) the storage adapter resolves a real local path via
  getBinaryBlobPath(), and (c) the metadata index exposes its idMapper.
  Cloud adapters return null on (b) and the backend is silently skipped;
  HNSWIndex's behaviour is then identical to pre-2.4.0.

- Provider interfaces in plugin.ts — VectorStoreMmapProvider and
  VectorStoreMmapInstance document the contract cortex's class fulfils
  (the class IS the provider — static factory methods). Brainy depends on
  the interfaces, not on cortex; the structural match is verified when
  cortex 2.4.0 picks up this brainy release.

Tests (1428 total, +11 vs pre-2.4.0):

- tests/unit/hnsw/mmap-vector-backend.test.ts — 6 unit tests with an
  in-memory mock provider. Covers round-trip, batch reads with interleaved
  misses, slot stability (no re-slotting on overwrite), file growth without
  data loss, idempotent open, and null returns for unwritten slots. The
  real perf integration with cortex's NativeMmapVectorStore is exercised
  when cortex 2.4.0 wires this in.

- tests/unit/utils/entity-id-mapper-stability.test.ts — moved here from
  tests/regression/ (which is NOT in the unit-config include glob, so the
  five #23 tests were not actually being run by npm test). The unit
  config matches tests/unit/**/*.test.ts.

The 2.4.0 #2 follow-up will be the chunked-segment layout for remote
storage adapters (S3 / R2 / GCS) where a single growing file doesn't fit
immutable objects. For 2.4.0 release: local-FS only.
2026-05-28 10:37:47 -07:00
4b6f63ed67 feat: export provider contracts for the plugin surface brainy consumes
Native accelerators (cortex) register providers for metadataIndex, graphIndex,
hnsw, entityIdMapper, cache, columnStore, and aggregation. Until now the exact
method/property surface brainy calls on each was implicit — a provider could
drop a member brainy depends on and only fail at runtime when that path ran.

This defines and exports the provider contracts from the stable
@soulcraft/brainy/plugin entrypoint:
- MetadataIndexProvider, GraphIndexProvider, HnswProvider,
  EntityIdMapperProvider, CacheProvider — each typed as exactly the surface
  brainy calls (optional/feature-detected members like HNSW setPersistMode and
  enableCOW are intentionally excluded).
- Re-exports ColumnStoreProvider and AggregationProvider (and the aggregate
  types) from the same entrypoint so a plugin author can import the whole
  provider surface from one place.

Brainy's own baseline classes now `implements` these contracts
(MetadataIndexManager, GraphAdjacencyIndex, HNSWIndex, EntityIdMapper,
UnifiedCache), so the interfaces can never silently diverge from what brainy
ships — and any provider that declares `implements` gets a compile error the
moment brainy starts requiring a new member.

The embeddings/embedBatch providers stay typed by the existing EmbeddingFunction
(no duplicate interface added). Type-only changes; no runtime behavior change.
2026-05-27 14:45:40 -07:00
f024e56ee7 feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows
Add a write-time incremental aggregation engine that maintains running
totals on every add/update/delete for O(1) read performance. Integrates
into brain.find({ aggregate }) for a unified query API.

Core features:
- AggregationIndex with defineAggregate()/removeAggregate() API
- Five aggregation operations: SUM, COUNT, AVG, MIN, MAX
- GROUP BY with multiple dimensions including time windows
- Time window bucketing: hour, day, week, month, quarter, year, custom
- Materialization of results as NounType.Measurement entities
- Debounced persistence of definitions and state to storage
- Definition change detection via FNV-1a hashing with auto-rebuild
- Infinite loop prevention for materialized entities
- 'aggregation' plugin provider key for native acceleration
- Lazy initialization (created on first defineAggregate() call)
- 73 tests (unit + integration) covering all functionality

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 16:57:53 -08:00
932fb9520b fix: set verb.source/target to entity UUID instead of NounType
relate() was setting verb.source = fromEntity.type (a NounType like
"concept") instead of the entity UUID. Cortex's graph index indexes by
verb.source, so lookups by UUID found nothing — causing in-session
reads to return 0 results.

Also fixes:
- PathResolver calling private getIdsFromChunks() → public getIds()
- Plugin auto-detection removed; cortex loads only via explicit config
- GraphVerb types accept number timestamps and sourceId/targetId aliases
- Dead autoDetect() method removed from PluginRegistry
- In-session regression tests added for getRelations after relate()
2026-02-02 09:20:38 -08:00
401e300ff2 feat: harden plugin system wiring and add developer diagnostics
Fix critical wiring bugs that prevented plugin-provided implementations
from being used at runtime. All CRUD operations, fork/checkout/clear,
batch embedding, neural APIs, and VFS path resolution now properly
dispatch through the plugin registry.

Changes:
- Wire graphIndex to storage for getVerbsBySource() fast path
- Replace instanceof checks with duck-typing (indexIsTypeAware flag)
  so plugin HNSW indexes work in add/update/delete/search
- Add createIndex() shared helper for plugin HNSW factory
- Fix fork/checkout/clear to use plugin factories for metadataIndex,
  graphIndex, and HNSW instead of hardcoding JS constructors
- Add three-tier embedBatch priority: embedBatch > embeddings > WASM
- Skip WASM warmup/eagerEmbeddings when plugin provides embeddings
- Fix PathResolver metadataIndex access (was looking on storage)
- Use global UnifiedCache in SemanticPathResolver
- Wire plugin distance function through neural APIs
- Add diagnostics() method and CLI command for provider inspection
- Add requireProviders() for production fail-fast assertions
- Add init-time provider summary log
- Add plugin developer documentation (docs/PLUGINS.md)
- Export DiagnosticsResult type
2026-02-01 13:03:15 -08:00
7f9d2a70a5 feat: update plugin references from @soulcraft/brainy-cortex to @soulcraft/cortex 2026-02-01 08:22:07 -08:00
cc50ac3776 feat: add plugin system for cortex and storage adapters
Simple plugin architecture with two use cases:
1. Native acceleration (@soulcraft/brainy-cortex) — auto-detected
2. Custom storage adapters (e.g., Redis, DynamoDB)

Plugins implement BrainyPlugin interface with activate/deactivate lifecycle.
Auto-detection tries dynamic import of known packages during init().
Manual registration via brain.use(plugin) before init().

Provider keys: metadataIndex, graphIndex, entityIdMapper, cache, hnsw,
roaring, embeddings, distance, msgpack, storage:<name>
2026-01-31 09:22:32 -08:00