Compare commits
10 commits
feat/warm-
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 415e824a1d | |||
| 069a889489 | |||
| d918c060f4 | |||
| 4c8e384bc8 | |||
| 9a99a7b962 | |||
| 6ba94c8c43 | |||
| 3a1efc9460 | |||
| 3be4ba96c2 | |||
| 55b867c998 | |||
| 9a5a9cccbc |
22 changed files with 1316 additions and 366 deletions
|
|
@ -2,6 +2,15 @@
|
||||||
|
|
||||||
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
||||||
|
|
||||||
|
### [8.10.0](https://github.com/soulcraftlabs/brainy/compare/v8.9.0...v8.10.0) (2026-07-23)
|
||||||
|
|
||||||
|
- docs: adoption storefront — contributing guide, security policy, README support + cor section (9a99a7b)
|
||||||
|
- fix(release): push the public mirror explicitly and verify the tag lands at the right commit before publishing (6ba94c8)
|
||||||
|
- docs: project guide version line points at npm instead of a hardcoded stale number (3a1efc9)
|
||||||
|
- feat: vector provider identity is a required name field (hnsw-js), rendered [vector-index:<name>] (3be4ba9)
|
||||||
|
- feat: warm contract (warm/warmOnOpen/provider warm hook), configurable transact budget floor, backend-neutral vector index op names (55b867c)
|
||||||
|
|
||||||
|
|
||||||
### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19)
|
### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19)
|
||||||
|
|
||||||
- docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78)
|
- docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78)
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,7 @@ Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
|
||||||
|
|
||||||
**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
|
**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
|
||||||
|
|
||||||
**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`)
|
**Current version:** run `npm view @soulcraft/brainy version` (never trust a hardcoded number here — this line went stale for months); consumer-facing changes tracked in `RELEASES.md`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
318
CONTRIBUTING.md
318
CONTRIBUTING.md
|
|
@ -1,298 +1,66 @@
|
||||||
# Contributing to Brainy
|
# Contributing to Brainy
|
||||||
|
|
||||||
Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for contributing to the project.
|
Brainy is MIT-licensed and genuinely open to outside contributions. This page
|
||||||
|
is the honest, current path — please don't rely on older instructions you
|
||||||
|
may find elsewhere in the repo's history.
|
||||||
|
|
||||||
## Code of Conduct
|
## Where the project lives
|
||||||
|
|
||||||
By participating in this project, you agree to abide by our Code of Conduct:
|
The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/brainy**.
|
||||||
- Be respectful and inclusive
|
It's anonymously readable and cloneable — no account needed to browse, clone,
|
||||||
- Welcome newcomers and help them get started
|
or build.
|
||||||
- Focus on constructive criticism
|
|
||||||
- Respect differing viewpoints and experiences
|
|
||||||
|
|
||||||
## How to Contribute
|
## How to contribute
|
||||||
|
|
||||||
### Reporting Issues
|
**Found a bug, or have an idea?** Email **brainy@soulcraft.com**. No account,
|
||||||
|
no ceremony — you'll get a receipt, and it goes to a human.
|
||||||
|
|
||||||
Before creating an issue, please check existing issues to avoid duplicates.
|
**Want to send a patch?** Two ways, both first-class:
|
||||||
|
|
||||||
When creating an issue, include:
|
- **Email a patch.** Run `git format-patch` against your change and email the
|
||||||
- Clear, descriptive title
|
output to **brainy@soulcraft.com**. This is a genuinely supported path, not
|
||||||
- Detailed description of the problem
|
a fallback — plenty of good contributions arrive this way.
|
||||||
- Steps to reproduce
|
- **Open a pull request on the forge.** Request an account at
|
||||||
- Expected vs actual behavior
|
**source.soulcraft.com** (registration is request-with-approval, so allow
|
||||||
- System information (OS, Node version, Brainy version)
|
a little lag), clone, push a branch, and open a PR there. Maintainers
|
||||||
- Code examples if applicable
|
review and land it.
|
||||||
|
|
||||||
### Suggesting Features
|
Either way, for anything beyond a small fix, opening an issue first (email is
|
||||||
|
fine) to talk through the approach saves everyone rework.
|
||||||
|
|
||||||
Feature requests are welcome! Please provide:
|
## Development setup
|
||||||
- Clear use case
|
|
||||||
- Proposed API/interface
|
|
||||||
- Examples of how it would work
|
|
||||||
- Any potential challenges or considerations
|
|
||||||
|
|
||||||
### Pull Requests
|
|
||||||
|
|
||||||
#### Before Starting
|
|
||||||
|
|
||||||
1. Check existing issues and PRs
|
|
||||||
2. Open an issue to discuss significant changes
|
|
||||||
3. Fork the repository
|
|
||||||
4. Create a feature branch from `main`
|
|
||||||
|
|
||||||
#### Development Setup
|
|
||||||
|
|
||||||
**Quick Setup (Recommended):**
|
|
||||||
```bash
|
```bash
|
||||||
# Clone your fork
|
git clone https://source.soulcraft.com/soulcraft/brainy.git
|
||||||
git clone https://github.com/your-username/brainy.git
|
|
||||||
cd brainy
|
cd brainy
|
||||||
|
|
||||||
# Run setup script (installs all dependencies including Rust)
|
|
||||||
./scripts/setup-dev.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
**Manual Setup:**
|
|
||||||
```bash
|
|
||||||
# Clone your fork
|
|
||||||
git clone https://github.com/your-username/brainy.git
|
|
||||||
cd brainy
|
|
||||||
|
|
||||||
# Install system dependencies (Ubuntu/Debian)
|
|
||||||
sudo apt-get install -y build-essential pkg-config libssl-dev
|
|
||||||
|
|
||||||
# Install Rust (for WASM embedding engine)
|
|
||||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
|
||||||
source ~/.cargo/env
|
|
||||||
rustup target add wasm32-unknown-unknown
|
|
||||||
cargo install wasm-pack
|
|
||||||
|
|
||||||
# Install Node.js dependencies
|
|
||||||
npm install
|
npm install
|
||||||
|
|
||||||
# Build Candle WASM embedding engine
|
|
||||||
npm run build:candle
|
|
||||||
|
|
||||||
# Build TypeScript
|
|
||||||
npm run build
|
npm run build
|
||||||
|
|
||||||
# Run tests
|
|
||||||
npm test
|
npm test
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Making Changes
|
Tests run on [Vitest](https://vitest.dev/). `npm test` runs the unit suite;
|
||||||
|
see `package.json` for `test:integration`, `test:coverage`, and friends.
|
||||||
|
|
||||||
1. **Follow the code style**
|
## Standards
|
||||||
- TypeScript for all source code
|
|
||||||
- Clear variable and function names
|
|
||||||
- Comments for complex logic
|
|
||||||
- JSDoc for public APIs
|
|
||||||
|
|
||||||
2. **Write tests**
|
- **Strict TypeScript.** No `any` escape hatches to dodge the type checker.
|
||||||
- Add tests for new features
|
- **Tests exercise real behavior.** No mocking away the thing you're supposed
|
||||||
- Update tests for changes
|
to be testing.
|
||||||
- Ensure all tests pass
|
- **No stubs, no TODO-code.** If something can't be finished, say so and
|
||||||
|
leave it out — don't merge a placeholder.
|
||||||
|
- **JSDoc on every exported function, class, and type.**
|
||||||
|
- **[Conventional Commits](https://www.conventionalcommits.org/).** `feat:`,
|
||||||
|
`fix:`, `docs:`, `perf:`, `refactor:`, `test:`, `chore:`. Never
|
||||||
|
`BREAKING CHANGE` in a commit message — major version bumps are a separate,
|
||||||
|
deliberate decision.
|
||||||
|
- **Performance claims are measured or labeled projected.** If a PR or its
|
||||||
|
description states a number, cite the benchmark that produced it (see
|
||||||
|
[docs/performance-envelopes.md](docs/performance-envelopes.md) for the
|
||||||
|
pattern). Don't state an estimate as if it were measured.
|
||||||
|
|
||||||
3. **Update documentation**
|
## License
|
||||||
- Update README if needed
|
|
||||||
- Add/update API documentation
|
|
||||||
- Include examples
|
|
||||||
|
|
||||||
#### Commit Guidelines
|
Brainy is [MIT licensed](LICENSE). Contributions are accepted under the same
|
||||||
|
license — there's no CLA to sign.
|
||||||
|
|
||||||
Follow conventional commits format:
|
Thank you for considering a contribution.
|
||||||
|
|
||||||
```
|
|
||||||
type(scope): description
|
|
||||||
|
|
||||||
[optional body]
|
|
||||||
|
|
||||||
[optional footer]
|
|
||||||
```
|
|
||||||
|
|
||||||
Types:
|
|
||||||
- `feat`: New feature
|
|
||||||
- `fix`: Bug fix
|
|
||||||
- `docs`: Documentation changes
|
|
||||||
- `style`: Code style changes
|
|
||||||
- `refactor`: Code refactoring
|
|
||||||
- `perf`: Performance improvements
|
|
||||||
- `test`: Test changes
|
|
||||||
- `chore`: Build/tooling changes
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
```bash
|
|
||||||
feat(triple): add graph traversal depth limit
|
|
||||||
fix(storage): handle concurrent write conflicts
|
|
||||||
docs(api): update search method documentation
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Submitting PR
|
|
||||||
|
|
||||||
1. Push to your fork
|
|
||||||
2. Create PR against `main` branch
|
|
||||||
3. Fill out PR template
|
|
||||||
4. Ensure CI checks pass
|
|
||||||
5. Wait for review
|
|
||||||
|
|
||||||
### Testing
|
|
||||||
|
|
||||||
#### Running Tests
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Run all tests
|
|
||||||
npm test
|
|
||||||
|
|
||||||
# Run specific test file
|
|
||||||
npm test tests/core.test.ts
|
|
||||||
|
|
||||||
# Run with coverage
|
|
||||||
npm run test:coverage
|
|
||||||
|
|
||||||
# Watch mode
|
|
||||||
npm run test:watch
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Writing Tests
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { describe, it, expect } from 'vitest'
|
|
||||||
import { Brainy } from '../src'
|
|
||||||
|
|
||||||
describe('Feature Name', () => {
|
|
||||||
it('should do something specific', async () => {
|
|
||||||
const brain = new Brainy()
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Test implementation
|
|
||||||
const result = await brain.search("test")
|
|
||||||
|
|
||||||
expect(result).toBeDefined()
|
|
||||||
expect(result.length).toBeGreaterThan(0)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
## Architecture Guidelines
|
|
||||||
|
|
||||||
### Adding New Features
|
|
||||||
|
|
||||||
1. **Check existing functionality**
|
|
||||||
- Review `ARCHITECTURE.md`
|
|
||||||
- Check if similar features exist
|
|
||||||
- Consider if it should be an augmentation
|
|
||||||
|
|
||||||
2. **Design considerations**
|
|
||||||
- Maintain backward compatibility
|
|
||||||
- Consider performance impact
|
|
||||||
- Think about all storage adapters
|
|
||||||
- Plan for extensibility
|
|
||||||
|
|
||||||
3. **Implementation checklist**
|
|
||||||
- [ ] Core functionality
|
|
||||||
- [ ] Tests (unit and integration)
|
|
||||||
- [ ] Documentation
|
|
||||||
- [ ] TypeScript types
|
|
||||||
- [ ] Examples
|
|
||||||
- [ ] Performance benchmarks (if applicable)
|
|
||||||
|
|
||||||
### Creating Augmentations
|
|
||||||
|
|
||||||
Augmentations extend Brainy's functionality:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { BrainyAugmentation } from '../types'
|
|
||||||
|
|
||||||
export class MyAugmentation extends BrainyAugmentation {
|
|
||||||
name = 'MyAugmentation'
|
|
||||||
|
|
||||||
async onInit(brain: Brainy): Promise<void> {
|
|
||||||
// Initialize augmentation
|
|
||||||
}
|
|
||||||
|
|
||||||
async onAdd(item: any, brain: Brainy): Promise<any> {
|
|
||||||
// Process before adding
|
|
||||||
return item
|
|
||||||
}
|
|
||||||
|
|
||||||
async onSearch(query: any, results: any[], brain: Brainy): Promise<any[]> {
|
|
||||||
// Process search results
|
|
||||||
return results
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Performance Considerations
|
|
||||||
|
|
||||||
- Use batch operations where possible
|
|
||||||
- Implement caching strategically
|
|
||||||
- Consider memory usage
|
|
||||||
- Profile performance impacts
|
|
||||||
- Add benchmarks for critical paths
|
|
||||||
|
|
||||||
## Documentation
|
|
||||||
|
|
||||||
### API Documentation
|
|
||||||
|
|
||||||
Use JSDoc for all public APIs:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
/**
|
|
||||||
* Searches for similar items using vector similarity
|
|
||||||
* @param query - Search query (text or vector)
|
|
||||||
* @param options - Search options
|
|
||||||
* @returns Array of search results with scores
|
|
||||||
* @example
|
|
||||||
* ```typescript
|
|
||||||
* const results = await brain.search("machine learning", { limit: 10 })
|
|
||||||
* ```
|
|
||||||
*/
|
|
||||||
async search(query: string | Vector, options?: SearchOptions): Promise<SearchResult[]> {
|
|
||||||
// Implementation
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Examples
|
|
||||||
|
|
||||||
Add examples for new features:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// examples/feature-name.ts
|
|
||||||
import { Brainy } from 'brainy'
|
|
||||||
|
|
||||||
async function exampleUsage() {
|
|
||||||
const brain = new Brainy()
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
// Show feature usage
|
|
||||||
// Include comments explaining what's happening
|
|
||||||
// Handle errors appropriately
|
|
||||||
}
|
|
||||||
|
|
||||||
exampleUsage().catch(console.error)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Release Process
|
|
||||||
|
|
||||||
1. **Version bump**: Follow semantic versioning
|
|
||||||
2. **Update CHANGELOG**: Document all changes
|
|
||||||
3. **Run tests**: Ensure all tests pass
|
|
||||||
4. **Build**: Generate distribution files
|
|
||||||
5. **Tag**: Create git tag for version
|
|
||||||
6. **Publish**: Release to npm
|
|
||||||
|
|
||||||
## Getting Help
|
|
||||||
|
|
||||||
- **Discord**: Join our community
|
|
||||||
- **Issues**: Ask questions on GitHub
|
|
||||||
- **Discussions**: Share ideas and get feedback
|
|
||||||
|
|
||||||
## Recognition
|
|
||||||
|
|
||||||
Contributors will be recognized in:
|
|
||||||
- CHANGELOG.md for their contributions
|
|
||||||
- README.md contributors section
|
|
||||||
- GitHub contributors page
|
|
||||||
|
|
||||||
Thank you for contributing to Brainy! 🧠
|
|
||||||
|
|
|
||||||
26
README.md
26
README.md
|
|
@ -13,7 +13,7 @@
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/v/@soulcraft/brainy.svg" alt="npm version"></a>
|
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/v/@soulcraft/brainy.svg" alt="npm version"></a>
|
||||||
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/dm/@soulcraft/brainy.svg" alt="npm downloads"></a>
|
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/dm/@soulcraft/brainy.svg" alt="npm downloads"></a>
|
||||||
<a href="https://github.com/soulcraftlabs/brainy/actions/workflows/ci.yml"><img src="https://github.com/soulcraftlabs/brainy/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
|
<a href="https://source.soulcraft.com/soulcraft/brainy/actions"><img src="https://source.soulcraft.com/soulcraft/brainy/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"></a>
|
||||||
<a href="https://soulcraft.com/docs"><img src="https://img.shields.io/badge/docs-soulcraft.com-blue.svg" alt="Documentation"></a>
|
<a href="https://soulcraft.com/docs"><img src="https://img.shields.io/badge/docs-soulcraft.com-blue.svg" alt="Documentation"></a>
|
||||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a>
|
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a>
|
||||||
<a href="https://www.typescriptlang.org/"><img src="https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg" alt="TypeScript"></a>
|
<a href="https://www.typescriptlang.org/"><img src="https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg" alt="TypeScript"></a>
|
||||||
|
|
@ -23,8 +23,9 @@
|
||||||
<a href="#quick-start">Quick start</a> ·
|
<a href="#quick-start">Quick start</a> ·
|
||||||
<a href="#one-query-three-engines">One query</a> ·
|
<a href="#one-query-three-engines">One query</a> ·
|
||||||
<a href="#feature-tour">Features</a> ·
|
<a href="#feature-tour">Features</a> ·
|
||||||
<a href="#from-laptop-to-hundreds-of-millions">Scale with Cor</a> ·
|
<a href="#when-you-outgrow-brainy">Scale with Cor</a> ·
|
||||||
<a href="#documentation">Docs</a>
|
<a href="#documentation">Docs</a> ·
|
||||||
|
<a href="#support--community">Support</a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -172,9 +173,11 @@ await brain.vfs.search('React components with hooks') // semantic file
|
||||||
|
|
||||||
**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
|
**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
|
||||||
|
|
||||||
## From laptop to hundreds of millions
|
## When you outgrow Brainy
|
||||||
|
|
||||||
Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**:
|
Brainy's pure-TypeScript engines carry real workloads a long way on their own — see the measured, per-operation numbers (not marketing figures) in **[docs/performance-envelopes.md](docs/performance-envelopes.md)** for what to expect, unaccelerated, on plain filesystem storage.
|
||||||
|
|
||||||
|
When a deployment needs native-scale vector/graph performance — memory-mapped indexes that don't need your dataset in RAM, billion-scale ambitions — add the native engine. **The API doesn't change:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm install @soulcraft/cor
|
npm install @soulcraft/cor
|
||||||
|
|
@ -187,13 +190,14 @@ await brain.init() // @soulcraft/cor detected — same code, native engines un
|
||||||
|
|
||||||
Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
|
Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
|
||||||
|
|
||||||
Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both.
|
Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor is more headroom for when you need it, not capability held back to sell you later. Licensing and support: **cor@soulcraft.com**.
|
||||||
|
|
||||||
## Performance
|
## Performance
|
||||||
|
|
||||||
|
- Per-operation p50/p95 at 1k and 10k entities, pure-JS floor, measured and re-run every release that touches a measured path: **[docs/performance-envelopes.md](docs/performance-envelopes.md)**.
|
||||||
- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
|
- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
|
||||||
- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
|
- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
|
||||||
- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
|
- Capacity planning and architecture: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
|
||||||
|
|
||||||
## Use cases
|
## Use cases
|
||||||
|
|
||||||
|
|
@ -212,6 +216,10 @@ Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is
|
||||||
|
|
||||||
**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
|
**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
|
||||||
|
|
||||||
## Contributing & license
|
## Support & community
|
||||||
|
|
||||||
Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors.
|
- **Bugs and ideas** → **brainy@soulcraft.com** — no account needed, you'll get a receipt.
|
||||||
|
- **Security reports** → **security@soulcraft.com** — see **[SECURITY.md](SECURITY.md)**.
|
||||||
|
- **Contributing** → see **[CONTRIBUTING.md](CONTRIBUTING.md)**.
|
||||||
|
|
||||||
|
MIT © Brainy Contributors.
|
||||||
|
|
|
||||||
66
RELEASES.md
66
RELEASES.md
|
|
@ -31,6 +31,72 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Unreleased (the warm contract: cold-restart writes stop paying demand-load latency)
|
||||||
|
|
||||||
|
From a production deployment's cold-restart incident: the FIRST writes after every
|
||||||
|
restart on a large brain measured 33–35s each (page cache cold) against the transact
|
||||||
|
apply budget — Brainy's own op-count-scaled budget, `max(30s, opCount × 2s)`, e.g.
|
||||||
|
32,000ms for a 16-op batch. There is no external deadline in this story, and no
|
||||||
|
post-completion veto either: every write is itself a multi-operation transaction (a
|
||||||
|
single `add()` applies several operations — canonical writes, the vector-index insert,
|
||||||
|
the metadata-index update), so ONE cold operation that legitimately runs ~33s consumes
|
||||||
|
the whole budget, the gate before the NEXT operation trips, and the write rolls back
|
||||||
|
atomically (zero loss, by design) — refused, retried, refused again, until the page
|
||||||
|
cache warms passively (~30 minutes). The cure is not weaker atomicity; it is the two
|
||||||
|
new knobs below — a budget floor sized for cold stores, and a warm contract that pays
|
||||||
|
demand-load cost OFF the transaction path.
|
||||||
|
|
||||||
|
- **The budget's start-gating contract is now explicit, documented, and pinned by
|
||||||
|
regression tests.** The budget gates STARTING the next operation — completed work is
|
||||||
|
never rolled back for elapsed time — and a transaction's first operation now
|
||||||
|
unconditionally starts by code, not merely because elapsed time happens to be ~0 when
|
||||||
|
it is checked. This has been the shipped schedule since 8.7.0 (no behavior change for
|
||||||
|
existing integrations); it is now stated in `Transaction.execute()`'s contract JSDoc
|
||||||
|
and enforced by tests so it cannot silently regress. Mid-batch atomicity is unchanged:
|
||||||
|
a trip before operation `i+1` still rolls back `0..i` and throws a retryable
|
||||||
|
`TransactionTimeoutError`.
|
||||||
|
- **The budget's 30s floor is now configurable**: `new Brainy({ transactionBudgetFloorMs })`
|
||||||
|
raises (or lowers) the floor of `max(transactionBudgetFloorMs, opCount × 2000)` for every
|
||||||
|
internal transact batch. Useful for a store whose cold writes legitimately run past 30s
|
||||||
|
per operation, so a bulk batch gets a proportionally larger runway instead of tripping
|
||||||
|
mid-batch on cold-cache latency.
|
||||||
|
- **New: `brain.warm()`** — eagerly loads/faults-in the vector index, metadata index, and
|
||||||
|
graph adjacency so the first real operation after a cold restart runs at steady-state
|
||||||
|
cost instead of paying demand-load latency on the critical path. Returns a `WarmReport`
|
||||||
|
with one honest outcome per surface — never conflate the first two:
|
||||||
|
- `'warmed'` — the surface's own provider `warm()` hook ran, or a full-hydration seam
|
||||||
|
loaded every shard/field/segment from storage. Steady-state cost is paid.
|
||||||
|
- `'probed'` — no `warm()` hook was available, so a best-effort read (one `search()` call
|
||||||
|
for the vector index) faulted in *some* backing storage as a side effect — real work,
|
||||||
|
but never reported as `'warmed'`.
|
||||||
|
- `'unavailable'` — nothing ran (no hook, no hydration seam, or nothing to probe).
|
||||||
|
- **New config: `warmOnOpen: true`** makes `init()` await `brain.warm()` before it resolves
|
||||||
|
— a deliberate blocking trade-off: startup takes longer, the first request doesn't.
|
||||||
|
Default `false` (unchanged lazy behavior).
|
||||||
|
- **New optional provider hook: `warm?(): Promise<void>`** on the vector and graph
|
||||||
|
acceleration provider contracts (`src/plugin.ts`) — a native provider can implement it to
|
||||||
|
eagerly pretouch its own backing storage (e.g. mmap pretouch); absence means brainy falls
|
||||||
|
back to the probe/hydration behavior above.
|
||||||
|
- **Transaction op-name strings changed in journals/timings**: the vector-index
|
||||||
|
transaction operations were renamed from `AddToHNSW`/`RemoveFromHNSW` to backend-neutral
|
||||||
|
`AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an
|
||||||
|
algorithm that may not be the one actually running (a non-HNSW native vector provider
|
||||||
|
emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist).
|
||||||
|
The parenthesized suffix names the ACTIVE backend: `hnsw-js` for the built-in engine, or
|
||||||
|
the native provider's own identity when it self-identifies. **If you parse these op-name
|
||||||
|
strings** (log processors, journal tooling), update your matcher from
|
||||||
|
`AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`.
|
||||||
|
- **Provider identity is now a REQUIRED `name` field** on the vector provider contract
|
||||||
|
(`VectorIndexProvider.name`, `src/plugin.ts`) — every implementation self-reports its own
|
||||||
|
identity truthfully (its algorithm/engine), never inheriting a default. It renders as the
|
||||||
|
op-name suffix above and, wherever the vector index identifies itself in prose log lines,
|
||||||
|
as the tag `[vector-index:<name>]`. **Native provider adoption is a one-line change**:
|
||||||
|
declare `readonly name = '<your-provider-name>'`. A provider instance that still lacks
|
||||||
|
`name` at runtime (an older native build compiled against the previous, optional field) is
|
||||||
|
never crashed on and never silently mislabeled: it stamps `unknown-provider` and emits one
|
||||||
|
loud warning naming the missing field, so the gap is discoverable instead of a permanent
|
||||||
|
fossil label in every journal line.
|
||||||
|
|
||||||
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
|
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
|
||||||
|
|
||||||
The write path stops paying maintenance costs — the last structural piece of the
|
The write path stops paying maintenance costs — the last structural piece of the
|
||||||
|
|
|
||||||
36
SECURITY.md
Normal file
36
SECURITY.md
Normal file
|
|
@ -0,0 +1,36 @@
|
||||||
|
# Security Policy
|
||||||
|
|
||||||
|
## Reporting a vulnerability
|
||||||
|
|
||||||
|
Email **security@soulcraft.com**. That's the one door for security reports
|
||||||
|
across the company, and it works the same way for Brainy: every report is
|
||||||
|
read by a human, you'll get a private receipt, and we'll work with you on
|
||||||
|
coordinated disclosure — please don't open a public issue for anything
|
||||||
|
that isn't already public.
|
||||||
|
|
||||||
|
Include what you'd want if you were on the other end: affected version,
|
||||||
|
how to reproduce, and what you think the impact is. If you have a patch or
|
||||||
|
a suggested fix, send it along — it's welcome but not required.
|
||||||
|
|
||||||
|
There is no bounty program today. We're saying that plainly so you know
|
||||||
|
what to expect going in.
|
||||||
|
|
||||||
|
## Response time
|
||||||
|
|
||||||
|
We respond as fast as truth allows. That means: no fixed SLA, no promise of
|
||||||
|
a reply within a specific number of hours — but a real report from a real
|
||||||
|
person gets read promptly and taken seriously. If you haven't heard anything
|
||||||
|
in a reasonable stretch, a follow-up email is completely fine.
|
||||||
|
|
||||||
|
## Supported versions
|
||||||
|
|
||||||
|
The latest `8.x` minor release line receives security fixes. If you're
|
||||||
|
running an older major version, please upgrade before reporting — we can't
|
||||||
|
commit to backporting fixes to unsupported lines.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This policy covers the `@soulcraft/brainy` package itself — the code in
|
||||||
|
this repository. If you're evaluating a deployment that also uses
|
||||||
|
`@soulcraft/cor`, report issues in that package the same way, to the same
|
||||||
|
address; we'll route internally.
|
||||||
4
package-lock.json
generated
4
package-lock.json
generated
|
|
@ -1,12 +1,12 @@
|
||||||
{
|
{
|
||||||
"name": "@soulcraft/brainy",
|
"name": "@soulcraft/brainy",
|
||||||
"version": "8.9.0",
|
"version": "8.10.0",
|
||||||
"lockfileVersion": 3,
|
"lockfileVersion": 3,
|
||||||
"requires": true,
|
"requires": true,
|
||||||
"packages": {
|
"packages": {
|
||||||
"": {
|
"": {
|
||||||
"name": "@soulcraft/brainy",
|
"name": "@soulcraft/brainy",
|
||||||
"version": "8.9.0",
|
"version": "8.10.0",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@msgpack/msgpack": "^3.1.2",
|
"@msgpack/msgpack": "^3.1.2",
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
{
|
{
|
||||||
"name": "@soulcraft/brainy",
|
"name": "@soulcraft/brainy",
|
||||||
"version": "8.9.0",
|
"version": "8.10.0",
|
||||||
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
|
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
|
||||||
"main": "dist/index.js",
|
"main": "dist/index.js",
|
||||||
"module": "dist/index.js",
|
"module": "dist/index.js",
|
||||||
|
|
@ -128,13 +128,13 @@
|
||||||
"publishConfig": {
|
"publishConfig": {
|
||||||
"access": "public"
|
"access": "public"
|
||||||
},
|
},
|
||||||
"homepage": "https://github.com/soulcraftlabs/brainy",
|
"homepage": "https://source.soulcraft.com/soulcraft/brainy",
|
||||||
"bugs": {
|
"bugs": {
|
||||||
"url": "https://github.com/soulcraftlabs/brainy/issues"
|
"url": "https://source.soulcraft.com/soulcraft/brainy/issues"
|
||||||
},
|
},
|
||||||
"repository": {
|
"repository": {
|
||||||
"type": "git",
|
"type": "git",
|
||||||
"url": "git+https://github.com/soulcraftlabs/brainy.git"
|
"url": "git+https://source.soulcraft.com/soulcraft/brainy.git"
|
||||||
},
|
},
|
||||||
"files": [
|
"files": [
|
||||||
"dist/**/*.js",
|
"dist/**/*.js",
|
||||||
|
|
|
||||||
|
|
@ -142,7 +142,7 @@ else
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Create new changelog entry
|
# Create new changelog entry
|
||||||
CHANGELOG_ENTRY="### [${NEW_VERSION}](https://github.com/soulcraftlabs/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
|
CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraft/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
|
||||||
|
|
||||||
${COMMITS}
|
${COMMITS}
|
||||||
"
|
"
|
||||||
|
|
@ -175,26 +175,59 @@ echo -e "${BLUE}7️⃣ Creating git tag v${NEW_VERSION}...${NC}"
|
||||||
git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}"
|
git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}"
|
||||||
echo -e "${GREEN}✅ Tag created${NC}\n"
|
echo -e "${GREEN}✅ Tag created${NC}\n"
|
||||||
|
|
||||||
# Step 9: Push to GitHub
|
# Step 9: Push to origin — the forge is the one home (ruled 2026-07-23; the
|
||||||
echo -e "${BLUE}8️⃣ Pushing to GitHub...${NC}"
|
# old public GitHub repo is archived history, no longer part of any release).
|
||||||
|
echo -e "${BLUE}8️⃣ Pushing to origin...${NC}"
|
||||||
git push --follow-tags origin "$CURRENT_BRANCH"
|
git push --follow-tags origin "$CURRENT_BRANCH"
|
||||||
echo -e "${GREEN}✅ Pushed to GitHub${NC}\n"
|
echo -e "${GREEN}✅ Pushed to origin${NC}\n"
|
||||||
|
|
||||||
# Step 10: Publish to npm
|
# Step 10: Publish — forge FIRST (home), npmjs second (the world's storefront).
|
||||||
echo -e "${BLUE}9️⃣ Publishing to npm (dist-tag: ${NPM_TAG})...${NC}"
|
# The fleet-wide ~/.npmrc maps the @soulcraft scope to the forge registry, and
|
||||||
npm publish --tag "$NPM_TAG"
|
# a scope mapping BEATS `--registry` on the command line — so each publish
|
||||||
# Brainy is the only PUBLIC @soulcraft package — verify visibility after every publish.
|
# names its registry via the scope override explicitly. Nothing implicit.
|
||||||
npm access get status @soulcraft/brainy || true
|
FORGE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraft/npm/"
|
||||||
echo -e "${GREEN}✅ Published to npm${NC}\n"
|
FORGE_NPM_TOKEN_FILE="$HOME/.config/soulcraft/npm-publish-brainy.token"
|
||||||
|
echo -e "${BLUE}9️⃣ Publishing to the forge registry (home)...${NC}"
|
||||||
# Step 11: Create GitHub release
|
if [ -f "$FORGE_NPM_TOKEN_FILE" ]; then
|
||||||
echo -e "${BLUE}🔟 Creating GitHub release...${NC}"
|
TMPRC="$(mktemp)"
|
||||||
if [ "$PRERELEASE" = true ]; then
|
chmod 600 "$TMPRC"
|
||||||
gh release create "v${NEW_VERSION}" --generate-notes --prerelease
|
{
|
||||||
|
echo "@soulcraft:registry=${FORGE_NPM_REG}"
|
||||||
|
echo "//source.soulcraft.com/api/packages/soulcraft/npm/:_authToken=$(cat "$FORGE_NPM_TOKEN_FILE")"
|
||||||
|
} > "$TMPRC"
|
||||||
|
if npm publish --tag "$NPM_TAG" --userconfig "$TMPRC"; then
|
||||||
|
echo -e "${GREEN}✅ Published to the forge${NC}\n"
|
||||||
|
else
|
||||||
|
rm -f "$TMPRC"
|
||||||
|
echo -e "${RED}❌ Forge publish FAILED — aborting before npmjs so the pair never diverges. Fix and re-run.${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
rm -f "$TMPRC"
|
||||||
else
|
else
|
||||||
gh release create "v${NEW_VERSION}" --generate-notes
|
echo -e "${RED}❌ Forge publish token missing (${FORGE_NPM_TOKEN_FILE}) — aborting. The forge is home; publish it first or restage the token.${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "${BLUE}9️⃣½ Publishing to npmjs (storefront, dist-tag: ${NPM_TAG})...${NC}"
|
||||||
|
npm publish --tag "$NPM_TAG" "--@soulcraft:registry=https://registry.npmjs.org/"
|
||||||
|
# Brainy is the only PUBLIC @soulcraft package — verify visibility after every publish.
|
||||||
|
npm access get status @soulcraft/brainy "--@soulcraft:registry=https://registry.npmjs.org/" || true
|
||||||
|
echo -e "${GREEN}✅ Published to npmjs${NC}\n"
|
||||||
|
|
||||||
|
# Step 11: Release object on the forge (presentational — the tag, CHANGELOG,
|
||||||
|
# and RELEASES.md are the record; this just gives the forge UI a release page).
|
||||||
|
echo -e "${BLUE}🔟 Creating forge release...${NC}"
|
||||||
|
if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then
|
||||||
|
if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \
|
||||||
|
-H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \
|
||||||
|
-d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then
|
||||||
|
echo -e "${GREEN}✅ Forge release created${NC}\n"
|
||||||
|
else
|
||||||
|
echo -e "${RED}⚠️ Forge release API call failed — tag + CHANGELOG remain the record; create the release page via the forge UI if wanted${NC}\n"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo -e "${RED}⚠️ FORGEJO_RELEASE_TOKEN unset — no release page created; tag + CHANGELOG remain the record${NC}\n"
|
||||||
fi
|
fi
|
||||||
echo -e "${GREEN}✅ GitHub release created${NC}\n"
|
|
||||||
|
|
||||||
# Step 12: Push public docs to the soulcraft.com docs ingest door
|
# Step 12: Push public docs to the soulcraft.com docs ingest door
|
||||||
# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when
|
# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when
|
||||||
|
|
@ -213,4 +246,4 @@ echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}"
|
||||||
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
echo -e "📦 npm: ${BLUE}https://www.npmjs.com/package/@soulcraft/brainy/v/${NEW_VERSION}${NC}"
|
echo -e "📦 npm: ${BLUE}https://www.npmjs.com/package/@soulcraft/brainy/v/${NEW_VERSION}${NC}"
|
||||||
echo -e "🐙 GitHub: ${BLUE}https://github.com/soulcraftlabs/brainy/releases/tag/v${NEW_VERSION}${NC}"
|
echo -e "🏠 Forge: ${BLUE}https://source.soulcraft.com/soulcraft/brainy/releases/tag/v${NEW_VERSION}${NC}"
|
||||||
|
|
|
||||||
205
src/brainy.ts
205
src/brainy.ts
|
|
@ -63,7 +63,9 @@ import type {
|
||||||
PathOptions,
|
PathOptions,
|
||||||
MetadataIndexProvider,
|
MetadataIndexProvider,
|
||||||
OpaqueIdSet,
|
OpaqueIdSet,
|
||||||
AtGenerationVectors
|
AtGenerationVectors,
|
||||||
|
VectorIndexProvider,
|
||||||
|
GraphIndexProvider
|
||||||
} from './plugin.js'
|
} from './plugin.js'
|
||||||
import type {
|
import type {
|
||||||
BrainyPlugin,
|
BrainyPlugin,
|
||||||
|
|
@ -88,12 +90,12 @@ import { findCallerLocation } from './utils/callerLocation.js'
|
||||||
import {
|
import {
|
||||||
SaveNounMetadataOperation,
|
SaveNounMetadataOperation,
|
||||||
SaveNounOperation,
|
SaveNounOperation,
|
||||||
AddToHNSWOperation,
|
AddToVectorIndexOperation,
|
||||||
AddToMetadataIndexOperation,
|
AddToMetadataIndexOperation,
|
||||||
SaveVerbMetadataOperation,
|
SaveVerbMetadataOperation,
|
||||||
SaveVerbOperation,
|
SaveVerbOperation,
|
||||||
AddToGraphIndexOperation,
|
AddToGraphIndexOperation,
|
||||||
RemoveFromHNSWOperation,
|
RemoveFromVectorIndexOperation,
|
||||||
RemoveFromMetadataIndexOperation,
|
RemoveFromMetadataIndexOperation,
|
||||||
RemoveFromGraphIndexOperation,
|
RemoveFromGraphIndexOperation,
|
||||||
UpdateNounMetadataOperation,
|
UpdateNounMetadataOperation,
|
||||||
|
|
@ -266,6 +268,7 @@ type ResolvedBrainyConfig = Required<
|
||||||
| 'retention'
|
| 'retention'
|
||||||
| 'eagerEmbeddings'
|
| 'eagerEmbeddings'
|
||||||
| 'migrationWaitTimeoutMs'
|
| 'migrationWaitTimeoutMs'
|
||||||
|
| 'transactionBudgetFloorMs'
|
||||||
>
|
>
|
||||||
> &
|
> &
|
||||||
Pick<
|
Pick<
|
||||||
|
|
@ -278,6 +281,7 @@ type ResolvedBrainyConfig = Required<
|
||||||
| 'retention'
|
| 'retention'
|
||||||
| 'eagerEmbeddings'
|
| 'eagerEmbeddings'
|
||||||
| 'migrationWaitTimeoutMs'
|
| 'migrationWaitTimeoutMs'
|
||||||
|
| 'transactionBudgetFloorMs'
|
||||||
>
|
>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -388,6 +392,38 @@ class InsertPreconditionExistsSignal extends Error {
|
||||||
*/
|
*/
|
||||||
export type IndexFamily = 'vector' | 'metadata' | 'graph'
|
export type IndexFamily = 'vector' | 'metadata' | 'graph'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Honest per-surface outcome for {@link Brainy.warm}. Literal
|
||||||
|
* meanings — never conflate the first two:
|
||||||
|
* - `'warmed'` — the provider's own `warm?()` hook ran (vector/graph), or the
|
||||||
|
* surface's full-hydration seam loaded EVERY shard/field/segment from
|
||||||
|
* storage (metadata; graph's fallback path). The surface is genuinely at
|
||||||
|
* steady-state cost for the next operation.
|
||||||
|
* - `'probed'` — no `warm?()` hook was available, so a best-effort read
|
||||||
|
* (e.g. one `search()` call) faulted in *some* backing storage as a side
|
||||||
|
* effect. Real work happened, but it is NOT the same guarantee as
|
||||||
|
* `'warmed'` — never reported as `'warmed'`.
|
||||||
|
* - `'unavailable'` — nothing ran: no hook, no hydration seam, and (for the
|
||||||
|
* vector probe fallback) nothing to probe (an empty index or unknown
|
||||||
|
* vector dimension). The surface is unchanged by this `warm()` call.
|
||||||
|
*/
|
||||||
|
export type WarmOutcome = 'warmed' | 'probed' | 'unavailable'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Result of {@link Brainy.warm}: one {@link WarmOutcome} +
|
||||||
|
* elapsed time per index surface, plus the total wall-clock time for the
|
||||||
|
* whole call. `durationMs` is measured around exactly the work described by
|
||||||
|
* that surface's `outcome` (e.g. the vector entry's `durationMs` times the
|
||||||
|
* provider `warm()` call OR the probe `search()` call — whichever ran).
|
||||||
|
*/
|
||||||
|
export interface WarmReport {
|
||||||
|
vector: { outcome: WarmOutcome; durationMs: number }
|
||||||
|
metadata: { outcome: WarmOutcome; durationMs: number }
|
||||||
|
graph: { outcome: WarmOutcome; durationMs: number }
|
||||||
|
/** Total wall-clock time for the whole `warm()` call (all three surfaces). */
|
||||||
|
totalDurationMs: number
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
|
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
|
||||||
* Within the window, queries rethrow the recorded failure instantly (loud,
|
* Within the window, queries rethrow the recorded failure instantly (loud,
|
||||||
|
|
@ -1382,6 +1418,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Eager index warm (operator opt-in, `warmOnOpen: true`). Runs AFTER
|
||||||
|
// every step above — index construction, crash recovery, migrations,
|
||||||
|
// VFS bootstrap — never in place of any of it, so `warm()` always
|
||||||
|
// operates on a fully-initialized brain. Blocking BY DESIGN: the
|
||||||
|
// operator traded a longer startup for a first-request that runs at
|
||||||
|
// steady-state cost instead of paying demand-load latency on the
|
||||||
|
// critical path. See the `warmOnOpen` JSDoc in brainy.types.ts.
|
||||||
|
if (this.config.warmOnOpen) {
|
||||||
|
await this.warm()
|
||||||
|
}
|
||||||
|
|
||||||
// Resolve ready Promise - consumers awaiting brain.ready will now proceed
|
// Resolve ready Promise - consumers awaiting brain.ready will now proceed
|
||||||
if (this._readyResolve) {
|
if (this._readyResolve) {
|
||||||
this._readyResolve()
|
this._readyResolve()
|
||||||
|
|
@ -1780,7 +1827,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
await this.generationStore.runWithoutGeneration(() =>
|
await this.generationStore.runWithoutGeneration(() =>
|
||||||
this.transactionManager.executeTransaction(run, {
|
this.transactionManager.executeTransaction(run, {
|
||||||
timeout: transactTimeoutBudget(
|
timeout: transactTimeoutBudget(
|
||||||
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
|
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
|
||||||
|
undefined,
|
||||||
|
this.config.transactionBudgetFloorMs
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
)
|
)
|
||||||
|
|
@ -1797,7 +1846,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
execute: () =>
|
execute: () =>
|
||||||
this.transactionManager.executeTransaction(run, {
|
this.transactionManager.executeTransaction(run, {
|
||||||
timeout: transactTimeoutBudget(
|
timeout: transactTimeoutBudget(
|
||||||
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
|
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
|
||||||
|
undefined,
|
||||||
|
this.config.transactionBudgetFloorMs
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
@ -2109,7 +2160,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
|
|
||||||
// Operation 3: Add to HNSW index (after entity saved)
|
// Operation 3: Add to HNSW index (after entity saved)
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new AddToHNSWOperation(this.index, id, vector)
|
new AddToVectorIndexOperation(this.index, id, vector)
|
||||||
)
|
)
|
||||||
|
|
||||||
// Operation 4: Add to metadata index
|
// Operation 4: Add to metadata index
|
||||||
|
|
@ -3098,10 +3149,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Operation 3-4: Update HNSW index (remove and re-add if reindexing needed)
|
// Operation 3-4: Update HNSW index (remove and re-add if reindexing needed)
|
||||||
if (needsReindexing) {
|
if (needsReindexing) {
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromHNSWOperation(this.index, params.id, existing.vector)
|
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector)
|
||||||
)
|
)
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new AddToHNSWOperation(this.index, params.id, vector)
|
new AddToVectorIndexOperation(this.index, params.id, vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -3217,7 +3268,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Operation 1: Remove from vector index
|
// Operation 1: Remove from vector index
|
||||||
if (noun) {
|
if (noun) {
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromHNSWOperation(this.index, id, noun.vector)
|
new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -7025,7 +7076,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Add delete operations to transaction
|
// Add delete operations to transaction
|
||||||
if (noun) {
|
if (noun) {
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromHNSWOperation(this.index, id, noun.vector)
|
new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -7888,7 +7939,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Budget scales with the batch (or the caller's explicit
|
// Budget scales with the batch (or the caller's explicit
|
||||||
// timeoutMs): a flat 30s cap silently limited honest bulk work
|
// timeoutMs): a flat 30s cap silently limited honest bulk work
|
||||||
// to ~15 ops on network disks (~2s/op measured in the field).
|
// to ~15 ops on network disks (~2s/op measured in the field).
|
||||||
{ timeout: transactTimeoutBudget(plan.operations.length, options?.timeoutMs) }
|
{
|
||||||
|
timeout: transactTimeoutBudget(
|
||||||
|
plan.operations.length,
|
||||||
|
options?.timeoutMs,
|
||||||
|
this.config.transactionBudgetFloorMs
|
||||||
|
)
|
||||||
|
}
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
}))
|
}))
|
||||||
|
|
@ -9311,7 +9368,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
|
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
|
||||||
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
|
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
|
||||||
new AddToHNSWOperation(this.index, id, vector),
|
new AddToVectorIndexOperation(this.index, id, vector),
|
||||||
new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing)
|
new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing)
|
||||||
)
|
)
|
||||||
plan.touchedNouns.push(id)
|
plan.touchedNouns.push(id)
|
||||||
|
|
@ -9479,8 +9536,8 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
)
|
)
|
||||||
if (needsReindexing) {
|
if (needsReindexing) {
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
new RemoveFromHNSWOperation(this.index, params.id, existing.vector),
|
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector),
|
||||||
new AddToHNSWOperation(this.index, params.id, vector)
|
new AddToVectorIndexOperation(this.index, params.id, vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
|
|
@ -9565,7 +9622,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
|
|
||||||
if (noun) {
|
if (noun) {
|
||||||
plan.operations.push(new RemoveFromHNSWOperation(this.index, id, noun.vector))
|
plan.operations.push(new RemoveFromVectorIndexOperation(this.index, id, noun.vector))
|
||||||
}
|
}
|
||||||
if (metadata) {
|
if (metadata) {
|
||||||
plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata))
|
plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata))
|
||||||
|
|
@ -14280,6 +14337,118 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return this.embedder(textToEmbed)
|
return this.embedder(textToEmbed)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Eagerly load/fault-in the vector index, metadata index, and graph
|
||||||
|
* adjacency so the FIRST real operation after this call runs at
|
||||||
|
* steady-state cost — no page-cache-miss / demand-load latency on the
|
||||||
|
* critical path. Complements {@link warmupEmbeddings} (which warms the
|
||||||
|
* embedding engine, not the storage indexes).
|
||||||
|
*
|
||||||
|
* Sequence per surface:
|
||||||
|
* - **Vector**: calls the provider's own `warm?()` when the active
|
||||||
|
* `'vector'` provider implements it (`'warmed'`). Otherwise runs a
|
||||||
|
* best-effort probe — one `search()` with a deterministic unit vector of
|
||||||
|
* the index's known dimension, `k = min(100, size())` — which faults in
|
||||||
|
* *some* backing storage as a side effect but is reported honestly as
|
||||||
|
* `'probed'`, never `'warmed'`. An empty index or unknown dimension has
|
||||||
|
* nothing to probe (`'unavailable'`).
|
||||||
|
* - **Metadata**: full hydration — every persisted field's sparse index is
|
||||||
|
* loaded from storage (`MetadataIndexManager.hydrateAll()`), not just the
|
||||||
|
* heuristic common-fields subset `init()` warms.
|
||||||
|
* - **Graph**: calls the provider's own `warm?()` when the active graph
|
||||||
|
* provider implements it; otherwise re-runs its existing eager cold-load
|
||||||
|
* `init()` seam (idempotent — the JS adjacency index's `init()` already
|
||||||
|
* loads the full LSM manifests + SSTables, so re-invoking it here is a
|
||||||
|
* genuine full hydration, not a stat call).
|
||||||
|
*
|
||||||
|
* A single surface's `'unavailable'`/probe outcome never fails the whole
|
||||||
|
* call — `warm()` is a best-effort readiness step, not a correctness gate;
|
||||||
|
* queries still demand-load normally regardless of what `warm()` achieved.
|
||||||
|
*
|
||||||
|
* @returns A {@link WarmReport}: honest per-surface outcome + timing.
|
||||||
|
* @example
|
||||||
|
* ```typescript
|
||||||
|
* const brain = new Brainy({ storage: { path: '/data' } })
|
||||||
|
* await brain.init()
|
||||||
|
* const report = await brain.warm() // blocking, explicit control over timing
|
||||||
|
* console.log(report.vector.outcome, report.vector.durationMs)
|
||||||
|
*
|
||||||
|
* // Or opt into the same work automatically during init():
|
||||||
|
* const eager = new Brainy({ storage: { path: '/data' }, warmOnOpen: true })
|
||||||
|
* await eager.init() // warm() already ran before this resolves
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
async warm(): Promise<WarmReport> {
|
||||||
|
await this.ensureInitialized({ needs: ['vector', 'metadata', 'graph'] })
|
||||||
|
|
||||||
|
const totalStart = Date.now()
|
||||||
|
|
||||||
|
// --- Vector ---------------------------------------------------------
|
||||||
|
const vectorStart = Date.now()
|
||||||
|
let vectorOutcome: WarmOutcome
|
||||||
|
const vectorProvider = this.index as VectorIndexProvider & { warm?: () => Promise<void> }
|
||||||
|
if (typeof vectorProvider.warm === 'function') {
|
||||||
|
await vectorProvider.warm()
|
||||||
|
vectorOutcome = 'warmed'
|
||||||
|
} else {
|
||||||
|
const size = this.index.size()
|
||||||
|
const dimension = this.dimensions
|
||||||
|
if (size > 0 && dimension && dimension > 0) {
|
||||||
|
// A deterministic unit vector (L2 norm 1) — same probe every call,
|
||||||
|
// no reliance on stored data shape.
|
||||||
|
const probeVector = new Array(dimension).fill(1 / Math.sqrt(dimension))
|
||||||
|
await this.index.search(probeVector, Math.min(100, size))
|
||||||
|
vectorOutcome = 'probed'
|
||||||
|
} else {
|
||||||
|
// Nothing to probe: an empty index, or the vector dimension is not
|
||||||
|
// yet known (no entity has ever been added).
|
||||||
|
vectorOutcome = 'unavailable'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const vectorDurationMs = Date.now() - vectorStart
|
||||||
|
|
||||||
|
// --- Metadata --------------------------------------------------------
|
||||||
|
const metadataStart = Date.now()
|
||||||
|
let metadataOutcome: WarmOutcome
|
||||||
|
const metadataWithHydrate = this.metadataIndex as unknown as { hydrateAll?: () => Promise<void> }
|
||||||
|
if (typeof metadataWithHydrate.hydrateAll === 'function') {
|
||||||
|
await metadataWithHydrate.hydrateAll()
|
||||||
|
metadataOutcome = 'warmed'
|
||||||
|
} else {
|
||||||
|
// No hydration seam on this metadata provider — nothing to run.
|
||||||
|
metadataOutcome = 'unavailable'
|
||||||
|
}
|
||||||
|
const metadataDurationMs = Date.now() - metadataStart
|
||||||
|
|
||||||
|
// --- Graph -------------------------------------------------------------
|
||||||
|
const graphStart = Date.now()
|
||||||
|
let graphOutcome: WarmOutcome
|
||||||
|
const graphProvider = this.graphIndex as GraphIndexProvider & {
|
||||||
|
warm?: () => Promise<void>
|
||||||
|
init?: () => Promise<void>
|
||||||
|
}
|
||||||
|
if (typeof graphProvider.warm === 'function') {
|
||||||
|
await graphProvider.warm()
|
||||||
|
graphOutcome = 'warmed'
|
||||||
|
} else if (typeof graphProvider.init === 'function') {
|
||||||
|
// Existing eager cold-load seam (readiness contract): idempotent, and
|
||||||
|
// for the JS adjacency index it's already a genuine full load of the
|
||||||
|
// LSM manifests + SSTables — a real hydration, not a stat call.
|
||||||
|
await graphProvider.init()
|
||||||
|
graphOutcome = 'warmed'
|
||||||
|
} else {
|
||||||
|
graphOutcome = 'unavailable'
|
||||||
|
}
|
||||||
|
const graphDurationMs = Date.now() - graphStart
|
||||||
|
|
||||||
|
return {
|
||||||
|
vector: { outcome: vectorOutcome, durationMs: vectorDurationMs },
|
||||||
|
metadata: { outcome: metadataOutcome, durationMs: metadataDurationMs },
|
||||||
|
graph: { outcome: graphOutcome, durationMs: graphDurationMs },
|
||||||
|
totalDurationMs: Date.now() - totalStart
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Explicitly warm up the embedding engine
|
* Explicitly warm up the embedding engine
|
||||||
*
|
*
|
||||||
|
|
@ -14798,6 +14967,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Migration LOCK wait budget — left undefined when omitted so
|
// Migration LOCK wait budget — left undefined when omitted so
|
||||||
// awaitMigrationLock() applies its 30 s default at the read site.
|
// awaitMigrationLock() applies its 30 s default at the read site.
|
||||||
migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined,
|
migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined,
|
||||||
|
// Apply-phase transaction budget floor — left undefined when omitted so
|
||||||
|
// transactTimeoutBudget() applies its 30 s default at the read site.
|
||||||
|
transactionBudgetFloorMs: config?.transactionBudgetFloorMs ?? undefined,
|
||||||
// Pre-upgrade backup — default-on; opt out with `migrationBackup: false`.
|
// Pre-upgrade backup — default-on; opt out with `migrationBackup: false`.
|
||||||
migrationBackup: config?.migrationBackup ?? true,
|
migrationBackup: config?.migrationBackup ?? true,
|
||||||
// Vector index configuration (8.0) — algorithm-neutral surface with
|
// Vector index configuration (8.0) — algorithm-neutral surface with
|
||||||
|
|
@ -14813,6 +14985,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// is the active one on a non-reader writer outside unit tests). Explicit
|
// is the active one on a non-reader writer outside unit tests). Explicit
|
||||||
// true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts.
|
// true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts.
|
||||||
eagerEmbeddings: config?.eagerEmbeddings ?? undefined,
|
eagerEmbeddings: config?.eagerEmbeddings ?? undefined,
|
||||||
|
// Eager index warm at init() — default off (lazy demand-load), the
|
||||||
|
// pre-8.9 behavior. See the warmOnOpen JSDoc in brainy.types.ts.
|
||||||
|
warmOnOpen: config?.warmOnOpen ?? false,
|
||||||
// Plugin configuration - undefined = auto-detect
|
// Plugin configuration - undefined = auto-detect
|
||||||
plugins: config?.plugins ?? undefined,
|
plugins: config?.plugins ?? undefined,
|
||||||
// Integration Hub - undefined/false = disabled
|
// Integration Hub - undefined/false = disabled
|
||||||
|
|
|
||||||
|
|
@ -54,6 +54,9 @@ export class HnswFlushError extends Error {
|
||||||
* acceleration provider).
|
* acceleration provider).
|
||||||
*/
|
*/
|
||||||
export class JsHnswVectorIndex implements VectorIndexProvider {
|
export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
|
/** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.name}. */
|
||||||
|
readonly name = 'hnsw-js'
|
||||||
|
|
||||||
private nouns: Map<string, HNSWNoun> = new Map()
|
private nouns: Map<string, HNSWNoun> = new Map()
|
||||||
/**
|
/**
|
||||||
* Reverse adjacency: `target id → (level → set of node ids that link TO target)`.
|
* Reverse adjacency: `target id → (level → set of node ids that link TO target)`.
|
||||||
|
|
|
||||||
|
|
@ -28,6 +28,9 @@ export type { FileVersion } from './vfs/types.js'
|
||||||
|
|
||||||
// Export diagnostics result type
|
// Export diagnostics result type
|
||||||
export type { DiagnosticsResult } from './brainy.js'
|
export type { DiagnosticsResult } from './brainy.js'
|
||||||
|
// brain.warm() — eager index/storage readiness report (per-surface honest
|
||||||
|
// outcome + timing). See the WarmReport JSDoc in brainy.ts.
|
||||||
|
export type { WarmReport, WarmOutcome } from './brainy.js'
|
||||||
export type {
|
export type {
|
||||||
GraphAuditReport,
|
GraphAuditReport,
|
||||||
GraphAuditDiscrepancy
|
GraphAuditDiscrepancy
|
||||||
|
|
|
||||||
|
|
@ -381,6 +381,20 @@ export interface GraphIndexProvider {
|
||||||
*/
|
*/
|
||||||
init?(): Promise<void>
|
init?(): Promise<void>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
|
||||||
|
* pretouch) so first operations run at steady-state cost. Optional;
|
||||||
|
* absence means the provider demand-loads. Distinct from `init?()`: `init`
|
||||||
|
* is called once automatically during brain startup as part of the
|
||||||
|
* readiness contract (cold-load + rebuild gating); `warm` is an explicit,
|
||||||
|
* separate readiness step a caller opts into via `brain.warm()` (or
|
||||||
|
* `warmOnOpen`) specifically to pre-pay demand-load cost that `init` left
|
||||||
|
* lazy. A provider that already loads everything eagerly in `init?()` may
|
||||||
|
* implement `warm` as a no-op or omit it — `brain.warm()` falls back to a
|
||||||
|
* best-effort read-through probe when absent.
|
||||||
|
*/
|
||||||
|
warm?(): Promise<void>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description OPTIONAL. A native provider returns true from the moment its
|
* @description OPTIONAL. A native provider returns true from the moment its
|
||||||
* `init()` detects a large epoch-drift until its background
|
* `init()` detects a large epoch-drift until its background
|
||||||
|
|
@ -947,6 +961,26 @@ export interface AtGenerationVectors {
|
||||||
* are retired and never looked up.
|
* are retired and never looked up.
|
||||||
*/
|
*/
|
||||||
export interface VectorIndexProvider {
|
export interface VectorIndexProvider {
|
||||||
|
/**
|
||||||
|
* @description REQUIRED self-reported implementation identity, rendered as
|
||||||
|
* `[vector-index:<name>]` in prose log lines and stamped into the
|
||||||
|
* transaction op-name strings that surface in journals/timings (e.g.
|
||||||
|
* `AddToVectorIndex(hnsw-js)`) — see
|
||||||
|
* `src/transaction/operations/IndexOperations.ts`. A provider must name
|
||||||
|
* itself TRUTHFULLY (its own algorithm/engine, e.g. its plugin/package
|
||||||
|
* name) and must never inherit a default — guessing a backend name would
|
||||||
|
* resurrect the exact bug this field exists to prevent: an operator
|
||||||
|
* hunting an index that isn't the one actually running. The built-in JS
|
||||||
|
* index always sets this to `'hnsw-js'`; a native provider picks its own
|
||||||
|
* string. At the TypeScript level this field is required; a runtime
|
||||||
|
* instance from an older provider compiled against the previous optional
|
||||||
|
* `providerId` contract is tolerated (never crashes, never silently
|
||||||
|
* mislabeled) — see `resolveVectorProviderId` in
|
||||||
|
* `src/transaction/operations/IndexOperations.ts`, which stamps
|
||||||
|
* `'unknown-provider'` and emits one loud warning for that case.
|
||||||
|
*/
|
||||||
|
readonly name: string
|
||||||
|
|
||||||
addItem(item: VectorDocument): Promise<string>
|
addItem(item: VectorDocument): Promise<string>
|
||||||
removeItem(id: string): Promise<boolean>
|
removeItem(id: string): Promise<boolean>
|
||||||
search(
|
search(
|
||||||
|
|
@ -1009,6 +1043,20 @@ export interface VectorIndexProvider {
|
||||||
*/
|
*/
|
||||||
init?(): Promise<void>
|
init?(): Promise<void>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
|
||||||
|
* pretouch) so first operations run at steady-state cost. Optional;
|
||||||
|
* absence means the provider demand-loads. Distinct from `init?()`: `init`
|
||||||
|
* runs automatically once during brain startup (the cold-load + rebuild
|
||||||
|
* readiness contract above); `warm` is a separate, explicit step a caller
|
||||||
|
* opts into via `brain.warm()` (or `warmOnOpen`) to pre-pay demand-load
|
||||||
|
* cost `init` left lazy — e.g. touching every mmap page rather than just
|
||||||
|
* opening the file. A provider that already loads everything eagerly in
|
||||||
|
* `init?()` may implement `warm` as a no-op or omit it — `brain.warm()`
|
||||||
|
* falls back to a best-effort probe `search()` when absent.
|
||||||
|
*/
|
||||||
|
warm?(): Promise<void>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description OPTIONAL honest durability signal (readiness contract,
|
* @description OPTIONAL honest durability signal (readiness contract,
|
||||||
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
|
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
|
||||||
|
|
|
||||||
|
|
@ -37,22 +37,47 @@ const DEFAULT_OPTIONS: Required<TransactionOptions> = {
|
||||||
maxRollbackRetries: 3
|
maxRollbackRetries: 3
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The floor term of {@link transactTimeoutBudget}'s scaling formula when the
|
||||||
|
* caller supplies neither an explicit override nor `config.transactionBudgetFloorMs`.
|
||||||
|
*/
|
||||||
|
const DEFAULT_BUDGET_FLOOR_MS = 30_000
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The apply-phase budget for a batch of `opCount` operations.
|
* The apply-phase budget for a batch of `opCount` operations.
|
||||||
*
|
*
|
||||||
* An explicit override wins untouched. Otherwise the budget SCALES with the
|
* An explicit `override` wins untouched (a caller-specified deadline for this
|
||||||
* batch: `max(30 000 ms, opCount × 2 000 ms)`. The per-op term is calibrated
|
* one batch). Otherwise the budget SCALES with the batch:
|
||||||
* from field data — bulk imports on network-attached disks measure ~2 s per
|
* `max(floorMs, opCount × 2 000)`, where `floorMs` defaults to `30 000` and
|
||||||
* operation (each op pays canonical writes + fsync + index maintenance) — so
|
* is overridable via `BrainyConfig.transactionBudgetFloorMs` for the whole
|
||||||
* a flat 30 s budget silently capped honest work at ~15 operations while
|
* brain. The per-op term is calibrated from field data — bulk imports on
|
||||||
* looking generous for small batches. Scaling keeps small transacts
|
* network-attached disks measure ~2 s per operation (each op pays canonical
|
||||||
* fast-failing and gives bulk ones a budget proportional to the work they
|
* writes + fsync + index maintenance) — so a flat 30 s budget silently capped
|
||||||
* actually asked for; a trip still rolls back atomically and throws a
|
* honest work at ~15 operations while looking generous for small batches. A
|
||||||
* retryable, fully-labeled TransactionTimeoutError.
|
* 16-op batch, for example, scales to `max(30 000, 16 × 2 000) = 32 000`.
|
||||||
|
* Scaling keeps small transacts fast-failing and gives bulk ones a budget
|
||||||
|
* proportional to the work they actually asked for.
|
||||||
|
*
|
||||||
|
* This is a **mid-batch** budget only: it governs whether the transaction's
|
||||||
|
* NEXT operation may start (see {@link Transaction.execute}), never whether
|
||||||
|
* already-completed work is rolled back after the fact. A trip mid-batch
|
||||||
|
* still rolls back every operation applied so far, atomically, and throws a
|
||||||
|
* retryable, fully-labeled TransactionTimeoutError — that zero-loss guarantee
|
||||||
|
* doesn't change; only the point at which the clock stops mattering does (at
|
||||||
|
* the last operation, not one check later).
|
||||||
|
*
|
||||||
|
* @param opCount - Number of operations in the batch.
|
||||||
|
* @param override - A full override for this call; wins over everything else.
|
||||||
|
* @param floorMs - The scaling floor for this call (e.g. `config.transactionBudgetFloorMs`).
|
||||||
|
* Ignored when `override` is set. Defaults to `30 000` when omitted.
|
||||||
*/
|
*/
|
||||||
export function transactTimeoutBudget(opCount: number, override?: number): number {
|
export function transactTimeoutBudget(
|
||||||
|
opCount: number,
|
||||||
|
override?: number,
|
||||||
|
floorMs?: number
|
||||||
|
): number {
|
||||||
if (override !== undefined) return override
|
if (override !== undefined) return override
|
||||||
return Math.max(30_000, opCount * 2_000)
|
return Math.max(floorMs ?? DEFAULT_BUDGET_FLOOR_MS, opCount * 2_000)
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -98,7 +123,20 @@ export class Transaction implements TransactionContext {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Execute all operations atomically
|
* Execute all operations atomically.
|
||||||
|
*
|
||||||
|
* Budget semantics: the configured budget gates starting the next
|
||||||
|
* operation; completed work is never rolled back for elapsed time. Elapsed
|
||||||
|
* time is checked ONLY before starting operation `i` for `i ≥ 1` — never
|
||||||
|
* before operation 0 (an operation always gets to start) and never again
|
||||||
|
* after the final operation completes. Concretely: a single-op transaction
|
||||||
|
* can never time out post-hoc — it either runs (and its result stands, no
|
||||||
|
* matter how long it took) or a `TransactionTimeoutError` is thrown before
|
||||||
|
* it starts, which cannot happen since there is no operation before it. A
|
||||||
|
* multi-op transaction whose op `i` overruns the budget stops op `i+1` from
|
||||||
|
* starting, rolls back operations `0..i` (reverse order), and throws
|
||||||
|
* `TransactionTimeoutError` — that zero-loss rollback guarantee for
|
||||||
|
* mid-batch trips is unchanged; only the post-completion check is gone.
|
||||||
*/
|
*/
|
||||||
async execute(): Promise<void> {
|
async execute(): Promise<void> {
|
||||||
if (this.state !== 'pending') {
|
if (this.state !== 'pending') {
|
||||||
|
|
@ -128,10 +166,14 @@ export class Transaction implements TransactionContext {
|
||||||
// already-applied writes as torn, generation-less state in canonical
|
// already-applied writes as torn, generation-less state in canonical
|
||||||
// storage.
|
// storage.
|
||||||
for (let i = 0; i < this.operations.length; i++) {
|
for (let i = 0; i < this.operations.length; i++) {
|
||||||
// Budget check BEFORE starting the next operation. A trip here throws
|
// Budget gates STARTING the next operation; completed work is never
|
||||||
// into the catch below and rolls back like any other failure — it must
|
// rolled back for elapsed time. Skipped for i === 0 (operation 0
|
||||||
// never bypass rollback.
|
// always gets to start — nothing has run yet to have overrun) and
|
||||||
if (Date.now() - this.startTime > this.options.timeout) {
|
// never re-checked after the final operation completes (there is no
|
||||||
|
// "next operation" left to gate). A trip here throws into the catch
|
||||||
|
// below and rolls back like any other failure — it must never bypass
|
||||||
|
// rollback.
|
||||||
|
if (i > 0 && Date.now() - this.startTime > this.options.timeout) {
|
||||||
throw new TransactionTimeoutError(this.options.timeout, i, {
|
throw new TransactionTimeoutError(this.options.timeout, i, {
|
||||||
elapsedMs: Date.now() - this.startTime,
|
elapsedMs: Date.now() - this.startTime,
|
||||||
totalOperations: this.operations.length,
|
totalOperations: this.operations.length,
|
||||||
|
|
|
||||||
|
|
@ -2,34 +2,73 @@
|
||||||
* Index Operations with Rollback Support
|
* Index Operations with Rollback Support
|
||||||
*
|
*
|
||||||
* Provides transactional operations for all indexes:
|
* Provides transactional operations for all indexes:
|
||||||
* - JsHnswVectorIndex (unified vector index)
|
* - VectorIndexProvider (the JS HNSW fallback, or a native acceleration provider)
|
||||||
* - MetadataIndexManager (roaring bitmap filtering)
|
* - MetadataIndexManager (roaring bitmap filtering)
|
||||||
* - GraphAdjacencyIndex (LSM-tree graph storage)
|
* - GraphAdjacencyIndex (LSM-tree graph storage)
|
||||||
*
|
*
|
||||||
* Each operation can be executed and rolled back atomically.
|
* Each operation can be executed and rolled back atomically.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { JsHnswVectorIndex } from '../../hnsw/hnswIndex.js'
|
import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js'
|
||||||
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
|
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
|
||||||
import type { GraphIndexProvider } from '../../plugin.js'
|
|
||||||
import type { GraphVerb } from '../../coreTypes.js'
|
import type { GraphVerb } from '../../coreTypes.js'
|
||||||
import type { Operation, RollbackAction } from '../types.js'
|
import type { Operation, RollbackAction } from '../types.js'
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Add to HNSW index with rollback support
|
* Backend identity stamped into an operation's emitted `name` string (e.g.
|
||||||
|
* `AddToVectorIndex(hnsw-js)`), resolved from the provider's own
|
||||||
|
* {@link VectorIndexProvider.name} self-report.
|
||||||
|
*
|
||||||
|
* These operation classes are backend-neutral (the vector index they wrap may
|
||||||
|
* be Brainy's own JS HNSW fallback OR a native acceleration provider), but
|
||||||
|
* their names surface directly in consumer-visible transaction journals and
|
||||||
|
* timings. `name` is REQUIRED at the TypeScript level — but a native provider
|
||||||
|
* instance compiled against the previous (pre-required) contract can still
|
||||||
|
* reach this function at runtime without it. That legacy case is tolerated,
|
||||||
|
* never crashed on and never silently mislabeled: it resolves to
|
||||||
|
* `'unknown-provider'` and emits ONE loud warning naming the missing contract
|
||||||
|
* field, so the fix (implement `name`) is discoverable rather than a silent
|
||||||
|
* fossil label in every journal line thereafter.
|
||||||
|
*/
|
||||||
|
const warnedMissingName = new WeakSet<VectorIndexProvider>()
|
||||||
|
|
||||||
|
function resolveVectorProviderId(index: VectorIndexProvider): string {
|
||||||
|
const name = (index as { name?: unknown }).name
|
||||||
|
if (typeof name === 'string') return name
|
||||||
|
if (!warnedMissingName.has(index)) {
|
||||||
|
warnedMissingName.add(index)
|
||||||
|
console.warn(
|
||||||
|
'[vector-index] provider is missing the required `name` field (VectorIndexProvider.name, ' +
|
||||||
|
'required since 8.10.0) — stamping "unknown-provider" in transaction op names until the ' +
|
||||||
|
'provider declares its own identity.'
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return 'unknown-provider'
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Add to the vector index with rollback support.
|
||||||
|
*
|
||||||
|
* Backend-neutral: `index` is whatever the `'vector'` provider factory
|
||||||
|
* returns — Brainy's own JS HNSW fallback, or a native acceleration provider
|
||||||
|
* (e.g. DiskANN). The emitted `name` stamps the active backend (see
|
||||||
|
* {@link resolveVectorProviderId}) so operators reading a transaction journal
|
||||||
|
* or timing trace see which engine actually ran, never a fossil name from
|
||||||
|
* whichever engine happened to be active when this op class was written.
|
||||||
*
|
*
|
||||||
* Rollback strategy:
|
* Rollback strategy:
|
||||||
* - Remove item from index
|
* - Remove item from index
|
||||||
|
|
||||||
*/
|
*/
|
||||||
export class AddToHNSWOperation implements Operation {
|
export class AddToVectorIndexOperation implements Operation {
|
||||||
readonly name = 'AddToHNSW'
|
readonly name: string
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private readonly index: JsHnswVectorIndex,
|
private readonly index: VectorIndexProvider,
|
||||||
private readonly id: string,
|
private readonly id: string,
|
||||||
private readonly vector: number[]
|
private readonly vector: number[]
|
||||||
) {}
|
) {
|
||||||
|
this.name = `AddToVectorIndex(${resolveVectorProviderId(index)})`
|
||||||
|
}
|
||||||
|
|
||||||
async execute(): Promise<RollbackAction> {
|
async execute(): Promise<RollbackAction> {
|
||||||
// Check if item already exists (for rollback decision)
|
// Check if item already exists (for rollback decision)
|
||||||
|
|
@ -59,11 +98,12 @@ export class AddToHNSWOperation implements Operation {
|
||||||
* pre-existence as "existed" made every rollback skip removeItem, leaving
|
* pre-existence as "existed" made every rollback skip removeItem, leaving
|
||||||
* phantom entries in the index after a failed transaction. The safe default
|
* phantom entries in the index after a failed transaction. The safe default
|
||||||
* is to remove what this operation added — update flows pair this op with a
|
* is to remove what this operation added — update flows pair this op with a
|
||||||
* RemoveFromHNSWOperation whose own rollback restores the prior vector, so
|
* RemoveFromVectorIndexOperation whose own rollback restores the prior
|
||||||
* reverse-order rollback reconstructs the original state either way.
|
* vector, so reverse-order rollback reconstructs the original state either
|
||||||
|
* way.
|
||||||
*/
|
*/
|
||||||
private async itemExists(id: string): Promise<boolean> {
|
private async itemExists(id: string): Promise<boolean> {
|
||||||
const index = this.index as JsHnswVectorIndex & {
|
const index = this.index as VectorIndexProvider & {
|
||||||
getItem?: (id: string) => Promise<unknown>
|
getItem?: (id: string) => Promise<unknown>
|
||||||
}
|
}
|
||||||
if (typeof index.getItem !== 'function') return false
|
if (typeof index.getItem !== 'function') return false
|
||||||
|
|
@ -77,21 +117,27 @@ export class AddToHNSWOperation implements Operation {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Remove from HNSW index with rollback support
|
* Remove from the vector index with rollback support.
|
||||||
|
*
|
||||||
|
* Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the
|
||||||
|
* JS HNSW fallback or a native acceleration provider; the emitted `name`
|
||||||
|
* stamps the active backend.
|
||||||
*
|
*
|
||||||
* Rollback strategy:
|
* Rollback strategy:
|
||||||
* - Re-add item to index with original vector
|
* - Re-add item to index with original vector
|
||||||
*
|
*
|
||||||
* Note: Requires storing the vector for rollback
|
* Note: Requires storing the vector for rollback
|
||||||
*/
|
*/
|
||||||
export class RemoveFromHNSWOperation implements Operation {
|
export class RemoveFromVectorIndexOperation implements Operation {
|
||||||
readonly name = 'RemoveFromHNSW'
|
readonly name: string
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private readonly index: JsHnswVectorIndex,
|
private readonly index: VectorIndexProvider,
|
||||||
private readonly id: string,
|
private readonly id: string,
|
||||||
private readonly vector: number[] // Required for rollback
|
private readonly vector: number[] // Required for rollback
|
||||||
) {}
|
) {
|
||||||
|
this.name = `RemoveFromVectorIndex(${resolveVectorProviderId(index)})`
|
||||||
|
}
|
||||||
|
|
||||||
async execute(): Promise<RollbackAction> {
|
async execute(): Promise<RollbackAction> {
|
||||||
// Remove from index
|
// Remove from index
|
||||||
|
|
@ -285,22 +331,24 @@ export class RemoveFromGraphIndexOperation implements Operation {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Batch operation: Add multiple items to HNSW index
|
* Batch operation: Add multiple items to the vector index (backend-neutral —
|
||||||
|
* see {@link AddToVectorIndexOperation}).
|
||||||
*
|
*
|
||||||
* Useful for bulk imports with transaction support.
|
* Useful for bulk imports with transaction support.
|
||||||
* Rolls back all items if any fail.
|
* Rolls back all items if any fail.
|
||||||
*/
|
*/
|
||||||
export class BatchAddToHNSWOperation implements Operation {
|
export class BatchAddToVectorIndexOperation implements Operation {
|
||||||
readonly name = 'BatchAddToHNSW'
|
readonly name: string
|
||||||
|
|
||||||
private operations: AddToHNSWOperation[]
|
private operations: AddToVectorIndexOperation[]
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
index: JsHnswVectorIndex,
|
index: VectorIndexProvider,
|
||||||
items: Array<{ id: string; vector: number[] }>
|
items: Array<{ id: string; vector: number[] }>
|
||||||
) {
|
) {
|
||||||
|
this.name = `BatchAddToVectorIndex(${resolveVectorProviderId(index)})`
|
||||||
this.operations = items.map(
|
this.operations = items.map(
|
||||||
item => new AddToHNSWOperation(index, item.id, item.vector)
|
item => new AddToVectorIndexOperation(index, item.id, item.vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -21,12 +21,12 @@ export {
|
||||||
|
|
||||||
// Index Operations
|
// Index Operations
|
||||||
export {
|
export {
|
||||||
AddToHNSWOperation,
|
AddToVectorIndexOperation,
|
||||||
RemoveFromHNSWOperation,
|
RemoveFromVectorIndexOperation,
|
||||||
AddToMetadataIndexOperation,
|
AddToMetadataIndexOperation,
|
||||||
RemoveFromMetadataIndexOperation,
|
RemoveFromMetadataIndexOperation,
|
||||||
AddToGraphIndexOperation,
|
AddToGraphIndexOperation,
|
||||||
RemoveFromGraphIndexOperation,
|
RemoveFromGraphIndexOperation,
|
||||||
BatchAddToHNSWOperation,
|
BatchAddToVectorIndexOperation,
|
||||||
BatchAddToMetadataIndexOperation
|
BatchAddToMetadataIndexOperation
|
||||||
} from './IndexOperations.js'
|
} from './IndexOperations.js'
|
||||||
|
|
|
||||||
|
|
@ -1651,6 +1651,25 @@ export interface BrainyConfig {
|
||||||
*/
|
*/
|
||||||
disableAutoRebuild?: boolean
|
disableAutoRebuild?: boolean
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The floor (ms) of the apply-phase transaction budget's scaling formula:
|
||||||
|
* `max(transactionBudgetFloorMs, opCount × 2 000)`. The budget governs
|
||||||
|
* whether a `transact()` / single-op write's NEXT internal operation may
|
||||||
|
* **start** — never whether already-completed work gets rolled back after
|
||||||
|
* the fact (a single-op write can never time out post-hoc: it either runs
|
||||||
|
* or it commits). A trip mid-batch still rolls back every applied operation
|
||||||
|
* atomically and throws a retryable `TransactionTimeoutError`; only the
|
||||||
|
* floor of the formula is configurable here.
|
||||||
|
*
|
||||||
|
* Raise this when a cold store's first writes after a restart legitimately
|
||||||
|
* take longer than 30s per operation (e.g. page-cache-cold canonical writes
|
||||||
|
* on a large brain) so a bulk `transact()` batch gets a proportionally
|
||||||
|
* larger runway instead of tripping mid-batch. Lower it to fail faster on a
|
||||||
|
* latency-sensitive write path. Default: `30000` (30 s) — unchanged from
|
||||||
|
* pre-8.9 behavior.
|
||||||
|
*/
|
||||||
|
transactionBudgetFloorMs?: number
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration
|
* How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration
|
||||||
* before throwing a retryable `MigrationInProgressError`.
|
* before throwing a retryable `MigrationInProgressError`.
|
||||||
|
|
@ -1806,6 +1825,23 @@ export interface BrainyConfig {
|
||||||
*/
|
*/
|
||||||
eagerEmbeddings?: boolean
|
eagerEmbeddings?: boolean
|
||||||
|
|
||||||
|
/**
|
||||||
|
* When `true`, `init()` **awaits `warm()`** (see {@link Brainy.warm}) before
|
||||||
|
* it resolves — blocking startup until the vector index, metadata index,
|
||||||
|
* and graph adjacency have all faulted their backing storage in. This is a
|
||||||
|
* deliberate operator opt-in trade: startup takes longer, but the FIRST
|
||||||
|
* request after a cold restart runs at steady-state cost instead of paying
|
||||||
|
* page-cache-miss / demand-load latency on the critical path.
|
||||||
|
*
|
||||||
|
* Runs AFTER `init()`'s own sequence completes (index construction, crash
|
||||||
|
* recovery, migrations, VFS bootstrap) — never in place of any of it — so
|
||||||
|
* `warm()` always operates on a fully-initialized brain.
|
||||||
|
*
|
||||||
|
* Default: `false` (lazy — the pre-8.9 behavior: indexes demand-load on
|
||||||
|
* first access).
|
||||||
|
*/
|
||||||
|
warmOnOpen?: boolean
|
||||||
|
|
||||||
// Plugin configuration
|
// Plugin configuration
|
||||||
// Controls which plugins are loaded during init().
|
// Controls which plugins are loaded during init().
|
||||||
// - undefined (default): guarded auto-detection of the first-party
|
// - undefined (default): guarded auto-detection of the first-party
|
||||||
|
|
|
||||||
|
|
@ -423,6 +423,46 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
prodLog.debug('✅ Type-aware cache warming completed')
|
prodLog.debug('✅ Type-aware cache warming completed')
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Full hydration — the {@link Brainy.warm} readiness seam for the metadata
|
||||||
|
* index. Unlike {@link warmCache} / {@link warmCacheForTopTypes} (which
|
||||||
|
* warm only a heuristic subset: common fields plus the top-N types' top
|
||||||
|
* fields), this loads EVERY field's sparse index the field registry knows
|
||||||
|
* about — a real read through {@link loadSparseIndex} into the unified
|
||||||
|
* cache for each field, not a stat/existence check. Idempotent: an
|
||||||
|
* already-cached field's `loadSparseIndex` call is a cheap cache hit.
|
||||||
|
*
|
||||||
|
* Re-reads the field registry first when `fieldIndexes` is empty (a warm()
|
||||||
|
* call issued before `init()` populated it would otherwise hydrate
|
||||||
|
* nothing), then loads every discovered field in parallel.
|
||||||
|
*/
|
||||||
|
async hydrateAll(): Promise<void> {
|
||||||
|
if (this.fieldIndexes.size === 0) {
|
||||||
|
await this.loadFieldRegistry()
|
||||||
|
}
|
||||||
|
|
||||||
|
const fields = Array.from(this.fieldIndexes.keys())
|
||||||
|
if (fields.length === 0) {
|
||||||
|
prodLog.debug('[MetadataIndex] hydrateAll: no persisted fields to hydrate')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
prodLog.debug(`[MetadataIndex] hydrateAll: loading ${fields.length} field(s) — ${fields.join(', ')}`)
|
||||||
|
|
||||||
|
await Promise.all(
|
||||||
|
fields.map(async field => {
|
||||||
|
try {
|
||||||
|
await this.loadSparseIndex(field)
|
||||||
|
} catch (error) {
|
||||||
|
// A single field's load failure doesn't abort the rest of the
|
||||||
|
// hydration — warm() is a best-effort readiness step, never a
|
||||||
|
// correctness gate (queries still demand-load on miss).
|
||||||
|
prodLog.debug(`[MetadataIndex] hydrateAll: field '${field}' failed to load:`, error)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Acquire an in-memory lock for coordinating concurrent metadata index writes
|
* Acquire an in-memory lock for coordinating concurrent metadata index writes
|
||||||
* Uses in-memory locks since MetadataIndexManager doesn't have direct file system access
|
* Uses in-memory locks since MetadataIndexManager doesn't have direct file system access
|
||||||
|
|
|
||||||
310
tests/unit/brainy/warm.test.ts
Normal file
310
tests/unit/brainy/warm.test.ts
Normal file
|
|
@ -0,0 +1,310 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/brainy/warm
|
||||||
|
* @description Coverage for `brain.warm()` / `warmOnOpen` / the provider
|
||||||
|
* `warm?()` contract (the cold-restart readiness fix): first operations after
|
||||||
|
* a cold restart should run at steady-state cost instead of paying
|
||||||
|
* demand-load latency on the critical path.
|
||||||
|
*
|
||||||
|
* Uses a real, in-process fake plugin provider implementing the actual
|
||||||
|
* `VectorIndexProvider` contract from `src/plugin.ts` — the real seam a
|
||||||
|
* native provider (e.g. a disk-native accelerator) plugs into. The real
|
||||||
|
* built-in JS metadata index and graph adjacency index run against real
|
||||||
|
* (filesystem or in-memory) storage with pre-existing data, so their
|
||||||
|
* hydration paths are exercised for real, not mocked.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, afterEach } from 'vitest'
|
||||||
|
import * as fs from 'node:fs'
|
||||||
|
import * as os from 'node:os'
|
||||||
|
import * as path from 'node:path'
|
||||||
|
import { Brainy } from '../../../src/brainy.js'
|
||||||
|
import { NounType, VerbType } from '../../../src/types/graphTypes.js'
|
||||||
|
import type { VectorIndexProvider } from '../../../src/plugin.js'
|
||||||
|
import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
|
||||||
|
import { MetadataIndexManager } from '../../../src/utils/metadataIndex.js'
|
||||||
|
import { GraphAdjacencyIndex } from '../../../src/graph/graphAdjacencyIndex.js'
|
||||||
|
|
||||||
|
const tmpDirs: string[] = []
|
||||||
|
function mkTmp(): string {
|
||||||
|
const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-warm-'))
|
||||||
|
tmpDirs.push(d)
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
afterEach(() => {
|
||||||
|
for (const d of tmpDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true })
|
||||||
|
})
|
||||||
|
|
||||||
|
// Brainy's ValidationConfig fixes vectors at exactly 384 dimensions
|
||||||
|
// (src/utils/paramValidation.ts) — match it so `add()` doesn't reject test data.
|
||||||
|
const DIM = 384
|
||||||
|
const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin(seed + i))
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A real (not mocked) VectorIndexProvider implementation, backed by a plain
|
||||||
|
* Map, that optionally implements `warm()` — the exact seam
|
||||||
|
* `AddToVectorIndexOperation` / `brain.warm()` call through.
|
||||||
|
*/
|
||||||
|
class FakeVectorProvider implements VectorIndexProvider {
|
||||||
|
readonly name = 'fake-vector-provider'
|
||||||
|
readonly items = new Map<string, Vector>()
|
||||||
|
warmCalls = 0
|
||||||
|
searchCalls: Array<{ k?: number }> = []
|
||||||
|
// Only present on the instance when the constructor is told to — mirrors a
|
||||||
|
// real provider that may or may not implement the optional hook. Assigned
|
||||||
|
// in the constructor BODY (not as a field initializer): under native ES
|
||||||
|
// class fields, field initializers run before constructor-body statements
|
||||||
|
// — including the parameter-property assignment — so referencing a
|
||||||
|
// parameter property from a field initializer would see it as still
|
||||||
|
// `undefined`.
|
||||||
|
warm?: () => Promise<void>
|
||||||
|
|
||||||
|
constructor(hasWarm: boolean) {
|
||||||
|
if (hasWarm) {
|
||||||
|
this.warm = async (): Promise<void> => {
|
||||||
|
this.warmCalls++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async addItem(item: VectorDocument): Promise<string> {
|
||||||
|
this.items.set(item.id, item.vector)
|
||||||
|
return item.id
|
||||||
|
}
|
||||||
|
async removeItem(id: string): Promise<boolean> {
|
||||||
|
return this.items.delete(id)
|
||||||
|
}
|
||||||
|
async search(_queryVector: Vector, k?: number): Promise<Array<[string, number]>> {
|
||||||
|
this.searchCalls.push({ k })
|
||||||
|
return [...this.items.keys()].slice(0, k ?? 10).map((id) => [id, 0])
|
||||||
|
}
|
||||||
|
size(): number {
|
||||||
|
return this.items.size
|
||||||
|
}
|
||||||
|
clear(): void {
|
||||||
|
this.items.clear()
|
||||||
|
}
|
||||||
|
async rebuild(): Promise<void> {}
|
||||||
|
async flush(): Promise<number> {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
getPersistMode(): 'immediate' | 'deferred' {
|
||||||
|
return 'deferred'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Registers `provider` under the `'vector'` plugin key, before `init()`. */
|
||||||
|
function useFakeVectorProvider(brain: Brainy<any>, provider: FakeVectorProvider): void {
|
||||||
|
brain.use({
|
||||||
|
name: 'fake-vector-provider',
|
||||||
|
activate: async (ctx: any) => {
|
||||||
|
ctx.registerProvider('vector', () => provider)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('brain.warm()', () => {
|
||||||
|
it('(a) calls the vector provider\'s warm() when present and reports "warmed"', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(true)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
await brain.init()
|
||||||
|
await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
expect(provider.warmCalls).toBe(1)
|
||||||
|
expect(provider.searchCalls.length).toBe(0) // warm() ran — no probe fallback
|
||||||
|
expect(report.vector.outcome).toBe('warmed')
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(b) falls back to a probe search() when warm() is absent and reports "probed", never "warmed"', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(false)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
await brain.init()
|
||||||
|
await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
|
||||||
|
await brain.add({ data: 'b', type: NounType.Thing, vector: V(2) })
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
expect(provider.warmCalls).toBe(0) // no warm() on this provider
|
||||||
|
expect(provider.searchCalls.length).toBe(1) // the probe ran
|
||||||
|
expect(provider.searchCalls[0].k).toBe(Math.min(100, provider.size())) // k = min(100, size)
|
||||||
|
expect(report.vector.outcome).toBe('probed')
|
||||||
|
expect(report.vector.outcome).not.toBe('warmed') // never conflated
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('vector: reports "unavailable" when there is nothing to probe (empty index, unknown dimension)', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
// A disk-native provider MAY honestly report 0 resident entries while
|
||||||
|
// durable data exists on disk (the same posture documented on
|
||||||
|
// `VectorIndexProvider.isReady` — "an mmap/disk-native index may
|
||||||
|
// legitimately report 0 resident entries"). warm()'s probe fallback
|
||||||
|
// reads `size()`, so this is the real trigger for "nothing to probe":
|
||||||
|
// never a k=0 search, an honest skip.
|
||||||
|
const provider = new FakeVectorProvider(false)
|
||||||
|
provider.size = () => 0
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
await brain.init() // the VFS root bootstrap write sets `dimensions`, but size() still reports 0
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
expect(provider.searchCalls.length).toBe(0) // nothing probed
|
||||||
|
expect(report.vector.outcome).toBe('unavailable')
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(c) metadata + graph hydration paths actually execute against filesystem storage with pre-existing data', async () => {
|
||||||
|
const dir = mkTmp()
|
||||||
|
|
||||||
|
// Build a brain with real data (including a field NOT in the metadata
|
||||||
|
// index's own common-fields warm subset — 'wave' — so hydrateAll()'s
|
||||||
|
// FULL hydration is distinguishable from init()'s partial warmCache()),
|
||||||
|
// and real graph edges, then close it (persisting everything).
|
||||||
|
const seed = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'filesystem', path: dir },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
await seed.init()
|
||||||
|
const ids: string[] = []
|
||||||
|
for (let i = 0; i < 6; i++) {
|
||||||
|
ids.push(
|
||||||
|
await seed.add({
|
||||||
|
data: `entity ${i}`,
|
||||||
|
type: NounType.Thing,
|
||||||
|
metadata: { wave: i % 3 },
|
||||||
|
vector: V(i + 1)
|
||||||
|
})
|
||||||
|
)
|
||||||
|
}
|
||||||
|
for (let i = 0; i + 1 < ids.length; i++) {
|
||||||
|
await seed.relate({ from: ids[i], to: ids[i + 1], type: VerbType.RelatedTo })
|
||||||
|
}
|
||||||
|
await seed.close()
|
||||||
|
|
||||||
|
// Cold-reopen a FRESH instance and spy on the real hydration seams before
|
||||||
|
// calling warm(), so we assert they actually ran (not just that the
|
||||||
|
// report claims they did).
|
||||||
|
const metaHydrateCalls: number[] = []
|
||||||
|
const loadedFields: string[] = []
|
||||||
|
const graphInitCalls: number[] = []
|
||||||
|
|
||||||
|
const origHydrateAll = MetadataIndexManager.prototype.hydrateAll
|
||||||
|
const origLoadSparseIndex = (MetadataIndexManager.prototype as any).loadSparseIndex
|
||||||
|
const origGraphInit = GraphAdjacencyIndex.prototype.init
|
||||||
|
|
||||||
|
MetadataIndexManager.prototype.hydrateAll = async function (...args: any[]) {
|
||||||
|
metaHydrateCalls.push(1)
|
||||||
|
return origHydrateAll.apply(this, args as any)
|
||||||
|
}
|
||||||
|
;(MetadataIndexManager.prototype as any).loadSparseIndex = async function (
|
||||||
|
field: string,
|
||||||
|
...args: any[]
|
||||||
|
) {
|
||||||
|
loadedFields.push(field)
|
||||||
|
return origLoadSparseIndex.apply(this, [field, ...args] as any)
|
||||||
|
}
|
||||||
|
GraphAdjacencyIndex.prototype.init = async function (...args: any[]) {
|
||||||
|
graphInitCalls.push(1)
|
||||||
|
return origGraphInit.apply(this, args as any)
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'filesystem', path: dir },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
expect(metaHydrateCalls.length).toBe(1) // hydrateAll() actually ran
|
||||||
|
expect(loadedFields).toContain('wave') // a NON-common field was loaded — full hydration, not the heuristic subset
|
||||||
|
expect(report.metadata.outcome).toBe('warmed')
|
||||||
|
|
||||||
|
expect(graphInitCalls.length).toBeGreaterThanOrEqual(1) // graph's full-load seam ran (once at brain init, once via warm())
|
||||||
|
expect(report.graph.outcome).toBe('warmed')
|
||||||
|
|
||||||
|
// Correctness survives: the hydrated data still answers queries.
|
||||||
|
const byWhere = await brain.find({ type: NounType.Thing, where: { wave: 1 } })
|
||||||
|
expect(byWhere.length).toBe(2) // waves 1,4 of 0..5
|
||||||
|
|
||||||
|
await brain.close()
|
||||||
|
} finally {
|
||||||
|
MetadataIndexManager.prototype.hydrateAll = origHydrateAll
|
||||||
|
;(MetadataIndexManager.prototype as any).loadSparseIndex = origLoadSparseIndex
|
||||||
|
GraphAdjacencyIndex.prototype.init = origGraphInit
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(d) warmOnOpen: true runs warm() during init() — observable via the fake provider', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
warmOnOpen: true,
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(true)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
|
||||||
|
// No explicit brain.warm() call — warmOnOpen must have run it as part of init().
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
expect(provider.warmCalls).toBe(1)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('warmOnOpen defaults to false — init() does NOT run warm() unless opted in', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(true)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
expect(provider.warmCalls).toBe(0)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(e) WarmReport shape: outcome literal + durationMs per surface, plus totalDurationMs', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(true)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
await brain.init()
|
||||||
|
await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
for (const surface of ['vector', 'metadata', 'graph'] as const) {
|
||||||
|
expect(['warmed', 'probed', 'unavailable']).toContain(report[surface].outcome)
|
||||||
|
expect(typeof report[surface].durationMs).toBe('number')
|
||||||
|
expect(report[surface].durationMs).toBeGreaterThanOrEqual(0)
|
||||||
|
}
|
||||||
|
expect(typeof report.totalDurationMs).toBe('number')
|
||||||
|
expect(report.totalDurationMs).toBeGreaterThanOrEqual(0)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
})
|
||||||
149
tests/unit/transaction/budget-commit-completed-work.test.ts
Normal file
149
tests/unit/transaction/budget-commit-completed-work.test.ts
Normal file
|
|
@ -0,0 +1,149 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/transaction/budget-commit-completed-work
|
||||||
|
* @description Regression coverage for the "commit-completed-work" transaction
|
||||||
|
* budget semantics: the budget gates STARTING the next operation; it never
|
||||||
|
* converts already-completed work into a rollback. Concretely:
|
||||||
|
*
|
||||||
|
* - A single-op transaction can never time out post-hoc — its one operation
|
||||||
|
* either runs (and the transaction commits, however long it took) or it
|
||||||
|
* never gets to start (which cannot happen: there is no operation before it
|
||||||
|
* to have overrun the budget).
|
||||||
|
* - A multi-op transaction whose op `i` overruns the budget stops op `i+1`
|
||||||
|
* from starting: everything applied so far rolls back atomically and a
|
||||||
|
* `TransactionTimeoutError` is thrown — the mid-batch zero-loss guarantee is
|
||||||
|
* unchanged.
|
||||||
|
* - `transactTimeoutBudget`'s scaling floor (`max(floorMs, opCount * 2000)`)
|
||||||
|
* is overridable per call, the seam `BrainyConfig.transactionBudgetFloorMs`
|
||||||
|
* feeds at the brainy.ts read sites.
|
||||||
|
*
|
||||||
|
* Uses a real class implementing the `Operation` interface (not an anonymous
|
||||||
|
* object literal, not a mock of Transaction internals) so the exercised path
|
||||||
|
* is exactly what a real caller's operation looks like.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect } from 'vitest'
|
||||||
|
import { Transaction, transactTimeoutBudget } from '../../../src/transaction/Transaction.js'
|
||||||
|
import type { Operation, RollbackAction } from '../../../src/transaction/types.js'
|
||||||
|
import { TransactionTimeoutError } from '../../../src/transaction/errors.js'
|
||||||
|
|
||||||
|
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms))
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A real `Operation` implementation that sleeps for `delayMs` before applying
|
||||||
|
* a write to `log`, and returns a rollback action that removes it again. Used
|
||||||
|
* to deterministically make one operation "slow" relative to a transaction's
|
||||||
|
* budget without touching any Transaction internals.
|
||||||
|
*/
|
||||||
|
class SlowOperation implements Operation {
|
||||||
|
readonly name: string
|
||||||
|
executed = false
|
||||||
|
rolledBack = false
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private readonly log: string[],
|
||||||
|
label: string,
|
||||||
|
private readonly delayMs: number
|
||||||
|
) {
|
||||||
|
this.name = label
|
||||||
|
}
|
||||||
|
|
||||||
|
async execute(): Promise<RollbackAction | undefined> {
|
||||||
|
this.executed = true
|
||||||
|
if (this.delayMs > 0) await sleep(this.delayMs)
|
||||||
|
this.log.push(this.name)
|
||||||
|
return async () => {
|
||||||
|
this.rolledBack = true
|
||||||
|
const idx = this.log.indexOf(this.name)
|
||||||
|
if (idx >= 0) this.log.splice(idx, 1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('Transaction budget — commit-completed-work semantics', () => {
|
||||||
|
it('(a) single-op transaction whose op overruns the budget COMMITS — no rollback, no throw', async () => {
|
||||||
|
const log: string[] = []
|
||||||
|
const op = new SlowOperation(log, 'slow-single-op', 40)
|
||||||
|
|
||||||
|
// Budget (5ms) is far smaller than the op's 40ms — under the OLD
|
||||||
|
// (post-completion-check) semantics this would have thrown and rolled
|
||||||
|
// back after the op finished. Under the new semantics there is no
|
||||||
|
// operation after it to gate, so it commits.
|
||||||
|
const tx = new Transaction({ timeout: 5 })
|
||||||
|
tx.addOperation(op)
|
||||||
|
|
||||||
|
await expect(tx.execute()).resolves.toBeUndefined()
|
||||||
|
|
||||||
|
expect(tx.getState()).toBe('committed')
|
||||||
|
expect(op.executed).toBe(true)
|
||||||
|
expect(op.rolledBack).toBe(false)
|
||||||
|
expect(log).toEqual(['slow-single-op'])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(b) two-op transaction: op0 overruns the budget → op1 never starts, op0 rolls back, throws TransactionTimeoutError', async () => {
|
||||||
|
const log: string[] = []
|
||||||
|
const op0 = new SlowOperation(log, 'op0-overruns', 40)
|
||||||
|
const op1 = new SlowOperation(log, 'op1-never-starts', 0)
|
||||||
|
|
||||||
|
const tx = new Transaction({ timeout: 5 })
|
||||||
|
tx.addOperation(op0)
|
||||||
|
tx.addOperation(op1)
|
||||||
|
|
||||||
|
const error = await tx.execute().catch((e) => e)
|
||||||
|
|
||||||
|
expect(error).toBeInstanceOf(TransactionTimeoutError)
|
||||||
|
expect(op0.executed).toBe(true)
|
||||||
|
expect(op0.rolledBack).toBe(true)
|
||||||
|
expect(op1.executed).toBe(false)
|
||||||
|
expect(op1.rolledBack).toBe(false)
|
||||||
|
expect(log).toEqual([]) // op0's write was undone; op1 never wrote
|
||||||
|
expect(tx.getState()).toBe('rolled_back')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a single-op transaction never checks the budget before its only operation starts', async () => {
|
||||||
|
// Budget of 0ms: under the old "check before every op including 0" code
|
||||||
|
// this could theoretically trip before op 0 even started (if any
|
||||||
|
// measurable time elapsed between startTime capture and the check). The
|
||||||
|
// documented contract is stronger: operation 0 ALWAYS gets to start.
|
||||||
|
const log: string[] = []
|
||||||
|
const op = new SlowOperation(log, 'only-op', 5)
|
||||||
|
|
||||||
|
const tx = new Transaction({ timeout: 0 })
|
||||||
|
tx.addOperation(op)
|
||||||
|
|
||||||
|
await expect(tx.execute()).resolves.toBeUndefined()
|
||||||
|
expect(tx.getState()).toBe('committed')
|
||||||
|
expect(op.executed).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(c) transactTimeoutBudget: an explicit floorMs overrides the 30s default floor', () => {
|
||||||
|
// Below the per-op scaling term, a raised floor wins.
|
||||||
|
expect(transactTimeoutBudget(1, undefined, 5_000)).toBe(5_000) // 1 op × 2000 = 2000 < 5000 floor
|
||||||
|
expect(transactTimeoutBudget(1)).toBe(30_000) // unchanged default when floorMs omitted
|
||||||
|
|
||||||
|
// Above the floor, opCount * 2000 still wins over a smaller floor.
|
||||||
|
expect(transactTimeoutBudget(10, undefined, 5_000)).toBe(20_000) // 10 × 2000 = 20000 > 5000 floor
|
||||||
|
|
||||||
|
// A full `override` always wins, regardless of floorMs.
|
||||||
|
expect(transactTimeoutBudget(10, 999, 5_000)).toBe(999)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(c) a configured floor changes real Transaction behavior end-to-end via TransactionManager-style options', async () => {
|
||||||
|
// Simulates how brainy.ts wires `this.config.transactionBudgetFloorMs`
|
||||||
|
// into `transactTimeoutBudget()`: opCount 0 isolates the floor term
|
||||||
|
// (0 × 2000 = 0), so a lowered floor makes a normally-generous budget
|
||||||
|
// fail-fast for a real 2-op Transaction.
|
||||||
|
const lowFloorBudget = transactTimeoutBudget(0, undefined, 10) // 10ms floor
|
||||||
|
expect(lowFloorBudget).toBe(10)
|
||||||
|
|
||||||
|
const log: string[] = []
|
||||||
|
const op0 = new SlowOperation(log, 'op0', 30)
|
||||||
|
const op1 = new SlowOperation(log, 'op1-gated-by-lowered-floor', 0)
|
||||||
|
|
||||||
|
const tx = new Transaction({ timeout: lowFloorBudget })
|
||||||
|
tx.addOperation(op0)
|
||||||
|
tx.addOperation(op1)
|
||||||
|
|
||||||
|
const error = await tx.execute().catch((e) => e)
|
||||||
|
expect(error).toBeInstanceOf(TransactionTimeoutError)
|
||||||
|
expect(op1.executed).toBe(false)
|
||||||
|
})
|
||||||
|
})
|
||||||
176
tests/unit/transaction/vectorIndexOperations-rename.test.ts
Normal file
176
tests/unit/transaction/vectorIndexOperations-rename.test.ts
Normal file
|
|
@ -0,0 +1,176 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/transaction/vectorIndexOperations-rename
|
||||||
|
* @description Coverage for the backend-neutral vector-index transaction
|
||||||
|
* operations (formerly `AddToHNSWOperation` / `RemoveFromHNSWOperation`).
|
||||||
|
* These op-name strings surface directly in consumer-visible transaction
|
||||||
|
* journals and timings — a fossil "HNSW" name misdirects an operator running
|
||||||
|
* a native (non-HNSW) vector provider. Verifies:
|
||||||
|
* 1. rollback wiring stayed byte-identical under the rename,
|
||||||
|
* 2. the emitted `name` stamps the active backend — `'hnsw-js'` for
|
||||||
|
* Brainy's own JS index, a provider's own required `name` when it
|
||||||
|
* self-identifies, and the tolerant-loud `'unknown-provider'` literal
|
||||||
|
* (plus one console.warn) when a runtime instance lacks `name`
|
||||||
|
* altogether (an older native provider compiled against the previous,
|
||||||
|
* optional `providerId` contract) — never a silently-wrong guess.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, vi, afterEach } from 'vitest'
|
||||||
|
import {
|
||||||
|
AddToVectorIndexOperation,
|
||||||
|
RemoveFromVectorIndexOperation,
|
||||||
|
BatchAddToVectorIndexOperation
|
||||||
|
} from '../../../src/transaction/operations/IndexOperations.js'
|
||||||
|
import type { VectorIndexProvider } from '../../../src/plugin.js'
|
||||||
|
import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
|
||||||
|
import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A minimal, real (not mocked) VectorIndexProvider implementation backed by
|
||||||
|
* a plain Map — exercises the exact contract the operations call through,
|
||||||
|
* without any Brainy internals.
|
||||||
|
*/
|
||||||
|
class FakeVectorProvider implements VectorIndexProvider {
|
||||||
|
readonly items = new Map<string, Vector>()
|
||||||
|
constructor(readonly name: string) {}
|
||||||
|
|
||||||
|
async addItem(item: VectorDocument): Promise<string> {
|
||||||
|
this.items.set(item.id, item.vector)
|
||||||
|
return item.id
|
||||||
|
}
|
||||||
|
async removeItem(id: string): Promise<boolean> {
|
||||||
|
return this.items.delete(id)
|
||||||
|
}
|
||||||
|
async getItem(id: string): Promise<VectorDocument | undefined> {
|
||||||
|
const vector = this.items.get(id)
|
||||||
|
return vector ? { id, vector } : undefined
|
||||||
|
}
|
||||||
|
async search(): Promise<Array<[string, number]>> {
|
||||||
|
return []
|
||||||
|
}
|
||||||
|
size(): number {
|
||||||
|
return this.items.size
|
||||||
|
}
|
||||||
|
clear(): void {
|
||||||
|
this.items.clear()
|
||||||
|
}
|
||||||
|
async rebuild(): Promise<void> {}
|
||||||
|
async flush(): Promise<number> {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
getPersistMode(): 'immediate' | 'deferred' {
|
||||||
|
return 'immediate'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('Vector index transaction operations — backend-neutral rename', () => {
|
||||||
|
describe('rollback wiring (byte-identical behavior under the rename)', () => {
|
||||||
|
it('AddToVectorIndexOperation rollback removes a newly-added item', async () => {
|
||||||
|
const provider = new FakeVectorProvider('fake-provider')
|
||||||
|
const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
||||||
|
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect(provider.items.has('id-1')).toBe(true)
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
expect(provider.items.has('id-1')).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('AddToVectorIndexOperation rollback is a no-op when the item pre-existed (update semantics)', async () => {
|
||||||
|
const provider = new FakeVectorProvider('fake-provider')
|
||||||
|
await provider.addItem({ id: 'id-1', vector: [9, 9, 9] })
|
||||||
|
|
||||||
|
const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect(provider.items.get('id-1')).toEqual([1, 2, 3])
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
// Pre-existing item is NOT removed by rollback — it stays (the update
|
||||||
|
// itself is not undone by this op; that's RemoveFromVectorIndexOperation's job).
|
||||||
|
expect(provider.items.has('id-1')).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('RemoveFromVectorIndexOperation rollback re-adds the item with its original vector', async () => {
|
||||||
|
const provider = new FakeVectorProvider('fake-provider')
|
||||||
|
await provider.addItem({ id: 'id-1', vector: [4, 5, 6] })
|
||||||
|
|
||||||
|
const op = new RemoveFromVectorIndexOperation(provider, 'id-1', [4, 5, 6])
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect(provider.items.has('id-1')).toBe(false)
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
expect(provider.items.get('id-1')).toEqual([4, 5, 6])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('BatchAddToVectorIndexOperation rolls back every item in reverse order on undo', async () => {
|
||||||
|
const provider = new FakeVectorProvider('fake-provider')
|
||||||
|
const op = new BatchAddToVectorIndexOperation(provider, [
|
||||||
|
{ id: 'a', vector: [1] },
|
||||||
|
{ id: 'b', vector: [2] },
|
||||||
|
{ id: 'c', vector: [3] }
|
||||||
|
])
|
||||||
|
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect([...provider.items.keys()].sort()).toEqual(['a', 'b', 'c'])
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
expect(provider.items.size).toBe(0)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('backend stamping in the emitted op name', () => {
|
||||||
|
afterEach(() => {
|
||||||
|
vi.restoreAllMocks()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('stamps the real JS HNSW index as hnsw-js (self-identifying name)', () => {
|
||||||
|
const index = new JsHnswVectorIndex()
|
||||||
|
const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3])
|
||||||
|
const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3])
|
||||||
|
|
||||||
|
expect(addOp.name).toBe('AddToVectorIndex(hnsw-js)')
|
||||||
|
expect(removeOp.name).toBe('RemoveFromVectorIndex(hnsw-js)')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('stamps a self-identifying provider\'s own name, never "HNSW"', () => {
|
||||||
|
const provider = new FakeVectorProvider('acme-diskann')
|
||||||
|
const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
||||||
|
const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
||||||
|
const batchOp = new BatchAddToVectorIndexOperation(provider, [{ id: 'a', vector: [1] }])
|
||||||
|
|
||||||
|
expect(addOp.name).toBe('AddToVectorIndex(acme-diskann)')
|
||||||
|
expect(removeOp.name).toBe('RemoveFromVectorIndex(acme-diskann)')
|
||||||
|
expect(batchOp.name).toBe('BatchAddToVectorIndex(acme-diskann)')
|
||||||
|
expect(addOp.name).not.toContain('HNSW')
|
||||||
|
expect(removeOp.name).not.toContain('HNSW')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('tolerant-loud: stamps "unknown-provider" and warns exactly once when a runtime provider lacks the required `name` (an older native provider compiled against the previous optional `providerId` contract)', () => {
|
||||||
|
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
|
||||||
|
// Simulate a native provider compiled before `name` was required — the
|
||||||
|
// TypeScript contract requires `name`, but `as unknown as` mirrors what
|
||||||
|
// actually reaches this code at runtime from an un-rebuilt native addon.
|
||||||
|
const legacyProvider = {
|
||||||
|
addItem: async (item: VectorDocument) => item.id,
|
||||||
|
removeItem: async () => true,
|
||||||
|
search: async () => [],
|
||||||
|
size: () => 0,
|
||||||
|
clear: () => {},
|
||||||
|
rebuild: async () => {},
|
||||||
|
flush: async () => 0,
|
||||||
|
getPersistMode: () => 'immediate' as const
|
||||||
|
} as unknown as VectorIndexProvider
|
||||||
|
|
||||||
|
const addOp = new AddToVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
|
||||||
|
const removeOp = new RemoveFromVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
|
||||||
|
|
||||||
|
expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)')
|
||||||
|
expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)')
|
||||||
|
// Never silently falls back to the JS engine's name for a provider it
|
||||||
|
// knows nothing about — that's the exact fossil-naming bug being fixed.
|
||||||
|
expect(addOp.name).not.toContain('hnsw-js')
|
||||||
|
// Loud, not silent — but exactly once per provider instance, not once
|
||||||
|
// per op stamped against it.
|
||||||
|
expect(warnSpy).toHaveBeenCalledTimes(1)
|
||||||
|
expect(warnSpy.mock.calls[0][0]).toContain('name')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
})
|
||||||
Loading…
Add table
Add a link
Reference in a new issue