Compare commits
No commits in common. "main" and "v8.8.2" have entirely different histories.
36 changed files with 403 additions and 2781 deletions
16
CHANGELOG.md
16
CHANGELOG.md
|
|
@ -2,22 +2,6 @@
|
||||||
|
|
||||||
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
||||||
|
|
||||||
### [8.10.0](https://github.com/soulcraftlabs/brainy/compare/v8.9.0...v8.10.0) (2026-07-23)
|
|
||||||
|
|
||||||
- docs: adoption storefront — contributing guide, security policy, README support + cor section (9a99a7b)
|
|
||||||
- fix(release): push the public mirror explicitly and verify the tag lands at the right commit before publishing (6ba94c8)
|
|
||||||
- docs: project guide version line points at npm instead of a hardcoded stale number (3a1efc9)
|
|
||||||
- feat: vector provider identity is a required name field (hnsw-js), rendered [vector-index:<name>] (3be4ba9)
|
|
||||||
- feat: warm contract (warm/warmOnOpen/provider warm hook), configurable transact budget floor, backend-neutral vector index op names (55b867c)
|
|
||||||
|
|
||||||
|
|
||||||
### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19)
|
|
||||||
|
|
||||||
- docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78)
|
|
||||||
- fix: release drains in-flight writer-lock heartbeat — no phantom lock after unlink (70e4bc8)
|
|
||||||
- feat: flush() never compacts — history maintenance moves to close() with bounded passes (300d9f2)
|
|
||||||
|
|
||||||
|
|
||||||
### [8.8.2](https://github.com/soulcraftlabs/brainy/compare/v8.8.1...v8.8.2) (2026-07-19)
|
### [8.8.2](https://github.com/soulcraftlabs/brainy/compare/v8.8.1...v8.8.2) (2026-07-19)
|
||||||
|
|
||||||
- fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings (945d92d)
|
- fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings (945d92d)
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,7 @@ Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
|
||||||
|
|
||||||
**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
|
**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
|
||||||
|
|
||||||
**Current version:** run `npm view @soulcraft/brainy version` (never trust a hardcoded number here — this line went stale for months); consumer-facing changes tracked in `RELEASES.md`
|
**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
318
CONTRIBUTING.md
318
CONTRIBUTING.md
|
|
@ -1,66 +1,298 @@
|
||||||
# Contributing to Brainy
|
# Contributing to Brainy
|
||||||
|
|
||||||
Brainy is MIT-licensed and genuinely open to outside contributions. This page
|
Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for contributing to the project.
|
||||||
is the honest, current path — please don't rely on older instructions you
|
|
||||||
may find elsewhere in the repo's history.
|
|
||||||
|
|
||||||
## Where the project lives
|
## Code of Conduct
|
||||||
|
|
||||||
The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/brainy**.
|
By participating in this project, you agree to abide by our Code of Conduct:
|
||||||
It's anonymously readable and cloneable — no account needed to browse, clone,
|
- Be respectful and inclusive
|
||||||
or build.
|
- Welcome newcomers and help them get started
|
||||||
|
- Focus on constructive criticism
|
||||||
|
- Respect differing viewpoints and experiences
|
||||||
|
|
||||||
## How to contribute
|
## How to Contribute
|
||||||
|
|
||||||
**Found a bug, or have an idea?** Email **brainy@soulcraft.com**. No account,
|
### Reporting Issues
|
||||||
no ceremony — you'll get a receipt, and it goes to a human.
|
|
||||||
|
|
||||||
**Want to send a patch?** Two ways, both first-class:
|
Before creating an issue, please check existing issues to avoid duplicates.
|
||||||
|
|
||||||
- **Email a patch.** Run `git format-patch` against your change and email the
|
When creating an issue, include:
|
||||||
output to **brainy@soulcraft.com**. This is a genuinely supported path, not
|
- Clear, descriptive title
|
||||||
a fallback — plenty of good contributions arrive this way.
|
- Detailed description of the problem
|
||||||
- **Open a pull request on the forge.** Request an account at
|
- Steps to reproduce
|
||||||
**source.soulcraft.com** (registration is request-with-approval, so allow
|
- Expected vs actual behavior
|
||||||
a little lag), clone, push a branch, and open a PR there. Maintainers
|
- System information (OS, Node version, Brainy version)
|
||||||
review and land it.
|
- Code examples if applicable
|
||||||
|
|
||||||
Either way, for anything beyond a small fix, opening an issue first (email is
|
### Suggesting Features
|
||||||
fine) to talk through the approach saves everyone rework.
|
|
||||||
|
|
||||||
## Development setup
|
Feature requests are welcome! Please provide:
|
||||||
|
- Clear use case
|
||||||
|
- Proposed API/interface
|
||||||
|
- Examples of how it would work
|
||||||
|
- Any potential challenges or considerations
|
||||||
|
|
||||||
|
### Pull Requests
|
||||||
|
|
||||||
|
#### Before Starting
|
||||||
|
|
||||||
|
1. Check existing issues and PRs
|
||||||
|
2. Open an issue to discuss significant changes
|
||||||
|
3. Fork the repository
|
||||||
|
4. Create a feature branch from `main`
|
||||||
|
|
||||||
|
#### Development Setup
|
||||||
|
|
||||||
|
**Quick Setup (Recommended):**
|
||||||
```bash
|
```bash
|
||||||
git clone https://source.soulcraft.com/soulcraft/brainy.git
|
# Clone your fork
|
||||||
|
git clone https://github.com/your-username/brainy.git
|
||||||
cd brainy
|
cd brainy
|
||||||
|
|
||||||
|
# Run setup script (installs all dependencies including Rust)
|
||||||
|
./scripts/setup-dev.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**Manual Setup:**
|
||||||
|
```bash
|
||||||
|
# Clone your fork
|
||||||
|
git clone https://github.com/your-username/brainy.git
|
||||||
|
cd brainy
|
||||||
|
|
||||||
|
# Install system dependencies (Ubuntu/Debian)
|
||||||
|
sudo apt-get install -y build-essential pkg-config libssl-dev
|
||||||
|
|
||||||
|
# Install Rust (for WASM embedding engine)
|
||||||
|
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||||
|
source ~/.cargo/env
|
||||||
|
rustup target add wasm32-unknown-unknown
|
||||||
|
cargo install wasm-pack
|
||||||
|
|
||||||
|
# Install Node.js dependencies
|
||||||
npm install
|
npm install
|
||||||
|
|
||||||
|
# Build Candle WASM embedding engine
|
||||||
|
npm run build:candle
|
||||||
|
|
||||||
|
# Build TypeScript
|
||||||
npm run build
|
npm run build
|
||||||
|
|
||||||
|
# Run tests
|
||||||
npm test
|
npm test
|
||||||
```
|
```
|
||||||
|
|
||||||
Tests run on [Vitest](https://vitest.dev/). `npm test` runs the unit suite;
|
#### Making Changes
|
||||||
see `package.json` for `test:integration`, `test:coverage`, and friends.
|
|
||||||
|
|
||||||
## Standards
|
1. **Follow the code style**
|
||||||
|
- TypeScript for all source code
|
||||||
|
- Clear variable and function names
|
||||||
|
- Comments for complex logic
|
||||||
|
- JSDoc for public APIs
|
||||||
|
|
||||||
- **Strict TypeScript.** No `any` escape hatches to dodge the type checker.
|
2. **Write tests**
|
||||||
- **Tests exercise real behavior.** No mocking away the thing you're supposed
|
- Add tests for new features
|
||||||
to be testing.
|
- Update tests for changes
|
||||||
- **No stubs, no TODO-code.** If something can't be finished, say so and
|
- Ensure all tests pass
|
||||||
leave it out — don't merge a placeholder.
|
|
||||||
- **JSDoc on every exported function, class, and type.**
|
|
||||||
- **[Conventional Commits](https://www.conventionalcommits.org/).** `feat:`,
|
|
||||||
`fix:`, `docs:`, `perf:`, `refactor:`, `test:`, `chore:`. Never
|
|
||||||
`BREAKING CHANGE` in a commit message — major version bumps are a separate,
|
|
||||||
deliberate decision.
|
|
||||||
- **Performance claims are measured or labeled projected.** If a PR or its
|
|
||||||
description states a number, cite the benchmark that produced it (see
|
|
||||||
[docs/performance-envelopes.md](docs/performance-envelopes.md) for the
|
|
||||||
pattern). Don't state an estimate as if it were measured.
|
|
||||||
|
|
||||||
## License
|
3. **Update documentation**
|
||||||
|
- Update README if needed
|
||||||
|
- Add/update API documentation
|
||||||
|
- Include examples
|
||||||
|
|
||||||
Brainy is [MIT licensed](LICENSE). Contributions are accepted under the same
|
#### Commit Guidelines
|
||||||
license — there's no CLA to sign.
|
|
||||||
|
|
||||||
Thank you for considering a contribution.
|
Follow conventional commits format:
|
||||||
|
|
||||||
|
```
|
||||||
|
type(scope): description
|
||||||
|
|
||||||
|
[optional body]
|
||||||
|
|
||||||
|
[optional footer]
|
||||||
|
```
|
||||||
|
|
||||||
|
Types:
|
||||||
|
- `feat`: New feature
|
||||||
|
- `fix`: Bug fix
|
||||||
|
- `docs`: Documentation changes
|
||||||
|
- `style`: Code style changes
|
||||||
|
- `refactor`: Code refactoring
|
||||||
|
- `perf`: Performance improvements
|
||||||
|
- `test`: Test changes
|
||||||
|
- `chore`: Build/tooling changes
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
```bash
|
||||||
|
feat(triple): add graph traversal depth limit
|
||||||
|
fix(storage): handle concurrent write conflicts
|
||||||
|
docs(api): update search method documentation
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Submitting PR
|
||||||
|
|
||||||
|
1. Push to your fork
|
||||||
|
2. Create PR against `main` branch
|
||||||
|
3. Fill out PR template
|
||||||
|
4. Ensure CI checks pass
|
||||||
|
5. Wait for review
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
#### Running Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run all tests
|
||||||
|
npm test
|
||||||
|
|
||||||
|
# Run specific test file
|
||||||
|
npm test tests/core.test.ts
|
||||||
|
|
||||||
|
# Run with coverage
|
||||||
|
npm run test:coverage
|
||||||
|
|
||||||
|
# Watch mode
|
||||||
|
npm run test:watch
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Writing Tests
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { describe, it, expect } from 'vitest'
|
||||||
|
import { Brainy } from '../src'
|
||||||
|
|
||||||
|
describe('Feature Name', () => {
|
||||||
|
it('should do something specific', async () => {
|
||||||
|
const brain = new Brainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
// Test implementation
|
||||||
|
const result = await brain.search("test")
|
||||||
|
|
||||||
|
expect(result).toBeDefined()
|
||||||
|
expect(result.length).toBeGreaterThan(0)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture Guidelines
|
||||||
|
|
||||||
|
### Adding New Features
|
||||||
|
|
||||||
|
1. **Check existing functionality**
|
||||||
|
- Review `ARCHITECTURE.md`
|
||||||
|
- Check if similar features exist
|
||||||
|
- Consider if it should be an augmentation
|
||||||
|
|
||||||
|
2. **Design considerations**
|
||||||
|
- Maintain backward compatibility
|
||||||
|
- Consider performance impact
|
||||||
|
- Think about all storage adapters
|
||||||
|
- Plan for extensibility
|
||||||
|
|
||||||
|
3. **Implementation checklist**
|
||||||
|
- [ ] Core functionality
|
||||||
|
- [ ] Tests (unit and integration)
|
||||||
|
- [ ] Documentation
|
||||||
|
- [ ] TypeScript types
|
||||||
|
- [ ] Examples
|
||||||
|
- [ ] Performance benchmarks (if applicable)
|
||||||
|
|
||||||
|
### Creating Augmentations
|
||||||
|
|
||||||
|
Augmentations extend Brainy's functionality:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
import { BrainyAugmentation } from '../types'
|
||||||
|
|
||||||
|
export class MyAugmentation extends BrainyAugmentation {
|
||||||
|
name = 'MyAugmentation'
|
||||||
|
|
||||||
|
async onInit(brain: Brainy): Promise<void> {
|
||||||
|
// Initialize augmentation
|
||||||
|
}
|
||||||
|
|
||||||
|
async onAdd(item: any, brain: Brainy): Promise<any> {
|
||||||
|
// Process before adding
|
||||||
|
return item
|
||||||
|
}
|
||||||
|
|
||||||
|
async onSearch(query: any, results: any[], brain: Brainy): Promise<any[]> {
|
||||||
|
// Process search results
|
||||||
|
return results
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Performance Considerations
|
||||||
|
|
||||||
|
- Use batch operations where possible
|
||||||
|
- Implement caching strategically
|
||||||
|
- Consider memory usage
|
||||||
|
- Profile performance impacts
|
||||||
|
- Add benchmarks for critical paths
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
### API Documentation
|
||||||
|
|
||||||
|
Use JSDoc for all public APIs:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
/**
|
||||||
|
* Searches for similar items using vector similarity
|
||||||
|
* @param query - Search query (text or vector)
|
||||||
|
* @param options - Search options
|
||||||
|
* @returns Array of search results with scores
|
||||||
|
* @example
|
||||||
|
* ```typescript
|
||||||
|
* const results = await brain.search("machine learning", { limit: 10 })
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
async search(query: string | Vector, options?: SearchOptions): Promise<SearchResult[]> {
|
||||||
|
// Implementation
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
|
||||||
|
Add examples for new features:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// examples/feature-name.ts
|
||||||
|
import { Brainy } from 'brainy'
|
||||||
|
|
||||||
|
async function exampleUsage() {
|
||||||
|
const brain = new Brainy()
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
// Show feature usage
|
||||||
|
// Include comments explaining what's happening
|
||||||
|
// Handle errors appropriately
|
||||||
|
}
|
||||||
|
|
||||||
|
exampleUsage().catch(console.error)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Release Process
|
||||||
|
|
||||||
|
1. **Version bump**: Follow semantic versioning
|
||||||
|
2. **Update CHANGELOG**: Document all changes
|
||||||
|
3. **Run tests**: Ensure all tests pass
|
||||||
|
4. **Build**: Generate distribution files
|
||||||
|
5. **Tag**: Create git tag for version
|
||||||
|
6. **Publish**: Release to npm
|
||||||
|
|
||||||
|
## Getting Help
|
||||||
|
|
||||||
|
- **Discord**: Join our community
|
||||||
|
- **Issues**: Ask questions on GitHub
|
||||||
|
- **Discussions**: Share ideas and get feedback
|
||||||
|
|
||||||
|
## Recognition
|
||||||
|
|
||||||
|
Contributors will be recognized in:
|
||||||
|
- CHANGELOG.md for their contributions
|
||||||
|
- README.md contributors section
|
||||||
|
- GitHub contributors page
|
||||||
|
|
||||||
|
Thank you for contributing to Brainy! 🧠
|
||||||
26
README.md
26
README.md
|
|
@ -13,7 +13,7 @@
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/v/@soulcraft/brainy.svg" alt="npm version"></a>
|
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/v/@soulcraft/brainy.svg" alt="npm version"></a>
|
||||||
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/dm/@soulcraft/brainy.svg" alt="npm downloads"></a>
|
<a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/dm/@soulcraft/brainy.svg" alt="npm downloads"></a>
|
||||||
<a href="https://source.soulcraft.com/soulcraft/brainy/actions"><img src="https://source.soulcraft.com/soulcraft/brainy/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"></a>
|
<a href="https://github.com/soulcraftlabs/brainy/actions/workflows/ci.yml"><img src="https://github.com/soulcraftlabs/brainy/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
|
||||||
<a href="https://soulcraft.com/docs"><img src="https://img.shields.io/badge/docs-soulcraft.com-blue.svg" alt="Documentation"></a>
|
<a href="https://soulcraft.com/docs"><img src="https://img.shields.io/badge/docs-soulcraft.com-blue.svg" alt="Documentation"></a>
|
||||||
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a>
|
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a>
|
||||||
<a href="https://www.typescriptlang.org/"><img src="https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg" alt="TypeScript"></a>
|
<a href="https://www.typescriptlang.org/"><img src="https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg" alt="TypeScript"></a>
|
||||||
|
|
@ -23,9 +23,8 @@
|
||||||
<a href="#quick-start">Quick start</a> ·
|
<a href="#quick-start">Quick start</a> ·
|
||||||
<a href="#one-query-three-engines">One query</a> ·
|
<a href="#one-query-three-engines">One query</a> ·
|
||||||
<a href="#feature-tour">Features</a> ·
|
<a href="#feature-tour">Features</a> ·
|
||||||
<a href="#when-you-outgrow-brainy">Scale with Cor</a> ·
|
<a href="#from-laptop-to-hundreds-of-millions">Scale with Cor</a> ·
|
||||||
<a href="#documentation">Docs</a> ·
|
<a href="#documentation">Docs</a>
|
||||||
<a href="#support--community">Support</a>
|
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
@ -173,11 +172,9 @@ await brain.vfs.search('React components with hooks') // semantic file
|
||||||
|
|
||||||
**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
|
**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
|
||||||
|
|
||||||
## When you outgrow Brainy
|
## From laptop to hundreds of millions
|
||||||
|
|
||||||
Brainy's pure-TypeScript engines carry real workloads a long way on their own — see the measured, per-operation numbers (not marketing figures) in **[docs/performance-envelopes.md](docs/performance-envelopes.md)** for what to expect, unaccelerated, on plain filesystem storage.
|
Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**:
|
||||||
|
|
||||||
When a deployment needs native-scale vector/graph performance — memory-mapped indexes that don't need your dataset in RAM, billion-scale ambitions — add the native engine. **The API doesn't change:**
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm install @soulcraft/cor
|
npm install @soulcraft/cor
|
||||||
|
|
@ -190,14 +187,13 @@ await brain.init() // @soulcraft/cor detected — same code, native engines un
|
||||||
|
|
||||||
Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
|
Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
|
||||||
|
|
||||||
Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor is more headroom for when you need it, not capability held back to sell you later. Licensing and support: **cor@soulcraft.com**.
|
Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both.
|
||||||
|
|
||||||
## Performance
|
## Performance
|
||||||
|
|
||||||
- Per-operation p50/p95 at 1k and 10k entities, pure-JS floor, measured and re-run every release that touches a measured path: **[docs/performance-envelopes.md](docs/performance-envelopes.md)**.
|
|
||||||
- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
|
- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
|
||||||
- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
|
- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
|
||||||
- Capacity planning and architecture: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
|
- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
|
||||||
|
|
||||||
## Use cases
|
## Use cases
|
||||||
|
|
||||||
|
|
@ -216,10 +212,6 @@ Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor
|
||||||
|
|
||||||
**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
|
**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
|
||||||
|
|
||||||
## Support & community
|
## Contributing & license
|
||||||
|
|
||||||
- **Bugs and ideas** → **brainy@soulcraft.com** — no account needed, you'll get a receipt.
|
Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors.
|
||||||
- **Security reports** → **security@soulcraft.com** — see **[SECURITY.md](SECURITY.md)**.
|
|
||||||
- **Contributing** → see **[CONTRIBUTING.md](CONTRIBUTING.md)**.
|
|
||||||
|
|
||||||
MIT © Brainy Contributors.
|
|
||||||
|
|
|
||||||
111
RELEASES.md
111
RELEASES.md
|
|
@ -8,119 +8,8 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
|
||||||
- Debugging data, query, or storage behaviour
|
- Debugging data, query, or storage behaviour
|
||||||
- A new Brainy feature is available that you want to adopt
|
- A new Brainy feature is available that you want to adopt
|
||||||
|
|
||||||
## Removed APIs — 7.x → 8.x (the complete ledger)
|
|
||||||
|
|
||||||
Every public API removed at the 8.0 major, with its sanctioned replacement. If your code
|
|
||||||
still calls a left-column name on 8.x it throws (or the config key is rejected) — the
|
|
||||||
replacement is always a one-line change. (Standing contract from 8.9.0 forward: removals
|
|
||||||
happen only at majors, after ≥1 minor of loud runtime deprecation naming the replacement.)
|
|
||||||
|
|
||||||
| Removed (7.x) | Replacement (8.x) |
|
|
||||||
|---|---|
|
|
||||||
| `brain.search(query, k)` | `find({ query })` — semantic; `find({ query, searchMode })` for hybrid |
|
|
||||||
| `brain.getRelations({...})` | `related(id, opts)` for adjacency; `find({ connected: {...} })` for scoped traversal |
|
|
||||||
| `brain.neural()` clustering | `find({ vector })` + aggregation `GROUP BY` |
|
|
||||||
| `Db.search()` | `db.find({ vector })` |
|
|
||||||
| Pre-8.0 storage path aliases (`directory`, `basePath`, …) | one `storage.path` key (old aliases throw) |
|
|
||||||
| Reserved keys inside `metadata` bags (silently remapped in 7.x) | top-level params (`subtype`, `visibility`, `confidence`, `weight`, …) — reserved-in-bag throws |
|
|
||||||
| 7.x COW branches layout (`branches/main/`) | generational MVCC (`asOf()`, `now()`, `db.persist(path)`) — on-disk migration is automatic at first 8.x open |
|
|
||||||
|
|
||||||
The fork/snapshot family (`brain.snapshot()`, `createSnapshot()`, `restoreSnapshot()`)
|
|
||||||
is sometimes cited as a 7.x removal — those methods never existed on 7.x; the 8.0 Db API
|
|
||||||
(`asOf`/`persist`/`restore({confirm})`) is their first real implementation.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Unreleased (the warm contract: cold-restart writes stop paying demand-load latency)
|
|
||||||
|
|
||||||
From a production deployment's cold-restart incident: the FIRST writes after every
|
|
||||||
restart on a large brain measured 33–35s each (page cache cold) against the transact
|
|
||||||
apply budget — Brainy's own op-count-scaled budget, `max(30s, opCount × 2s)`, e.g.
|
|
||||||
32,000ms for a 16-op batch. There is no external deadline in this story, and no
|
|
||||||
post-completion veto either: every write is itself a multi-operation transaction (a
|
|
||||||
single `add()` applies several operations — canonical writes, the vector-index insert,
|
|
||||||
the metadata-index update), so ONE cold operation that legitimately runs ~33s consumes
|
|
||||||
the whole budget, the gate before the NEXT operation trips, and the write rolls back
|
|
||||||
atomically (zero loss, by design) — refused, retried, refused again, until the page
|
|
||||||
cache warms passively (~30 minutes). The cure is not weaker atomicity; it is the two
|
|
||||||
new knobs below — a budget floor sized for cold stores, and a warm contract that pays
|
|
||||||
demand-load cost OFF the transaction path.
|
|
||||||
|
|
||||||
- **The budget's start-gating contract is now explicit, documented, and pinned by
|
|
||||||
regression tests.** The budget gates STARTING the next operation — completed work is
|
|
||||||
never rolled back for elapsed time — and a transaction's first operation now
|
|
||||||
unconditionally starts by code, not merely because elapsed time happens to be ~0 when
|
|
||||||
it is checked. This has been the shipped schedule since 8.7.0 (no behavior change for
|
|
||||||
existing integrations); it is now stated in `Transaction.execute()`'s contract JSDoc
|
|
||||||
and enforced by tests so it cannot silently regress. Mid-batch atomicity is unchanged:
|
|
||||||
a trip before operation `i+1` still rolls back `0..i` and throws a retryable
|
|
||||||
`TransactionTimeoutError`.
|
|
||||||
- **The budget's 30s floor is now configurable**: `new Brainy({ transactionBudgetFloorMs })`
|
|
||||||
raises (or lowers) the floor of `max(transactionBudgetFloorMs, opCount × 2000)` for every
|
|
||||||
internal transact batch. Useful for a store whose cold writes legitimately run past 30s
|
|
||||||
per operation, so a bulk batch gets a proportionally larger runway instead of tripping
|
|
||||||
mid-batch on cold-cache latency.
|
|
||||||
- **New: `brain.warm()`** — eagerly loads/faults-in the vector index, metadata index, and
|
|
||||||
graph adjacency so the first real operation after a cold restart runs at steady-state
|
|
||||||
cost instead of paying demand-load latency on the critical path. Returns a `WarmReport`
|
|
||||||
with one honest outcome per surface — never conflate the first two:
|
|
||||||
- `'warmed'` — the surface's own provider `warm()` hook ran, or a full-hydration seam
|
|
||||||
loaded every shard/field/segment from storage. Steady-state cost is paid.
|
|
||||||
- `'probed'` — no `warm()` hook was available, so a best-effort read (one `search()` call
|
|
||||||
for the vector index) faulted in *some* backing storage as a side effect — real work,
|
|
||||||
but never reported as `'warmed'`.
|
|
||||||
- `'unavailable'` — nothing ran (no hook, no hydration seam, or nothing to probe).
|
|
||||||
- **New config: `warmOnOpen: true`** makes `init()` await `brain.warm()` before it resolves
|
|
||||||
— a deliberate blocking trade-off: startup takes longer, the first request doesn't.
|
|
||||||
Default `false` (unchanged lazy behavior).
|
|
||||||
- **New optional provider hook: `warm?(): Promise<void>`** on the vector and graph
|
|
||||||
acceleration provider contracts (`src/plugin.ts`) — a native provider can implement it to
|
|
||||||
eagerly pretouch its own backing storage (e.g. mmap pretouch); absence means brainy falls
|
|
||||||
back to the probe/hydration behavior above.
|
|
||||||
- **Transaction op-name strings changed in journals/timings**: the vector-index
|
|
||||||
transaction operations were renamed from `AddToHNSW`/`RemoveFromHNSW` to backend-neutral
|
|
||||||
`AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an
|
|
||||||
algorithm that may not be the one actually running (a non-HNSW native vector provider
|
|
||||||
emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist).
|
|
||||||
The parenthesized suffix names the ACTIVE backend: `hnsw-js` for the built-in engine, or
|
|
||||||
the native provider's own identity when it self-identifies. **If you parse these op-name
|
|
||||||
strings** (log processors, journal tooling), update your matcher from
|
|
||||||
`AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`.
|
|
||||||
- **Provider identity is now a REQUIRED `name` field** on the vector provider contract
|
|
||||||
(`VectorIndexProvider.name`, `src/plugin.ts`) — every implementation self-reports its own
|
|
||||||
identity truthfully (its algorithm/engine), never inheriting a default. It renders as the
|
|
||||||
op-name suffix above and, wherever the vector index identifies itself in prose log lines,
|
|
||||||
as the tag `[vector-index:<name>]`. **Native provider adoption is a one-line change**:
|
|
||||||
declare `readonly name = '<your-provider-name>'`. A provider instance that still lacks
|
|
||||||
`name` at runtime (an older native build compiled against the previous, optional field) is
|
|
||||||
never crashed on and never silently mislabeled: it stamps `unknown-provider` and emits one
|
|
||||||
loud warning naming the missing field, so the gap is discoverable instead of a permanent
|
|
||||||
fossil label in every journal line.
|
|
||||||
|
|
||||||
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
|
|
||||||
|
|
||||||
The write path stops paying maintenance costs — the last structural piece of the
|
|
||||||
flush-storm class (a production deployment measured single writes blocked 25–191s behind
|
|
||||||
history reclaim running inline on flush under memory pressure):
|
|
||||||
|
|
||||||
- **`flush()` never compacts history.** It persists the current window's deltas and
|
|
||||||
nothing else — its cost no longer depends on history backlog or retention mode, in any
|
|
||||||
configuration. **`close()` is the auto-compaction site** (time-bounded per pass, ~5s;
|
|
||||||
an early stop is a consistent prefix and the next pass resumes).
|
|
||||||
- **`compactHistory()` gains `timeBudgetMs`** — bound your own maintenance windows; the
|
|
||||||
same resumable-prefix guarantee applies.
|
|
||||||
- **The documented trade**: a long-lived writer that never closes accumulates history
|
|
||||||
until its next explicit `compactHistory()`. Predictable writes, explicit maintenance.
|
|
||||||
If you run bounded retention on an always-on service, schedule a periodic
|
|
||||||
`compactHistory({ ...caps, timeBudgetMs })` in your maintenance window.
|
|
||||||
- **New public doc: `docs/performance-envelopes.md`** — measured per-op envelopes
|
|
||||||
(p50/p95 at stated scales, hardware, and backend, with the measuring script cited).
|
|
||||||
Refresh rule going forward: any release touching a measured path re-runs that op's
|
|
||||||
benchmark and updates the envelope in the same release.
|
|
||||||
- **New in this file: the Removed APIs 7.x→8.x table** (top of this document) — every
|
|
||||||
removal with its sanctioned replacement, one place, per the engine-currency contract.
|
|
||||||
Standing from here: removals only at majors, after ≥1 minor of loud runtime deprecation.
|
|
||||||
|
|
||||||
## v8.8.2 — 2026-07-19 (one field-resolution law: reserved-field aggregates stop drifting)
|
## v8.8.2 — 2026-07-19 (one field-resolution law: reserved-field aggregates stop drifting)
|
||||||
|
|
||||||
Four fixes from a consumer conformance audit, all rooted in the same disease — two field-resolution
|
Four fixes from a consumer conformance audit, all rooted in the same disease — two field-resolution
|
||||||
|
|
|
||||||
36
SECURITY.md
36
SECURITY.md
|
|
@ -1,36 +0,0 @@
|
||||||
# Security Policy
|
|
||||||
|
|
||||||
## Reporting a vulnerability
|
|
||||||
|
|
||||||
Email **security@soulcraft.com**. That's the one door for security reports
|
|
||||||
across the company, and it works the same way for Brainy: every report is
|
|
||||||
read by a human, you'll get a private receipt, and we'll work with you on
|
|
||||||
coordinated disclosure — please don't open a public issue for anything
|
|
||||||
that isn't already public.
|
|
||||||
|
|
||||||
Include what you'd want if you were on the other end: affected version,
|
|
||||||
how to reproduce, and what you think the impact is. If you have a patch or
|
|
||||||
a suggested fix, send it along — it's welcome but not required.
|
|
||||||
|
|
||||||
There is no bounty program today. We're saying that plainly so you know
|
|
||||||
what to expect going in.
|
|
||||||
|
|
||||||
## Response time
|
|
||||||
|
|
||||||
We respond as fast as truth allows. That means: no fixed SLA, no promise of
|
|
||||||
a reply within a specific number of hours — but a real report from a real
|
|
||||||
person gets read promptly and taken seriously. If you haven't heard anything
|
|
||||||
in a reasonable stretch, a follow-up email is completely fine.
|
|
||||||
|
|
||||||
## Supported versions
|
|
||||||
|
|
||||||
The latest `8.x` minor release line receives security fixes. If you're
|
|
||||||
running an older major version, please upgrade before reporting — we can't
|
|
||||||
commit to backporting fixes to unsupported lines.
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
This policy covers the `@soulcraft/brainy` package itself — the code in
|
|
||||||
this repository. If you're evaluating a deployment that also uses
|
|
||||||
`@soulcraft/cor`, report issues in that package the same way, to the same
|
|
||||||
address; we'll route internally.
|
|
||||||
|
|
@ -344,12 +344,8 @@ For per-entity write coordination (rather than whole-store history), the
|
||||||
## Keeping history bounded
|
## Keeping history bounded
|
||||||
|
|
||||||
Under Model-B every write is a generation, so history can grow quickly —
|
Under Model-B every write is a generation, so history can grow quickly —
|
||||||
Brainy auto-compacts at `close()` (time-bounded per pass) under the
|
Brainy auto-compacts on every `flush()`/`close()` under the **`retention`**
|
||||||
**`retention`** knob (configured on the constructor). Since 8.9.0, `flush()`
|
knob (configured on the constructor):
|
||||||
never compacts: flushing is durability work and costs only what the current
|
|
||||||
window's writes cost, regardless of history backlog. A long-lived writer that
|
|
||||||
never closes keeps its history until its next explicit `compactHistory()` —
|
|
||||||
schedule one in your maintenance window if you run bounded retention:
|
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// Zero-config: ADAPTIVE — keep as much history as free disk/RAM allows,
|
// Zero-config: ADAPTIVE — keep as much history as free disk/RAM allows,
|
||||||
|
|
@ -363,13 +359,10 @@ new Brainy({ retention: 'all' })
|
||||||
new Brainy({ retention: { maxGenerations: 1000, maxAge: 7 * 86_400_000, maxBytes: 512 * 1024 ** 2 } })
|
new Brainy({ retention: { maxGenerations: 1000, maxAge: 7 * 86_400_000, maxBytes: 512 * 1024 ** 2 } })
|
||||||
```
|
```
|
||||||
|
|
||||||
Reclaim manually at any time (the same caps, plus an optional per-pass time
|
Reclaim manually at any time (the same caps):
|
||||||
budget for maintenance windows — an early stop is a consistent prefix and the
|
|
||||||
next pass resumes):
|
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 24 * 60 * 60 * 1000 })
|
await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 24 * 60 * 60 * 1000 })
|
||||||
await brain.compactHistory({ maxBytes: 512 * 1024 ** 2, timeBudgetMs: 10_000 })
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Compaction never breaks a pinned read — record-sets are reclaimed only when
|
Compaction never breaks a pinned read — record-sets are reclaimed only when
|
||||||
|
|
|
||||||
|
|
@ -1,83 +0,0 @@
|
||||||
---
|
|
||||||
title: Performance Envelopes
|
|
||||||
slug: guides/performance-envelopes
|
|
||||||
public: true
|
|
||||||
category: guides
|
|
||||||
template: guide
|
|
||||||
order: 40
|
|
||||||
description: Measured per-operation latency envelopes at stated scales — what to expect, on what hardware, and exactly how each number was produced.
|
|
||||||
next:
|
|
||||||
- guides/find-limits
|
|
||||||
---
|
|
||||||
|
|
||||||
# Performance Envelopes
|
|
||||||
|
|
||||||
Every number on this page is **measured, never projected** — produced by the script
|
|
||||||
cited at the bottom, against the built package (the artifact you install), on the stated
|
|
||||||
hardware. Each entry says what was measured, at what scale, on which storage backend.
|
|
||||||
When a release touches a measured path, that operation is re-measured and this page
|
|
||||||
updates in the same release.
|
|
||||||
|
|
||||||
Two scopes to keep straight:
|
|
||||||
|
|
||||||
- **These envelopes are the pure-JS engine** (no native accelerator registered) on
|
|
||||||
filesystem storage. This is the floor every deployment gets from `npm install` alone.
|
|
||||||
- **Accelerated deployments** (the optional native provider) publish their own numbers —
|
|
||||||
this page never claims them.
|
|
||||||
|
|
||||||
## Read operations
|
|
||||||
|
|
||||||
Reads are where the architecture pays off: after the write path has done its indexing
|
|
||||||
work, queries answer from purpose-built indexes without scanning.
|
|
||||||
|
|
||||||
| Operation | 1,000 entities | 10,000 entities | Notes |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `get(id)` (warm) | p50 < 0.1ms | p50 < 0.1ms | served from cache/metadata index |
|
|
||||||
| `find` (metadata: indexed equality + range, limit 100) | p50 1.0ms · p95 1.8ms | p50 7.0ms · p95 8.9ms | column-store bitmap paths |
|
|
||||||
| `related(id)` (per-node adjacency) | p50 < 0.1ms · p95 0.2ms | p50 < 0.1ms | LSM adjacency index — O(degree), scale-independent |
|
|
||||||
| `find` (semantic: embed + HNSW, 1k docs) | p50 178ms · p95 393ms | — | dominated by WASM query embedding (measured on a machine under concurrent load — treat the p95 as an upper bound); the vector search itself is single-digit ms |
|
|
||||||
|
|
||||||
## Write operations
|
|
||||||
|
|
||||||
Under Model-B **every write is its own durable generation** — a single-op `add` pays
|
|
||||||
serialization, before-image staging, and fsync before it acks. That durability is priced
|
|
||||||
into the write path visibly, by design:
|
|
||||||
|
|
||||||
| Operation | 1,000 entities | 10,000 entities | Notes |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `add` (single-op) | p50 167ms · p95 171ms | p50 165ms · p95 172ms | full durable generation per write — flat across scale |
|
|
||||||
| `addMany` (bulk) | ~163ms/entity | ~187ms/entity | **currently per-item commits** — see the honest note below |
|
|
||||||
| `relateMany` | ~0.8ms/edge | ~0.9ms/edge | edges batch efficiently today |
|
|
||||||
| `flush` (steady-state, 1 pending write) | p50 8ms · p95 10ms | p50 45ms · p95 52ms | durability-only since 8.9.0 — cost no longer depends on history backlog or retention mode |
|
|
||||||
|
|
||||||
**The honest note on bulk writes:** `addMany` today commits each item as its own
|
|
||||||
generation (the same durability as single-op `add`, serialized by the single-writer
|
|
||||||
lock), so bulk-load cost is N × single-op cost. Batched chunk commits (one generation
|
|
||||||
and one fsync window per chunk, as `removeMany` already does) are designed into the
|
|
||||||
unified-commit work on the current roadmap. Until that ships, size bulk imports
|
|
||||||
accordingly — 10k entities is minutes, not seconds, on filesystem storage.
|
|
||||||
|
|
||||||
## Open / close
|
|
||||||
|
|
||||||
| Operation | 1,000 entities | 10,000 entities | Notes |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `open` (empty store) | ~560ms | ~190ms | includes embedder initialization |
|
|
||||||
| `open` (warm, populated, clean shutdown) | 763ms | 4.9s | pure-JS vector index load dominates and grows with entity count; the native accelerator exists precisely to remove this |
|
|
||||||
| `close` | bounded | bounded | auto-compaction pass is time-bounded (~5s max) since 8.9.0 |
|
|
||||||
|
|
||||||
A store that was NOT cleanly closed pays index rebuilds on top of the warm-open
|
|
||||||
number (tens of seconds at 10k) — clean shutdown is worth engineering for.
|
|
||||||
|
|
||||||
## How these were produced
|
|
||||||
|
|
||||||
- **Hardware**: Intel Core i9-14900HX (32 threads), 62GB RAM, NVMe, Linux, Node v22.
|
|
||||||
- **Backend**: `storage: { type: 'filesystem' }`, pure JS (no native providers).
|
|
||||||
- **Embeddings**: deterministic stub for non-semantic ops (isolates engine cost);
|
|
||||||
the real WASM embedder for the semantic row (that's what you'll run).
|
|
||||||
- **Method**: p50/p95 over 50–200 samples per op against the built `dist/`;
|
|
||||||
the measuring script ships in the repo history and re-runs per release.
|
|
||||||
|
|
||||||
Numbers on different hardware will differ; the *shape* (sub-2ms indexed reads,
|
|
||||||
~160ms embedding-bound semantic queries, durability-priced writes) is the envelope
|
|
||||||
you should hold your deployment against. If your measurements diverge from these
|
|
||||||
shapes by an order of magnitude, something is wrong — file it.
|
|
||||||
4
package-lock.json
generated
4
package-lock.json
generated
|
|
@ -1,12 +1,12 @@
|
||||||
{
|
{
|
||||||
"name": "@soulcraft/brainy",
|
"name": "@soulcraft/brainy",
|
||||||
"version": "8.10.0",
|
"version": "8.8.2",
|
||||||
"lockfileVersion": 3,
|
"lockfileVersion": 3,
|
||||||
"requires": true,
|
"requires": true,
|
||||||
"packages": {
|
"packages": {
|
||||||
"": {
|
"": {
|
||||||
"name": "@soulcraft/brainy",
|
"name": "@soulcraft/brainy",
|
||||||
"version": "8.10.0",
|
"version": "8.8.2",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@msgpack/msgpack": "^3.1.2",
|
"@msgpack/msgpack": "^3.1.2",
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
{
|
{
|
||||||
"name": "@soulcraft/brainy",
|
"name": "@soulcraft/brainy",
|
||||||
"version": "8.10.0",
|
"version": "8.8.2",
|
||||||
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
|
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
|
||||||
"main": "dist/index.js",
|
"main": "dist/index.js",
|
||||||
"module": "dist/index.js",
|
"module": "dist/index.js",
|
||||||
|
|
@ -128,13 +128,13 @@
|
||||||
"publishConfig": {
|
"publishConfig": {
|
||||||
"access": "public"
|
"access": "public"
|
||||||
},
|
},
|
||||||
"homepage": "https://source.soulcraft.com/soulcraft/brainy",
|
"homepage": "https://github.com/soulcraftlabs/brainy",
|
||||||
"bugs": {
|
"bugs": {
|
||||||
"url": "https://source.soulcraft.com/soulcraft/brainy/issues"
|
"url": "https://github.com/soulcraftlabs/brainy/issues"
|
||||||
},
|
},
|
||||||
"repository": {
|
"repository": {
|
||||||
"type": "git",
|
"type": "git",
|
||||||
"url": "git+https://source.soulcraft.com/soulcraft/brainy.git"
|
"url": "git+https://github.com/soulcraftlabs/brainy.git"
|
||||||
},
|
},
|
||||||
"files": [
|
"files": [
|
||||||
"dist/**/*.js",
|
"dist/**/*.js",
|
||||||
|
|
|
||||||
|
|
@ -142,7 +142,7 @@ else
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# Create new changelog entry
|
# Create new changelog entry
|
||||||
CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraft/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
|
CHANGELOG_ENTRY="### [${NEW_VERSION}](https://github.com/soulcraftlabs/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
|
||||||
|
|
||||||
${COMMITS}
|
${COMMITS}
|
||||||
"
|
"
|
||||||
|
|
@ -175,59 +175,26 @@ echo -e "${BLUE}7️⃣ Creating git tag v${NEW_VERSION}...${NC}"
|
||||||
git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}"
|
git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}"
|
||||||
echo -e "${GREEN}✅ Tag created${NC}\n"
|
echo -e "${GREEN}✅ Tag created${NC}\n"
|
||||||
|
|
||||||
# Step 9: Push to origin — the forge is the one home (ruled 2026-07-23; the
|
# Step 9: Push to GitHub
|
||||||
# old public GitHub repo is archived history, no longer part of any release).
|
echo -e "${BLUE}8️⃣ Pushing to GitHub...${NC}"
|
||||||
echo -e "${BLUE}8️⃣ Pushing to origin...${NC}"
|
|
||||||
git push --follow-tags origin "$CURRENT_BRANCH"
|
git push --follow-tags origin "$CURRENT_BRANCH"
|
||||||
echo -e "${GREEN}✅ Pushed to origin${NC}\n"
|
echo -e "${GREEN}✅ Pushed to GitHub${NC}\n"
|
||||||
|
|
||||||
# Step 10: Publish — forge FIRST (home), npmjs second (the world's storefront).
|
# Step 10: Publish to npm
|
||||||
# The fleet-wide ~/.npmrc maps the @soulcraft scope to the forge registry, and
|
echo -e "${BLUE}9️⃣ Publishing to npm (dist-tag: ${NPM_TAG})...${NC}"
|
||||||
# a scope mapping BEATS `--registry` on the command line — so each publish
|
npm publish --tag "$NPM_TAG"
|
||||||
# names its registry via the scope override explicitly. Nothing implicit.
|
|
||||||
FORGE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraft/npm/"
|
|
||||||
FORGE_NPM_TOKEN_FILE="$HOME/.config/soulcraft/npm-publish-brainy.token"
|
|
||||||
echo -e "${BLUE}9️⃣ Publishing to the forge registry (home)...${NC}"
|
|
||||||
if [ -f "$FORGE_NPM_TOKEN_FILE" ]; then
|
|
||||||
TMPRC="$(mktemp)"
|
|
||||||
chmod 600 "$TMPRC"
|
|
||||||
{
|
|
||||||
echo "@soulcraft:registry=${FORGE_NPM_REG}"
|
|
||||||
echo "//source.soulcraft.com/api/packages/soulcraft/npm/:_authToken=$(cat "$FORGE_NPM_TOKEN_FILE")"
|
|
||||||
} > "$TMPRC"
|
|
||||||
if npm publish --tag "$NPM_TAG" --userconfig "$TMPRC"; then
|
|
||||||
echo -e "${GREEN}✅ Published to the forge${NC}\n"
|
|
||||||
else
|
|
||||||
rm -f "$TMPRC"
|
|
||||||
echo -e "${RED}❌ Forge publish FAILED — aborting before npmjs so the pair never diverges. Fix and re-run.${NC}"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
rm -f "$TMPRC"
|
|
||||||
else
|
|
||||||
echo -e "${RED}❌ Forge publish token missing (${FORGE_NPM_TOKEN_FILE}) — aborting. The forge is home; publish it first or restage the token.${NC}"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo -e "${BLUE}9️⃣½ Publishing to npmjs (storefront, dist-tag: ${NPM_TAG})...${NC}"
|
|
||||||
npm publish --tag "$NPM_TAG" "--@soulcraft:registry=https://registry.npmjs.org/"
|
|
||||||
# Brainy is the only PUBLIC @soulcraft package — verify visibility after every publish.
|
# Brainy is the only PUBLIC @soulcraft package — verify visibility after every publish.
|
||||||
npm access get status @soulcraft/brainy "--@soulcraft:registry=https://registry.npmjs.org/" || true
|
npm access get status @soulcraft/brainy || true
|
||||||
echo -e "${GREEN}✅ Published to npmjs${NC}\n"
|
echo -e "${GREEN}✅ Published to npm${NC}\n"
|
||||||
|
|
||||||
# Step 11: Release object on the forge (presentational — the tag, CHANGELOG,
|
# Step 11: Create GitHub release
|
||||||
# and RELEASES.md are the record; this just gives the forge UI a release page).
|
echo -e "${BLUE}🔟 Creating GitHub release...${NC}"
|
||||||
echo -e "${BLUE}🔟 Creating forge release...${NC}"
|
if [ "$PRERELEASE" = true ]; then
|
||||||
if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then
|
gh release create "v${NEW_VERSION}" --generate-notes --prerelease
|
||||||
if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \
|
|
||||||
-H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \
|
|
||||||
-d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then
|
|
||||||
echo -e "${GREEN}✅ Forge release created${NC}\n"
|
|
||||||
else
|
else
|
||||||
echo -e "${RED}⚠️ Forge release API call failed — tag + CHANGELOG remain the record; create the release page via the forge UI if wanted${NC}\n"
|
gh release create "v${NEW_VERSION}" --generate-notes
|
||||||
fi
|
|
||||||
else
|
|
||||||
echo -e "${RED}⚠️ FORGEJO_RELEASE_TOKEN unset — no release page created; tag + CHANGELOG remain the record${NC}\n"
|
|
||||||
fi
|
fi
|
||||||
|
echo -e "${GREEN}✅ GitHub release created${NC}\n"
|
||||||
|
|
||||||
# Step 12: Push public docs to the soulcraft.com docs ingest door
|
# Step 12: Push public docs to the soulcraft.com docs ingest door
|
||||||
# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when
|
# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when
|
||||||
|
|
@ -246,4 +213,4 @@ echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}"
|
||||||
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
echo -e "📦 npm: ${BLUE}https://www.npmjs.com/package/@soulcraft/brainy/v/${NEW_VERSION}${NC}"
|
echo -e "📦 npm: ${BLUE}https://www.npmjs.com/package/@soulcraft/brainy/v/${NEW_VERSION}${NC}"
|
||||||
echo -e "🏠 Forge: ${BLUE}https://source.soulcraft.com/soulcraft/brainy/releases/tag/v${NEW_VERSION}${NC}"
|
echo -e "🐙 GitHub: ${BLUE}https://github.com/soulcraftlabs/brainy/releases/tag/v${NEW_VERSION}${NC}"
|
||||||
|
|
|
||||||
304
src/brainy.ts
304
src/brainy.ts
|
|
@ -63,9 +63,7 @@ import type {
|
||||||
PathOptions,
|
PathOptions,
|
||||||
MetadataIndexProvider,
|
MetadataIndexProvider,
|
||||||
OpaqueIdSet,
|
OpaqueIdSet,
|
||||||
AtGenerationVectors,
|
AtGenerationVectors
|
||||||
VectorIndexProvider,
|
|
||||||
GraphIndexProvider
|
|
||||||
} from './plugin.js'
|
} from './plugin.js'
|
||||||
import type {
|
import type {
|
||||||
BrainyPlugin,
|
BrainyPlugin,
|
||||||
|
|
@ -90,12 +88,12 @@ import { findCallerLocation } from './utils/callerLocation.js'
|
||||||
import {
|
import {
|
||||||
SaveNounMetadataOperation,
|
SaveNounMetadataOperation,
|
||||||
SaveNounOperation,
|
SaveNounOperation,
|
||||||
AddToVectorIndexOperation,
|
AddToHNSWOperation,
|
||||||
AddToMetadataIndexOperation,
|
AddToMetadataIndexOperation,
|
||||||
SaveVerbMetadataOperation,
|
SaveVerbMetadataOperation,
|
||||||
SaveVerbOperation,
|
SaveVerbOperation,
|
||||||
AddToGraphIndexOperation,
|
AddToGraphIndexOperation,
|
||||||
RemoveFromVectorIndexOperation,
|
RemoveFromHNSWOperation,
|
||||||
RemoveFromMetadataIndexOperation,
|
RemoveFromMetadataIndexOperation,
|
||||||
RemoveFromGraphIndexOperation,
|
RemoveFromGraphIndexOperation,
|
||||||
UpdateNounMetadataOperation,
|
UpdateNounMetadataOperation,
|
||||||
|
|
@ -268,7 +266,6 @@ type ResolvedBrainyConfig = Required<
|
||||||
| 'retention'
|
| 'retention'
|
||||||
| 'eagerEmbeddings'
|
| 'eagerEmbeddings'
|
||||||
| 'migrationWaitTimeoutMs'
|
| 'migrationWaitTimeoutMs'
|
||||||
| 'transactionBudgetFloorMs'
|
|
||||||
>
|
>
|
||||||
> &
|
> &
|
||||||
Pick<
|
Pick<
|
||||||
|
|
@ -281,7 +278,6 @@ type ResolvedBrainyConfig = Required<
|
||||||
| 'retention'
|
| 'retention'
|
||||||
| 'eagerEmbeddings'
|
| 'eagerEmbeddings'
|
||||||
| 'migrationWaitTimeoutMs'
|
| 'migrationWaitTimeoutMs'
|
||||||
| 'transactionBudgetFloorMs'
|
|
||||||
>
|
>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -392,38 +388,6 @@ class InsertPreconditionExistsSignal extends Error {
|
||||||
*/
|
*/
|
||||||
export type IndexFamily = 'vector' | 'metadata' | 'graph'
|
export type IndexFamily = 'vector' | 'metadata' | 'graph'
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Honest per-surface outcome for {@link Brainy.warm}. Literal
|
|
||||||
* meanings — never conflate the first two:
|
|
||||||
* - `'warmed'` — the provider's own `warm?()` hook ran (vector/graph), or the
|
|
||||||
* surface's full-hydration seam loaded EVERY shard/field/segment from
|
|
||||||
* storage (metadata; graph's fallback path). The surface is genuinely at
|
|
||||||
* steady-state cost for the next operation.
|
|
||||||
* - `'probed'` — no `warm?()` hook was available, so a best-effort read
|
|
||||||
* (e.g. one `search()` call) faulted in *some* backing storage as a side
|
|
||||||
* effect. Real work happened, but it is NOT the same guarantee as
|
|
||||||
* `'warmed'` — never reported as `'warmed'`.
|
|
||||||
* - `'unavailable'` — nothing ran: no hook, no hydration seam, and (for the
|
|
||||||
* vector probe fallback) nothing to probe (an empty index or unknown
|
|
||||||
* vector dimension). The surface is unchanged by this `warm()` call.
|
|
||||||
*/
|
|
||||||
export type WarmOutcome = 'warmed' | 'probed' | 'unavailable'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Result of {@link Brainy.warm}: one {@link WarmOutcome} +
|
|
||||||
* elapsed time per index surface, plus the total wall-clock time for the
|
|
||||||
* whole call. `durationMs` is measured around exactly the work described by
|
|
||||||
* that surface's `outcome` (e.g. the vector entry's `durationMs` times the
|
|
||||||
* provider `warm()` call OR the probe `search()` call — whichever ran).
|
|
||||||
*/
|
|
||||||
export interface WarmReport {
|
|
||||||
vector: { outcome: WarmOutcome; durationMs: number }
|
|
||||||
metadata: { outcome: WarmOutcome; durationMs: number }
|
|
||||||
graph: { outcome: WarmOutcome; durationMs: number }
|
|
||||||
/** Total wall-clock time for the whole `warm()` call (all three surfaces). */
|
|
||||||
totalDurationMs: number
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
|
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
|
||||||
* Within the window, queries rethrow the recorded failure instantly (loud,
|
* Within the window, queries rethrow the recorded failure instantly (loud,
|
||||||
|
|
@ -432,15 +396,6 @@ export interface WarmReport {
|
||||||
*/
|
*/
|
||||||
const AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS = 30_000
|
const AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS = 30_000
|
||||||
|
|
||||||
/**
|
|
||||||
* Time budget for the auto-compaction pass at close() (8.9.0). Bounds how long
|
|
||||||
* a clean shutdown spends reclaiming history backlog — an early stop is a
|
|
||||||
* consistent prefix and the next close/explicit pass resumes. Explicit
|
|
||||||
* `compactHistory()` calls are unbounded unless the caller passes their own
|
|
||||||
* `timeBudgetMs` (maintenance windows choose their own budgets).
|
|
||||||
*/
|
|
||||||
const CLOSE_COMPACTION_BUDGET_MS = 5_000
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The main Brainy class - Clean, Beautiful, Powerful
|
* The main Brainy class - Clean, Beautiful, Powerful
|
||||||
* REAL IMPLEMENTATION - No stubs, no mocks
|
* REAL IMPLEMENTATION - No stubs, no mocks
|
||||||
|
|
@ -1418,17 +1373,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// Eager index warm (operator opt-in, `warmOnOpen: true`). Runs AFTER
|
|
||||||
// every step above — index construction, crash recovery, migrations,
|
|
||||||
// VFS bootstrap — never in place of any of it, so `warm()` always
|
|
||||||
// operates on a fully-initialized brain. Blocking BY DESIGN: the
|
|
||||||
// operator traded a longer startup for a first-request that runs at
|
|
||||||
// steady-state cost instead of paying demand-load latency on the
|
|
||||||
// critical path. See the `warmOnOpen` JSDoc in brainy.types.ts.
|
|
||||||
if (this.config.warmOnOpen) {
|
|
||||||
await this.warm()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Resolve ready Promise - consumers awaiting brain.ready will now proceed
|
// Resolve ready Promise - consumers awaiting brain.ready will now proceed
|
||||||
if (this._readyResolve) {
|
if (this._readyResolve) {
|
||||||
this._readyResolve()
|
this._readyResolve()
|
||||||
|
|
@ -1827,9 +1771,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
await this.generationStore.runWithoutGeneration(() =>
|
await this.generationStore.runWithoutGeneration(() =>
|
||||||
this.transactionManager.executeTransaction(run, {
|
this.transactionManager.executeTransaction(run, {
|
||||||
timeout: transactTimeoutBudget(
|
timeout: transactTimeoutBudget(
|
||||||
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
|
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
|
||||||
undefined,
|
|
||||||
this.config.transactionBudgetFloorMs
|
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
)
|
)
|
||||||
|
|
@ -1846,9 +1788,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
execute: () =>
|
execute: () =>
|
||||||
this.transactionManager.executeTransaction(run, {
|
this.transactionManager.executeTransaction(run, {
|
||||||
timeout: transactTimeoutBudget(
|
timeout: transactTimeoutBudget(
|
||||||
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
|
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
|
||||||
undefined,
|
|
||||||
this.config.transactionBudgetFloorMs
|
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
@ -2160,7 +2100,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
|
|
||||||
// Operation 3: Add to HNSW index (after entity saved)
|
// Operation 3: Add to HNSW index (after entity saved)
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new AddToVectorIndexOperation(this.index, id, vector)
|
new AddToHNSWOperation(this.index, id, vector)
|
||||||
)
|
)
|
||||||
|
|
||||||
// Operation 4: Add to metadata index
|
// Operation 4: Add to metadata index
|
||||||
|
|
@ -3149,10 +3089,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Operation 3-4: Update HNSW index (remove and re-add if reindexing needed)
|
// Operation 3-4: Update HNSW index (remove and re-add if reindexing needed)
|
||||||
if (needsReindexing) {
|
if (needsReindexing) {
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector)
|
new RemoveFromHNSWOperation(this.index, params.id, existing.vector)
|
||||||
)
|
)
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new AddToVectorIndexOperation(this.index, params.id, vector)
|
new AddToHNSWOperation(this.index, params.id, vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -3268,7 +3208,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Operation 1: Remove from vector index
|
// Operation 1: Remove from vector index
|
||||||
if (noun) {
|
if (noun) {
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
|
new RemoveFromHNSWOperation(this.index, id, noun.vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -7076,7 +7016,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Add delete operations to transaction
|
// Add delete operations to transaction
|
||||||
if (noun) {
|
if (noun) {
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
|
new RemoveFromHNSWOperation(this.index, id, noun.vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -7939,13 +7879,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Budget scales with the batch (or the caller's explicit
|
// Budget scales with the batch (or the caller's explicit
|
||||||
// timeoutMs): a flat 30s cap silently limited honest bulk work
|
// timeoutMs): a flat 30s cap silently limited honest bulk work
|
||||||
// to ~15 ops on network disks (~2s/op measured in the field).
|
// to ~15 ops on network disks (~2s/op measured in the field).
|
||||||
{
|
{ timeout: transactTimeoutBudget(plan.operations.length, options?.timeoutMs) }
|
||||||
timeout: transactTimeoutBudget(
|
|
||||||
plan.operations.length,
|
|
||||||
options?.timeoutMs,
|
|
||||||
this.config.transactionBudgetFloorMs
|
|
||||||
)
|
|
||||||
}
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
}))
|
}))
|
||||||
|
|
@ -8322,27 +8256,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return this.generationStore.compact(options)
|
return this.generationStore.compact(options)
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Repack cold generation history into sealed segments —
|
|
||||||
* re-representation, never deletion: every record and delta stays readable
|
|
||||||
* (`asOf()` unchanged); the physical file count drops by orders of
|
|
||||||
* magnitude. Runs automatically (time-bounded) at `close()`; call this for
|
|
||||||
* explicit maintenance windows on long-lived writers. The ONLY history
|
|
||||||
* transform permitted under the archival profile (`retention: 'all'`).
|
|
||||||
* @param options - `timeBudgetMs` bounds the pass (early stop = consistent
|
|
||||||
* prefix, next pass resumes); `batchGenerations` sizes each fold.
|
|
||||||
* @returns Folded generation count and segments created.
|
|
||||||
*/
|
|
||||||
async repackHistory(options?: {
|
|
||||||
timeBudgetMs?: number
|
|
||||||
batchGenerations?: number
|
|
||||||
}): Promise<{ foldedGenerations: number; segmentsCreated: number }> {
|
|
||||||
this.assertWritable('repackHistory')
|
|
||||||
await this.ensureInitialized()
|
|
||||||
await this.generationStore.flushPendingSingleOps()
|
|
||||||
return this.generationStore.repackHistory(options)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Read-only generational-history footprint for fleet audits:
|
* @description Read-only generational-history footprint for fleet audits:
|
||||||
* generation count, total on-disk bytes, generation/timestamp range, the
|
* generation count, total on-disk bytes, generation/timestamp range, the
|
||||||
|
|
@ -8370,24 +8283,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* @description A deterministic content digest of the generation log through
|
|
||||||
* `g` (D8 — gate-to-generation provenance): identical history produces the
|
|
||||||
* identical digest on any machine; divergence produces a different one.
|
|
||||||
* Release gates and suite verdicts pin `{generation, digest}` and verify
|
|
||||||
* both at execution time instead of pinning a git commit. O(segments +
|
|
||||||
* live-tier window), never O(all generations). Throws `RangeError` out of
|
|
||||||
* range and `GenerationCompactedError` below the horizon — a gate can
|
|
||||||
* never silently pin reclaimed history.
|
|
||||||
* @example
|
|
||||||
* const gate = { generation: brain.generation(), digest: await brain.generationDigest(brain.generation()) }
|
|
||||||
*/
|
|
||||||
async generationDigest(g: number): Promise<string> {
|
|
||||||
await this.ensureInitialized()
|
|
||||||
await this.generationStore.flushPendingSingleOps()
|
|
||||||
return this.generationStore.generationDigest(g)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Drive the adaptive retention byte budget at runtime — the
|
* @description Drive the adaptive retention byte budget at runtime — the
|
||||||
* settable input a machine-level coordinator (e.g. cor's `ResourceManager`,
|
* settable input a machine-level coordinator (e.g. cor's `ResourceManager`,
|
||||||
|
|
@ -8467,24 +8362,19 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Run history compaction under the resolved `retention` policy
|
* @description Run history compaction under the resolved `retention` policy
|
||||||
* when `autoCompact` is on (the default). Invoked from `close()` ONLY
|
* when `autoCompact` is on (the default). Invoked from `flush()` and
|
||||||
* (8.9.0) — flush() is durability work and never pays maintenance costs; a
|
* `close()` so generational record-sets cannot accumulate unbounded across a
|
||||||
* production deployment measured reclaim-on-flush blocking single writes
|
* long-lived writer's lifetime.
|
||||||
* for 25-191s under memory pressure. Long-lived writers that never close
|
|
||||||
* accumulate history until their next explicit `compactHistory()` — the
|
|
||||||
* documented trade: predictable writes, explicit maintenance.
|
|
||||||
*
|
*
|
||||||
* - `'all'` → returns without reclaiming (index compaction for speed still
|
* - `'all'` → returns without reclaiming (index compaction for speed still
|
||||||
* runs elsewhere; history is decoupled and kept).
|
* runs elsewhere; history is decoupled and kept).
|
||||||
* - `'adaptive'` → reclaim oldest-unpinned history down to the byte budget.
|
* - `'adaptive'` → reclaim oldest-unpinned history down to the byte budget.
|
||||||
* - `'explicit'` → apply the supplied `maxGenerations`/`maxAge`/`maxBytes` caps.
|
* - `'explicit'` → apply the supplied `maxGenerations`/`maxAge`/`maxBytes` caps.
|
||||||
*
|
*
|
||||||
* Every auto pass is TIME-BOUNDED ({@link CLOSE_COMPACTION_BUDGET_MS}) so a
|
|
||||||
* large backlog can never stall a clean shutdown; the next pass resumes.
|
|
||||||
* Read-only instances and an explicit `autoCompact: false` skip silently.
|
* Read-only instances and an explicit `autoCompact: false` skip silently.
|
||||||
* Pinned generations are never reclaimed ({@link GenerationStore.compact}).
|
* Pinned generations are never reclaimed ({@link GenerationStore.compact}).
|
||||||
* Failures are logged and swallowed — compaction is housekeeping and must
|
* Failures are logged and swallowed — compaction is housekeeping and must
|
||||||
* never fail a clean shutdown.
|
* never fail a flush or a clean shutdown.
|
||||||
*/
|
*/
|
||||||
private async autoCompactHistory(): Promise<void> {
|
private async autoCompactHistory(): Promise<void> {
|
||||||
// Nothing to compact on a read-only instance or before init wired up the
|
// Nothing to compact on a read-only instance or before init wired up the
|
||||||
|
|
@ -8500,16 +8390,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
if (policy.mode === 'adaptive') {
|
if (policy.mode === 'adaptive') {
|
||||||
const budget = this.adaptiveHistoryBudgetBytes(policy.budgetBytes)
|
const budget = this.adaptiveHistoryBudgetBytes(policy.budgetBytes)
|
||||||
if (budget === Infinity) return // no pressure signal → keep everything this pass
|
if (budget === Infinity) return // no pressure signal → keep everything this pass
|
||||||
await this.generationStore.compact({
|
await this.generationStore.compact({ maxBytes: budget })
|
||||||
maxBytes: budget,
|
|
||||||
timeBudgetMs: CLOSE_COMPACTION_BUDGET_MS
|
|
||||||
})
|
|
||||||
} else {
|
} else {
|
||||||
await this.generationStore.compact({
|
await this.generationStore.compact({
|
||||||
maxGenerations: policy.maxGenerations,
|
maxGenerations: policy.maxGenerations,
|
||||||
maxAge: policy.maxAge,
|
maxAge: policy.maxAge,
|
||||||
maxBytes: policy.maxBytes,
|
maxBytes: policy.maxBytes
|
||||||
timeBudgetMs: CLOSE_COMPACTION_BUDGET_MS
|
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
|
@ -9368,7 +9254,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
|
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
|
||||||
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
|
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
|
||||||
new AddToVectorIndexOperation(this.index, id, vector),
|
new AddToHNSWOperation(this.index, id, vector),
|
||||||
new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing)
|
new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing)
|
||||||
)
|
)
|
||||||
plan.touchedNouns.push(id)
|
plan.touchedNouns.push(id)
|
||||||
|
|
@ -9536,8 +9422,8 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
)
|
)
|
||||||
if (needsReindexing) {
|
if (needsReindexing) {
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector),
|
new RemoveFromHNSWOperation(this.index, params.id, existing.vector),
|
||||||
new AddToVectorIndexOperation(this.index, params.id, vector)
|
new AddToHNSWOperation(this.index, params.id, vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
|
|
@ -9622,7 +9508,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
|
|
||||||
if (noun) {
|
if (noun) {
|
||||||
plan.operations.push(new RemoveFromVectorIndexOperation(this.index, id, noun.vector))
|
plan.operations.push(new RemoveFromHNSWOperation(this.index, id, noun.vector))
|
||||||
}
|
}
|
||||||
if (metadata) {
|
if (metadata) {
|
||||||
plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata))
|
plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata))
|
||||||
|
|
@ -10654,14 +10540,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
this.generationStore.persistCounterNow()
|
this.generationStore.persistCounterNow()
|
||||||
])
|
])
|
||||||
|
|
||||||
// NOTE (8.9.0): flush() no longer compacts history. Flush is DURABILITY
|
// 6. Auto-compact generational history per config.retention (default on).
|
||||||
// work — it must cost what this window's deltas cost, never what the
|
// Runs after the flush so durable state is in place; respects live
|
||||||
// history backlog costs. Auto-compaction (a MAINTENANCE concern) runs at
|
// Db pins and an explicit autoCompact: false.
|
||||||
// close() and via explicit compactHistory(); a production deployment
|
await this.autoCompactHistory()
|
||||||
// measured reclaim-on-flush stalling single writes for 25-191s under
|
|
||||||
// memory pressure, which is exactly the class this separation ends.
|
|
||||||
|
|
||||||
// 6. Stamp the entity tree: which source generation the canonical tree
|
// 7. Stamp the entity tree: which source generation the canonical tree
|
||||||
// reflects + the rollup invariants that verify it whole (the counters
|
// reflects + the rollup invariants that verify it whole (the counters
|
||||||
// persisted in step 1). Written at flush boundaries — the tree tracks
|
// persisted in step 1). Written at flush boundaries — the tree tracks
|
||||||
// every commit by construction, so the stamp is a durable checkpoint,
|
// every commit by construction, so the stamp is a durable checkpoint,
|
||||||
|
|
@ -14337,118 +14221,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return this.embedder(textToEmbed)
|
return this.embedder(textToEmbed)
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Eagerly load/fault-in the vector index, metadata index, and graph
|
|
||||||
* adjacency so the FIRST real operation after this call runs at
|
|
||||||
* steady-state cost — no page-cache-miss / demand-load latency on the
|
|
||||||
* critical path. Complements {@link warmupEmbeddings} (which warms the
|
|
||||||
* embedding engine, not the storage indexes).
|
|
||||||
*
|
|
||||||
* Sequence per surface:
|
|
||||||
* - **Vector**: calls the provider's own `warm?()` when the active
|
|
||||||
* `'vector'` provider implements it (`'warmed'`). Otherwise runs a
|
|
||||||
* best-effort probe — one `search()` with a deterministic unit vector of
|
|
||||||
* the index's known dimension, `k = min(100, size())` — which faults in
|
|
||||||
* *some* backing storage as a side effect but is reported honestly as
|
|
||||||
* `'probed'`, never `'warmed'`. An empty index or unknown dimension has
|
|
||||||
* nothing to probe (`'unavailable'`).
|
|
||||||
* - **Metadata**: full hydration — every persisted field's sparse index is
|
|
||||||
* loaded from storage (`MetadataIndexManager.hydrateAll()`), not just the
|
|
||||||
* heuristic common-fields subset `init()` warms.
|
|
||||||
* - **Graph**: calls the provider's own `warm?()` when the active graph
|
|
||||||
* provider implements it; otherwise re-runs its existing eager cold-load
|
|
||||||
* `init()` seam (idempotent — the JS adjacency index's `init()` already
|
|
||||||
* loads the full LSM manifests + SSTables, so re-invoking it here is a
|
|
||||||
* genuine full hydration, not a stat call).
|
|
||||||
*
|
|
||||||
* A single surface's `'unavailable'`/probe outcome never fails the whole
|
|
||||||
* call — `warm()` is a best-effort readiness step, not a correctness gate;
|
|
||||||
* queries still demand-load normally regardless of what `warm()` achieved.
|
|
||||||
*
|
|
||||||
* @returns A {@link WarmReport}: honest per-surface outcome + timing.
|
|
||||||
* @example
|
|
||||||
* ```typescript
|
|
||||||
* const brain = new Brainy({ storage: { path: '/data' } })
|
|
||||||
* await brain.init()
|
|
||||||
* const report = await brain.warm() // blocking, explicit control over timing
|
|
||||||
* console.log(report.vector.outcome, report.vector.durationMs)
|
|
||||||
*
|
|
||||||
* // Or opt into the same work automatically during init():
|
|
||||||
* const eager = new Brainy({ storage: { path: '/data' }, warmOnOpen: true })
|
|
||||||
* await eager.init() // warm() already ran before this resolves
|
|
||||||
* ```
|
|
||||||
*/
|
|
||||||
async warm(): Promise<WarmReport> {
|
|
||||||
await this.ensureInitialized({ needs: ['vector', 'metadata', 'graph'] })
|
|
||||||
|
|
||||||
const totalStart = Date.now()
|
|
||||||
|
|
||||||
// --- Vector ---------------------------------------------------------
|
|
||||||
const vectorStart = Date.now()
|
|
||||||
let vectorOutcome: WarmOutcome
|
|
||||||
const vectorProvider = this.index as VectorIndexProvider & { warm?: () => Promise<void> }
|
|
||||||
if (typeof vectorProvider.warm === 'function') {
|
|
||||||
await vectorProvider.warm()
|
|
||||||
vectorOutcome = 'warmed'
|
|
||||||
} else {
|
|
||||||
const size = this.index.size()
|
|
||||||
const dimension = this.dimensions
|
|
||||||
if (size > 0 && dimension && dimension > 0) {
|
|
||||||
// A deterministic unit vector (L2 norm 1) — same probe every call,
|
|
||||||
// no reliance on stored data shape.
|
|
||||||
const probeVector = new Array(dimension).fill(1 / Math.sqrt(dimension))
|
|
||||||
await this.index.search(probeVector, Math.min(100, size))
|
|
||||||
vectorOutcome = 'probed'
|
|
||||||
} else {
|
|
||||||
// Nothing to probe: an empty index, or the vector dimension is not
|
|
||||||
// yet known (no entity has ever been added).
|
|
||||||
vectorOutcome = 'unavailable'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const vectorDurationMs = Date.now() - vectorStart
|
|
||||||
|
|
||||||
// --- Metadata --------------------------------------------------------
|
|
||||||
const metadataStart = Date.now()
|
|
||||||
let metadataOutcome: WarmOutcome
|
|
||||||
const metadataWithHydrate = this.metadataIndex as unknown as { hydrateAll?: () => Promise<void> }
|
|
||||||
if (typeof metadataWithHydrate.hydrateAll === 'function') {
|
|
||||||
await metadataWithHydrate.hydrateAll()
|
|
||||||
metadataOutcome = 'warmed'
|
|
||||||
} else {
|
|
||||||
// No hydration seam on this metadata provider — nothing to run.
|
|
||||||
metadataOutcome = 'unavailable'
|
|
||||||
}
|
|
||||||
const metadataDurationMs = Date.now() - metadataStart
|
|
||||||
|
|
||||||
// --- Graph -------------------------------------------------------------
|
|
||||||
const graphStart = Date.now()
|
|
||||||
let graphOutcome: WarmOutcome
|
|
||||||
const graphProvider = this.graphIndex as GraphIndexProvider & {
|
|
||||||
warm?: () => Promise<void>
|
|
||||||
init?: () => Promise<void>
|
|
||||||
}
|
|
||||||
if (typeof graphProvider.warm === 'function') {
|
|
||||||
await graphProvider.warm()
|
|
||||||
graphOutcome = 'warmed'
|
|
||||||
} else if (typeof graphProvider.init === 'function') {
|
|
||||||
// Existing eager cold-load seam (readiness contract): idempotent, and
|
|
||||||
// for the JS adjacency index it's already a genuine full load of the
|
|
||||||
// LSM manifests + SSTables — a real hydration, not a stat call.
|
|
||||||
await graphProvider.init()
|
|
||||||
graphOutcome = 'warmed'
|
|
||||||
} else {
|
|
||||||
graphOutcome = 'unavailable'
|
|
||||||
}
|
|
||||||
const graphDurationMs = Date.now() - graphStart
|
|
||||||
|
|
||||||
return {
|
|
||||||
vector: { outcome: vectorOutcome, durationMs: vectorDurationMs },
|
|
||||||
metadata: { outcome: metadataOutcome, durationMs: metadataDurationMs },
|
|
||||||
graph: { outcome: graphOutcome, durationMs: graphDurationMs },
|
|
||||||
totalDurationMs: Date.now() - totalStart
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Explicitly warm up the embedding engine
|
* Explicitly warm up the embedding engine
|
||||||
*
|
*
|
||||||
|
|
@ -14967,9 +14739,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Migration LOCK wait budget — left undefined when omitted so
|
// Migration LOCK wait budget — left undefined when omitted so
|
||||||
// awaitMigrationLock() applies its 30 s default at the read site.
|
// awaitMigrationLock() applies its 30 s default at the read site.
|
||||||
migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined,
|
migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined,
|
||||||
// Apply-phase transaction budget floor — left undefined when omitted so
|
|
||||||
// transactTimeoutBudget() applies its 30 s default at the read site.
|
|
||||||
transactionBudgetFloorMs: config?.transactionBudgetFloorMs ?? undefined,
|
|
||||||
// Pre-upgrade backup — default-on; opt out with `migrationBackup: false`.
|
// Pre-upgrade backup — default-on; opt out with `migrationBackup: false`.
|
||||||
migrationBackup: config?.migrationBackup ?? true,
|
migrationBackup: config?.migrationBackup ?? true,
|
||||||
// Vector index configuration (8.0) — algorithm-neutral surface with
|
// Vector index configuration (8.0) — algorithm-neutral surface with
|
||||||
|
|
@ -14985,9 +14754,6 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// is the active one on a non-reader writer outside unit tests). Explicit
|
// is the active one on a non-reader writer outside unit tests). Explicit
|
||||||
// true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts.
|
// true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts.
|
||||||
eagerEmbeddings: config?.eagerEmbeddings ?? undefined,
|
eagerEmbeddings: config?.eagerEmbeddings ?? undefined,
|
||||||
// Eager index warm at init() — default off (lazy demand-load), the
|
|
||||||
// pre-8.9 behavior. See the warmOnOpen JSDoc in brainy.types.ts.
|
|
||||||
warmOnOpen: config?.warmOnOpen ?? false,
|
|
||||||
// Plugin configuration - undefined = auto-detect
|
// Plugin configuration - undefined = auto-detect
|
||||||
plugins: config?.plugins ?? undefined,
|
plugins: config?.plugins ?? undefined,
|
||||||
// Integration Hub - undefined/false = disabled
|
// Integration Hub - undefined/false = disabled
|
||||||
|
|
@ -16406,21 +16172,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
await this.generationStore.flushPendingSingleOps()
|
await this.generationStore.flushPendingSingleOps()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Phase 0b: REPACK cold history into sealed segments (D1+D3 —
|
// Phase 0b: Auto-compact generational history per config.retention (default
|
||||||
// re-representation, never deletion; the only history transform under the
|
// on) BEFORE the generation store closes below. Respects live Db pins and
|
||||||
// archival profile), then auto-compact per config.retention. Repack runs
|
// an explicit autoCompact: false; no-op on read-only instances.
|
||||||
// FIRST so bounded-retention reclaim can drop whole segments. Both are
|
|
||||||
// time-bounded maintenance passes (8.9.0 law: flush() never pays these);
|
|
||||||
// both are housekeeping — failures warn, never fail a clean shutdown.
|
|
||||||
if (!this.isReadOnly && this.generationStore) {
|
|
||||||
try {
|
|
||||||
await this.generationStore.repackHistory({ timeBudgetMs: 5_000 })
|
|
||||||
} catch (error) {
|
|
||||||
console.warn(
|
|
||||||
`History repacking failed (non-fatal): ${error instanceof Error ? error.message : String(error)}`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
await this.autoCompactHistory()
|
await this.autoCompactHistory()
|
||||||
|
|
||||||
// Phase 1: Flush ALL components in parallel to persist buffered data
|
// Phase 1: Flush ALL components in parallel to persist buffered data
|
||||||
|
|
|
||||||
|
|
@ -102,26 +102,12 @@ export interface FactScanBatch {
|
||||||
segmentId: string
|
segmentId: string
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Liveness bound on a scan's FIRST batch (Stage-2 co-freeze, D1 contract):
|
|
||||||
* `batches()` must yield its first batch — or fail loudly — within this many
|
|
||||||
* ms of the first pull. A backlogged or damaged store may be SLOW, but it may
|
|
||||||
* never be SILENT: a consumer awaiting the first batch is otherwise
|
|
||||||
* indistinguishable from a wedge (the exact failure shape a production heal
|
|
||||||
* hit against a generations-backlogged brain).
|
|
||||||
*/
|
|
||||||
export const SCANFACTS_FIRST_BATCH_MS = 10_000
|
|
||||||
|
|
||||||
/** The telemetry a scan OPEN returns (frozen shape). */
|
/** The telemetry a scan OPEN returns (frozen shape). */
|
||||||
export interface FactScanHandle {
|
export interface FactScanHandle {
|
||||||
headGeneration: number
|
headGeneration: number
|
||||||
segmentCount: number
|
segmentCount: number
|
||||||
approxFactCount: number
|
approxFactCount: number
|
||||||
/**
|
/** Ordered batches; a detected gap aborts LOUDLY, never a silent skip. */
|
||||||
* Ordered batches; a detected gap aborts LOUDLY, never a silent skip.
|
|
||||||
* Liveness contract: the FIRST batch resolves or rejects within
|
|
||||||
* {@link SCANFACTS_FIRST_BATCH_MS} of the first pull — never a silent hang.
|
|
||||||
*/
|
|
||||||
batches: () => AsyncGenerator<FactScanBatch>
|
batches: () => AsyncGenerator<FactScanBatch>
|
||||||
/** Close telemetry — the invariant cross-check, valid after iteration ends. */
|
/** Close telemetry — the invariant cross-check, valid after iteration ends. */
|
||||||
summary: () => { factsYielded: number; segmentsRead: number }
|
summary: () => { factsYielded: number; segmentsRead: number }
|
||||||
|
|
@ -454,8 +440,6 @@ export class FactLog {
|
||||||
toGeneration?: number
|
toGeneration?: number
|
||||||
kinds?: Array<'noun' | 'verb'>
|
kinds?: Array<'noun' | 'verb'>
|
||||||
batchSize?: number
|
batchSize?: number
|
||||||
/** Test override for the first-batch liveness bound (default {@link SCANFACTS_FIRST_BATCH_MS}). */
|
|
||||||
firstBatchTimeoutMs?: number
|
|
||||||
}): FactScanHandle {
|
}): FactScanHandle {
|
||||||
const from = options?.fromGeneration ?? 1
|
const from = options?.fromGeneration ?? 1
|
||||||
const to = options?.toGeneration ?? this.head
|
const to = options?.toGeneration ?? this.head
|
||||||
|
|
@ -530,43 +514,11 @@ export class FactLog {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Liveness wrapper: the FIRST pull races the contract deadline. Only the
|
|
||||||
// first — the bound is time-to-first-batch (proof the producer is alive),
|
|
||||||
// not per-batch pacing; and it runs only while a pull is actually pending,
|
|
||||||
// so consumer think-time between pulls never counts against the producer.
|
|
||||||
const firstBatchTimeoutMs = options?.firstBatchTimeoutMs ?? SCANFACTS_FIRST_BATCH_MS
|
|
||||||
async function* batchesWithLiveness(this: void): AsyncGenerator<FactScanBatch> {
|
|
||||||
const inner = batches()
|
|
||||||
let timer: NodeJS.Timeout | undefined
|
|
||||||
try {
|
|
||||||
const deadline = new Promise<never>((_, reject) => {
|
|
||||||
timer = setTimeout(
|
|
||||||
() =>
|
|
||||||
reject(
|
|
||||||
new Error(
|
|
||||||
`fact log: scanFacts produced no first batch within ${firstBatchTimeoutMs}ms ` +
|
|
||||||
`(liveness contract) — the store is wedged or unreadably slow; aborting scan LOUDLY ` +
|
|
||||||
`instead of hanging the consumer.`
|
|
||||||
)
|
|
||||||
),
|
|
||||||
firstBatchTimeoutMs
|
|
||||||
)
|
|
||||||
timer.unref?.()
|
|
||||||
})
|
|
||||||
const first = await Promise.race([inner.next(), deadline])
|
|
||||||
if (first.done) return
|
|
||||||
yield first.value
|
|
||||||
} finally {
|
|
||||||
clearTimeout(timer)
|
|
||||||
}
|
|
||||||
yield* inner
|
|
||||||
}
|
|
||||||
|
|
||||||
return {
|
return {
|
||||||
headGeneration: this.head,
|
headGeneration: this.head,
|
||||||
segmentCount: segments.length + (tailSnapshot.length > 0 ? 1 : 0),
|
segmentCount: segments.length + (tailSnapshot.length > 0 ? 1 : 0),
|
||||||
approxFactCount,
|
approxFactCount,
|
||||||
batches: batchesWithLiveness,
|
batches,
|
||||||
summary: () => ({ factsYielded, segmentsRead })
|
summary: () => ({ factsYielded, segmentsRead })
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,459 +0,0 @@
|
||||||
/**
|
|
||||||
* @module db/generationSegments
|
|
||||||
* @description The generation-segment store — Stage-2 D1+D3+repacking's file
|
|
||||||
* format (co-frozen 2026-07-19; design: the d1-d3-repacking spec).
|
|
||||||
*
|
|
||||||
* Packs CONSECUTIVE cold generations' record-sets (before-images + delta)
|
|
||||||
* into append-once segment files with derived sidecar indexes, so history
|
|
||||||
* scales in SEGMENTS (tens) instead of FILES-PER-GENERATION (hundreds of
|
|
||||||
* thousands), and cold-open reads ONE manifest instead of listing the
|
|
||||||
* backlog. Layout under `_generations/segments/`:
|
|
||||||
*
|
|
||||||
* - `seg-<firstGen, zero-padded 20>.bgs` — magic "BGS1", then one frame per
|
|
||||||
* generation: `u32 payloadLen | u32 crc32c | msgpack payload`. Payload is
|
|
||||||
* POSITIONAL: `[generation, timestamp, delta, records[], flags]` with
|
|
||||||
* records `[kindByte, id, record]`. `flags` reserves encoding evolution
|
|
||||||
* (bit 0 = compressed payload — v1 always 0; a future writer upgrade,
|
|
||||||
* never a format break). Sealed segments are IMMUTABLE — the fact log's
|
|
||||||
* own law, generalized.
|
|
||||||
* - `seg-<firstGen>.idx` — DERIVED sidecar (msgpack): per-generation frame
|
|
||||||
* offsets (point reads = one ranged read, never a listing) + per-id
|
|
||||||
* generation postings (per-id chain rebuilds read only what they need).
|
|
||||||
* Corrupt/missing → rebuilt from its segment in one sequential read,
|
|
||||||
* loudly.
|
|
||||||
* - `manifest.json` — the segment catalogue + `compactedBelow` (D3's
|
|
||||||
* horizon marker). Cold-open reads THIS; the packed backlog is never
|
|
||||||
* listed.
|
|
||||||
*
|
|
||||||
* D3 semantics carried here: bounded-retention reclaim drops WHOLE segments
|
|
||||||
* at boundaries (O(1) per segment, no rewrite); under the archival profile
|
|
||||||
* (`retention: 'all'`) nothing here is ever dropped — folding is the only
|
|
||||||
* transform (re-representation, never deletion).
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { encode as msgpackEncode, decode as msgpackDecode } from '@msgpack/msgpack'
|
|
||||||
import { crc32c } from '../utils/crc32c.js'
|
|
||||||
import type { FactLogStorage } from './factLog.js'
|
|
||||||
import { prodLog } from '../utils/logger.js'
|
|
||||||
|
|
||||||
/** Directory for segment files + manifest, under the generations prefix. */
|
|
||||||
export const SEGMENTS_PREFIX = '_generations/segments'
|
|
||||||
|
|
||||||
/** Target sealed-segment size (co-freeze proposal; tunable on evidence). */
|
|
||||||
export const SEGMENT_TARGET_BYTES = 64 * 1024 * 1024
|
|
||||||
|
|
||||||
const MAGIC = new TextEncoder().encode('BGS1')
|
|
||||||
const FRAME_PREFIX_BYTES = 8 // u32 payloadLen + u32 crc32c
|
|
||||||
const MANIFEST_PATH = `${SEGMENTS_PREFIX}/manifest.json`
|
|
||||||
|
|
||||||
/** One generation's fold input — exactly what the live tier holds for it. */
|
|
||||||
export interface FoldGeneration {
|
|
||||||
generation: number
|
|
||||||
timestamp: number
|
|
||||||
/** The tx.json delta object, carried verbatim. */
|
|
||||||
delta: unknown
|
|
||||||
/** The before-image record-set (empty for record-less generations). */
|
|
||||||
records: Array<{ kind: 'noun' | 'verb'; id: string; record: unknown }>
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Manifest entry for one sealed segment. */
|
|
||||||
export interface SegmentMeta {
|
|
||||||
file: string
|
|
||||||
firstGeneration: number
|
|
||||||
lastGeneration: number
|
|
||||||
frames: number
|
|
||||||
bytes: number
|
|
||||||
/** crc32c of the full segment byte stream — the digest chain's link. */
|
|
||||||
checksum: number
|
|
||||||
}
|
|
||||||
|
|
||||||
interface SegmentManifest {
|
|
||||||
version: 1
|
|
||||||
compactedBelow: number
|
|
||||||
segments: SegmentMeta[]
|
|
||||||
}
|
|
||||||
|
|
||||||
interface SidecarIndex {
|
|
||||||
version: 1
|
|
||||||
/** [generation, frameOffset, frameLen] ascending by generation. */
|
|
||||||
generations: Array<[number, number, number]>
|
|
||||||
/** `${kindByte}:${id}` → ascending generations holding a record for it. */
|
|
||||||
ids: Record<string, number[]>
|
|
||||||
}
|
|
||||||
|
|
||||||
const segmentFileName = (firstGeneration: number): string =>
|
|
||||||
`seg-${String(firstGeneration).padStart(20, '0')}.bgs`
|
|
||||||
const sidecarFileName = (firstGeneration: number): string =>
|
|
||||||
`seg-${String(firstGeneration).padStart(20, '0')}.idx`
|
|
||||||
|
|
||||||
/**
|
|
||||||
* The generation-segment store. Owns the packed tier ONLY — the live
|
|
||||||
* per-generation tier and the routing between tiers belong to
|
|
||||||
* `GenerationStore`. All mutating entry points here are called under the
|
|
||||||
* generation store's commit mutex.
|
|
||||||
*/
|
|
||||||
export class GenerationSegmentStore {
|
|
||||||
private readonly storage: FactLogStorage
|
|
||||||
private manifest: SegmentManifest = { version: 1, compactedBelow: 0, segments: [] }
|
|
||||||
/** Sidecar cache — segments are immutable, so entries never invalidate. */
|
|
||||||
private readonly sidecars = new Map<string, SidecarIndex>()
|
|
||||||
|
|
||||||
constructor(storage: FactLogStorage) {
|
|
||||||
this.storage = storage
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Load the manifest (ONE read — never a directory listing). */
|
|
||||||
async open(): Promise<void> {
|
|
||||||
const raw = (await this.storage.readRawObject(MANIFEST_PATH)) as SegmentManifest | null
|
|
||||||
if (raw) {
|
|
||||||
if (raw.version !== 1) {
|
|
||||||
throw new Error(
|
|
||||||
`[GenerationSegments] manifest version ${String(raw.version)} is newer than this ` +
|
|
||||||
`engine understands — refusing to serve partial history. Upgrade the engine.`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
this.manifest = raw
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The packed tier's catalogue (ascending, immutable snapshot). */
|
|
||||||
segments(): readonly SegmentMeta[] {
|
|
||||||
return this.manifest.segments
|
|
||||||
}
|
|
||||||
|
|
||||||
/** D3's horizon marker: generations below this were reclaimed (bounded profiles only). */
|
|
||||||
compactedBelow(): number {
|
|
||||||
return this.manifest.compactedBelow
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The covering sealed segment for `gen`, or null if it lives outside the packed tier. */
|
|
||||||
private coveringSegment(gen: number): SegmentMeta | null {
|
|
||||||
// Manifest is ascending and ranges never overlap — binary search.
|
|
||||||
const segs = this.manifest.segments
|
|
||||||
let lo = 0
|
|
||||||
let hi = segs.length - 1
|
|
||||||
while (lo <= hi) {
|
|
||||||
const mid = (lo + hi) >> 1
|
|
||||||
const s = segs[mid]
|
|
||||||
if (gen < s.firstGeneration) hi = mid - 1
|
|
||||||
else if (gen > s.lastGeneration) lo = mid + 1
|
|
||||||
else return s
|
|
||||||
}
|
|
||||||
return null
|
|
||||||
}
|
|
||||||
|
|
||||||
/** True when `gen` is packed (readable from this tier). */
|
|
||||||
hasGeneration(gen: number): boolean {
|
|
||||||
return this.coveringSegment(gen) !== null
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Fold consecutive generations into ONE new sealed segment + sidecar and
|
|
||||||
* append it to the manifest atomically. Caller guarantees: `gens` is
|
|
||||||
* ascending, contiguous with the packed tier (first = last packed + 1 when
|
|
||||||
* segments exist), and already durable in the live tier. Crash between the
|
|
||||||
* segment write and the caller's live-tier delete leaves a DUPLICATE
|
|
||||||
* representation — resolved live-tier-wins by the reader; never a gap.
|
|
||||||
*/
|
|
||||||
async fold(gens: FoldGeneration[]): Promise<SegmentMeta> {
|
|
||||||
if (gens.length === 0) {
|
|
||||||
throw new Error('[GenerationSegments] fold() requires at least one generation')
|
|
||||||
}
|
|
||||||
for (let i = 1; i < gens.length; i++) {
|
|
||||||
if (gens[i].generation <= gens[i - 1].generation) {
|
|
||||||
throw new Error('[GenerationSegments] fold() input must be strictly ascending')
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const last = this.manifest.segments[this.manifest.segments.length - 1]
|
|
||||||
if (last && gens[0].generation <= last.lastGeneration) {
|
|
||||||
throw new Error(
|
|
||||||
`[GenerationSegments] fold() overlaps the packed tier: ${gens[0].generation} ≤ ` +
|
|
||||||
`sealed ${last.lastGeneration} — segments are immutable, never rewritten`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
const first = gens[0].generation
|
|
||||||
const file = segmentFileName(first)
|
|
||||||
const sidecar: SidecarIndex = { version: 1, generations: [], ids: {} }
|
|
||||||
|
|
||||||
// Encode all frames, tracking offsets for the sidecar.
|
|
||||||
const parts: Uint8Array[] = [MAGIC]
|
|
||||||
let offset = MAGIC.length
|
|
||||||
for (const g of gens) {
|
|
||||||
const payload = msgpackEncode([
|
|
||||||
g.generation,
|
|
||||||
g.timestamp,
|
|
||||||
g.delta,
|
|
||||||
g.records.map((r) => [r.kind === 'noun' ? 0 : 1, r.id, r.record]),
|
|
||||||
0 // flags: v1 = uncompressed
|
|
||||||
])
|
|
||||||
const frame = new Uint8Array(FRAME_PREFIX_BYTES + payload.length)
|
|
||||||
const view = new DataView(frame.buffer)
|
|
||||||
view.setUint32(0, payload.length, true)
|
|
||||||
view.setUint32(4, crc32c(payload), true)
|
|
||||||
frame.set(payload, FRAME_PREFIX_BYTES)
|
|
||||||
sidecar.generations.push([g.generation, offset, frame.length])
|
|
||||||
for (const r of g.records) {
|
|
||||||
const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}`
|
|
||||||
;(sidecar.ids[key] ??= []).push(g.generation)
|
|
||||||
}
|
|
||||||
parts.push(frame)
|
|
||||||
offset += frame.length
|
|
||||||
}
|
|
||||||
const total = parts.reduce((n, p) => n + p.length, 0)
|
|
||||||
const bytes = new Uint8Array(total)
|
|
||||||
let at = 0
|
|
||||||
for (const p of parts) {
|
|
||||||
bytes.set(p, at)
|
|
||||||
at += p.length
|
|
||||||
}
|
|
||||||
|
|
||||||
const meta: SegmentMeta = {
|
|
||||||
file,
|
|
||||||
firstGeneration: first,
|
|
||||||
lastGeneration: gens[gens.length - 1].generation,
|
|
||||||
frames: gens.length,
|
|
||||||
bytes: total,
|
|
||||||
checksum: crc32c(bytes)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Durability order: segment + sidecar fsync'd BEFORE the manifest names
|
|
||||||
// them (a crash before the manifest = invisible orphan files, harmless);
|
|
||||||
// manifest last, atomically.
|
|
||||||
const segPath = `${SEGMENTS_PREFIX}/${file}`
|
|
||||||
const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(first)}`
|
|
||||||
await this.storage.writeRawBytes(segPath, bytes)
|
|
||||||
await this.storage.writeRawBytes(idxPath, msgpackEncode(sidecar))
|
|
||||||
await this.storage.syncRawObjects([segPath, idxPath])
|
|
||||||
const next: SegmentManifest = {
|
|
||||||
...this.manifest,
|
|
||||||
segments: [...this.manifest.segments, meta]
|
|
||||||
}
|
|
||||||
await this.storage.writeRawObject(MANIFEST_PATH, next)
|
|
||||||
await this.storage.syncRawObjects([MANIFEST_PATH])
|
|
||||||
this.manifest = next
|
|
||||||
this.sidecars.set(file, sidecar)
|
|
||||||
return meta
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Load (or rebuild, loudly) a segment's sidecar. */
|
|
||||||
private async sidecarFor(meta: SegmentMeta): Promise<SidecarIndex> {
|
|
||||||
const cached = this.sidecars.get(meta.file)
|
|
||||||
if (cached) return cached
|
|
||||||
const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(meta.firstGeneration)}`
|
|
||||||
const raw = await this.storage.readRawBytes(idxPath)
|
|
||||||
if (raw) {
|
|
||||||
try {
|
|
||||||
const idx = msgpackDecode(raw) as SidecarIndex
|
|
||||||
if (idx.version === 1) {
|
|
||||||
this.sidecars.set(meta.file, idx)
|
|
||||||
return idx
|
|
||||||
}
|
|
||||||
} catch {
|
|
||||||
// fall through to rebuild
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// Sidecars are DERIVED: rebuild from the segment, loudly — never serve
|
|
||||||
// wrong offsets silently.
|
|
||||||
prodLog.warn(
|
|
||||||
`[GenerationSegments] sidecar for ${meta.file} missing or unreadable — rebuilding from the segment`
|
|
||||||
)
|
|
||||||
const rebuilt = await this.rebuildSidecar(meta)
|
|
||||||
await this.storage.writeRawBytes(idxPath, msgpackEncode(rebuilt))
|
|
||||||
this.sidecars.set(meta.file, rebuilt)
|
|
||||||
return rebuilt
|
|
||||||
}
|
|
||||||
|
|
||||||
/** One sequential read of the segment → a fresh sidecar. Verifies every frame CRC. */
|
|
||||||
private async rebuildSidecar(meta: SegmentMeta): Promise<SidecarIndex> {
|
|
||||||
const frames = await this.readAllFrames(meta)
|
|
||||||
const idx: SidecarIndex = { version: 1, generations: [], ids: {} }
|
|
||||||
for (const f of frames) {
|
|
||||||
idx.generations.push([f.generation, f.offset, f.frameLen])
|
|
||||||
for (const r of f.records) {
|
|
||||||
const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}`
|
|
||||||
;(idx.ids[key] ??= []).push(f.generation)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return idx
|
|
||||||
}
|
|
||||||
|
|
||||||
private decodeFrame(
|
|
||||||
payload: Uint8Array
|
|
||||||
): { generation: number; timestamp: number; delta: unknown; records: FoldGeneration['records'] } {
|
|
||||||
const [generation, timestamp, delta, rawRecords] = msgpackDecode(payload) as [
|
|
||||||
number,
|
|
||||||
number,
|
|
||||||
unknown,
|
|
||||||
Array<[number, string, unknown]>,
|
|
||||||
number
|
|
||||||
]
|
|
||||||
return {
|
|
||||||
generation,
|
|
||||||
timestamp,
|
|
||||||
delta,
|
|
||||||
records: rawRecords.map(([kindByte, id, record]) => ({
|
|
||||||
kind: kindByte === 0 ? ('noun' as const) : ('verb' as const),
|
|
||||||
id,
|
|
||||||
record
|
|
||||||
}))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private async readAllFrames(meta: SegmentMeta): Promise<
|
|
||||||
Array<ReturnType<GenerationSegmentStore['decodeFrame']> & { offset: number; frameLen: number }>
|
|
||||||
> {
|
|
||||||
const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`)
|
|
||||||
if (!bytes) {
|
|
||||||
throw new Error(
|
|
||||||
`[GenerationSegments] sealed segment ${meta.file} is MISSING — packed history is damaged; ` +
|
|
||||||
`refusing to continue silently`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
const out: Array<ReturnType<GenerationSegmentStore['decodeFrame']> & { offset: number; frameLen: number }> = []
|
|
||||||
let at = MAGIC.length
|
|
||||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
|
|
||||||
while (at + FRAME_PREFIX_BYTES <= bytes.length) {
|
|
||||||
const payloadLen = view.getUint32(at, true)
|
|
||||||
const crc = view.getUint32(at + 4, true)
|
|
||||||
const payload = bytes.subarray(at + FRAME_PREFIX_BYTES, at + FRAME_PREFIX_BYTES + payloadLen)
|
|
||||||
if (payload.length !== payloadLen || crc32c(payload) !== crc) {
|
|
||||||
throw new Error(
|
|
||||||
`[GenerationSegments] frame CRC mismatch in ${meta.file} at offset ${at} — ` +
|
|
||||||
`packed history is damaged; refusing to serve it`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
out.push({ ...this.decodeFrame(payload), offset: at, frameLen: FRAME_PREFIX_BYTES + payloadLen })
|
|
||||||
at += FRAME_PREFIX_BYTES + payloadLen
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Read one packed generation's frame via its sidecar offset (one ranged read). */
|
|
||||||
private async readFrame(
|
|
||||||
gen: number
|
|
||||||
): Promise<ReturnType<GenerationSegmentStore['decodeFrame']> | null> {
|
|
||||||
const meta = this.coveringSegment(gen)
|
|
||||||
if (!meta) return null
|
|
||||||
const idx = await this.sidecarFor(meta)
|
|
||||||
// generations ascending → binary search.
|
|
||||||
const gens = idx.generations
|
|
||||||
let lo = 0
|
|
||||||
let hi = gens.length - 1
|
|
||||||
while (lo <= hi) {
|
|
||||||
const mid = (lo + hi) >> 1
|
|
||||||
if (gens[mid][0] < gen) lo = mid + 1
|
|
||||||
else if (gens[mid][0] > gen) hi = mid - 1
|
|
||||||
else {
|
|
||||||
const [, offset, frameLen] = gens[mid]
|
|
||||||
const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`)
|
|
||||||
if (!bytes) {
|
|
||||||
throw new Error(`[GenerationSegments] sealed segment ${meta.file} is MISSING`)
|
|
||||||
}
|
|
||||||
const frame = bytes.subarray(offset, offset + frameLen)
|
|
||||||
const view = new DataView(frame.buffer, frame.byteOffset, frame.byteLength)
|
|
||||||
const payloadLen = view.getUint32(0, true)
|
|
||||||
const crc = view.getUint32(4, true)
|
|
||||||
const payload = frame.subarray(FRAME_PREFIX_BYTES, FRAME_PREFIX_BYTES + payloadLen)
|
|
||||||
if (payload.length !== payloadLen || crc32c(payload) !== crc) {
|
|
||||||
throw new Error(
|
|
||||||
`[GenerationSegments] frame CRC mismatch for generation ${gen} in ${meta.file} — ` +
|
|
||||||
`packed history is damaged; refusing to serve it`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
return this.decodeFrame(payload)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// In the covering range but not present: the packed tier is dense by
|
|
||||||
// construction (fold packs every generation it is handed, including
|
|
||||||
// record-less ones) — absence inside a sealed range is damage.
|
|
||||||
throw new Error(
|
|
||||||
`[GenerationSegments] generation ${gen} is inside sealed segment ${meta.file}'s declared ` +
|
|
||||||
`range but has no frame — packed history is damaged`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The packed tier's delta for `gen` (null = not packed). */
|
|
||||||
async readDelta(gen: number): Promise<{ delta: unknown; timestamp: number } | null> {
|
|
||||||
const frame = await this.readFrame(gen)
|
|
||||||
return frame ? { delta: frame.delta, timestamp: frame.timestamp } : null
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The packed tier's full record-set for `gen` (null = not packed). */
|
|
||||||
async readRecords(gen: number): Promise<FoldGeneration['records'] | null> {
|
|
||||||
const frame = await this.readFrame(gen)
|
|
||||||
return frame ? frame.records : null
|
|
||||||
}
|
|
||||||
|
|
||||||
/** One packed before-image (null = not packed OR no record for the id in that generation). */
|
|
||||||
async readRecord(gen: number, kind: 'noun' | 'verb', id: string): Promise<unknown | null> {
|
|
||||||
const frame = await this.readFrame(gen)
|
|
||||||
if (!frame) return null
|
|
||||||
const hit = frame.records.find((r) => r.kind === kind && r.id === id)
|
|
||||||
return hit ? hit.record : null
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* D3 reclaim: drop WHOLE segments whose lastGeneration < `belowGeneration`
|
|
||||||
* and bump `compactedBelow`. Partial segments are never dropped — the
|
|
||||||
* boundary waits. NEVER called under the archival profile (the caller
|
|
||||||
* enforces retention semantics; this method only executes boundary drops).
|
|
||||||
*/
|
|
||||||
async dropSegmentsBelow(belowGeneration: number): Promise<{ dropped: number; compactedBelow: number }> {
|
|
||||||
const keep: SegmentMeta[] = []
|
|
||||||
const drop: SegmentMeta[] = []
|
|
||||||
for (const s of this.manifest.segments) {
|
|
||||||
;(s.lastGeneration < belowGeneration ? drop : keep).push(s)
|
|
||||||
}
|
|
||||||
if (drop.length === 0) {
|
|
||||||
return { dropped: 0, compactedBelow: this.manifest.compactedBelow }
|
|
||||||
}
|
|
||||||
const compactedBelow = Math.max(
|
|
||||||
this.manifest.compactedBelow,
|
|
||||||
drop[drop.length - 1].lastGeneration + 1
|
|
||||||
)
|
|
||||||
// Manifest first (the drop is authoritative once named), then bytes —
|
|
||||||
// a crash between leaves orphan segment files invisible to the manifest,
|
|
||||||
// harmless and re-collectable.
|
|
||||||
const next: SegmentManifest = { ...this.manifest, compactedBelow, segments: keep }
|
|
||||||
await this.storage.writeRawObject(MANIFEST_PATH, next)
|
|
||||||
await this.storage.syncRawObjects([MANIFEST_PATH])
|
|
||||||
this.manifest = next
|
|
||||||
for (const s of drop) {
|
|
||||||
await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${s.file}`)
|
|
||||||
await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${sidecarFileName(s.firstGeneration)}`)
|
|
||||||
this.sidecars.delete(s.file)
|
|
||||||
}
|
|
||||||
return { dropped: drop.length, compactedBelow }
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* D8 rider — the packed portion of `generationDigest(g)`: a deterministic
|
|
||||||
* crc32c chain over sealed-segment checksums fully below `g`, plus the
|
|
||||||
* frame CRC of `g`'s own frame when `g` is mid-segment. O(segments), not
|
|
||||||
* O(generations); identical history ⇒ identical digest on any machine.
|
|
||||||
* The live-tier portion is composed by the caller.
|
|
||||||
*/
|
|
||||||
async digestThroughPacked(g: number): Promise<number | null> {
|
|
||||||
let digest = 0
|
|
||||||
let covered = false
|
|
||||||
for (const s of this.manifest.segments) {
|
|
||||||
if (s.lastGeneration <= g) {
|
|
||||||
digest = crc32c(new TextEncoder().encode(`${digest}:${s.checksum}`))
|
|
||||||
if (s.lastGeneration === g) covered = true
|
|
||||||
} else if (s.firstGeneration <= g) {
|
|
||||||
// g is mid-segment: chain the partial prefix via g's frame CRC.
|
|
||||||
const frame = await this.readFrame(g)
|
|
||||||
if (frame === null) return null
|
|
||||||
const idx = await this.sidecarFor(s)
|
|
||||||
const upTo = idx.generations.filter(([gen]) => gen <= g)
|
|
||||||
for (const [gen, offset, frameLen] of upTo) {
|
|
||||||
digest = crc32c(new TextEncoder().encode(`${digest}:${gen}:${offset}:${frameLen}`))
|
|
||||||
}
|
|
||||||
covered = true
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return covered || this.manifest.segments.length > 0 ? digest : null
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -46,8 +46,6 @@ import type {
|
||||||
TxLogEntry
|
TxLogEntry
|
||||||
} from './types.js'
|
} from './types.js'
|
||||||
import { FactLog, storageSupportsFactLog, type CommitFact, type FactOp } from './factLog.js'
|
import { FactLog, storageSupportsFactLog, type CommitFact, type FactOp } from './factLog.js'
|
||||||
import { GenerationSegmentStore, type FoldGeneration } from './generationSegments.js'
|
|
||||||
import { crc32c } from '../utils/crc32c.js'
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The byte-identical before-images of every id a commit touches, read UNDER
|
* The byte-identical before-images of every id a commit touches, read UNDER
|
||||||
|
|
@ -268,21 +266,6 @@ export class GenerationStore {
|
||||||
*/
|
*/
|
||||||
private historyBytesTotal: number | null = null
|
private historyBytesTotal: number | null = null
|
||||||
|
|
||||||
/**
|
|
||||||
* The packed tier (D1+D3): sealed segments holding folded cold
|
|
||||||
* generations. Null until {@link open} wires it (and on storage adapters
|
|
||||||
* without raw-byte primitives — the live tier then carries everything,
|
|
||||||
* exactly as before the packed tier existed).
|
|
||||||
*/
|
|
||||||
private segments: GenerationSegmentStore | null = null
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Live-tier window: generations newer than `committed - REPACK_LIVE_WINDOW`
|
|
||||||
* are never folded — the hot tail stays in the per-generation layout the
|
|
||||||
* write path owns. Matches the resident chain window's scale.
|
|
||||||
*/
|
|
||||||
static readonly REPACK_LIVE_WINDOW = 1024
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Model-B per-write group-commit — the in-memory PENDING tier.
|
* Model-B per-write group-commit — the in-memory PENDING tier.
|
||||||
*
|
*
|
||||||
|
|
@ -450,33 +433,6 @@ export class GenerationStore {
|
||||||
this.factLog = null
|
this.factLog = null
|
||||||
}
|
}
|
||||||
|
|
||||||
// PACKED TIER (D1+D3): same capability gate as the fact log. Opening
|
|
||||||
// reads ONE manifest — never a listing of the packed backlog — and seeds
|
|
||||||
// committedRanges with the sealed ranges so packed generations resolve
|
|
||||||
// exactly like live ones.
|
|
||||||
if (storageSupportsFactLog(this.storage)) {
|
|
||||||
this.segments = new GenerationSegmentStore(this.storage)
|
|
||||||
await this.segments.open()
|
|
||||||
const packedRanges = this.segments
|
|
||||||
.segments()
|
|
||||||
.map((s): [number, number] => [s.firstGeneration, Math.min(s.lastGeneration, this.committed)])
|
|
||||||
.filter(([lo, hi]) => lo <= hi)
|
|
||||||
if (packedRanges.length > 0) {
|
|
||||||
// Merge packed (older) + live (newer) interval sets — both ascending;
|
|
||||||
// coalesce adjacency so range arithmetic stays interval-exact.
|
|
||||||
const merged: Array<[number, number]> = []
|
|
||||||
for (const r of [...packedRanges, ...this.committedRanges].sort((a, b) => a[0] - b[0])) {
|
|
||||||
const last = merged[merged.length - 1]
|
|
||||||
if (last && r[0] <= last[1] + 1) last[1] = Math.max(last[1], r[1])
|
|
||||||
else merged.push([r[0], r[1]])
|
|
||||||
}
|
|
||||||
this.committedRanges = merged
|
|
||||||
}
|
|
||||||
this.horizonGen = Math.max(this.horizonGen, this.segments.compactedBelow() - 1)
|
|
||||||
} else {
|
|
||||||
this.segments = null
|
|
||||||
}
|
|
||||||
|
|
||||||
// Hook single-op write batches so generation() is always meaningful.
|
// Hook single-op write batches so generation() is always meaningful.
|
||||||
// Suppressed while a transact batch executes (the batch is ONE generation).
|
// Suppressed while a transact batch executes (the batch is ONE generation).
|
||||||
if (!options?.readOnly) {
|
if (!options?.readOnly) {
|
||||||
|
|
@ -544,51 +500,6 @@ export class GenerationStore {
|
||||||
* deltas (cache-bounded reads).
|
* deltas (cache-bounded reads).
|
||||||
* @returns Counts, bytes, generation range, and the compaction horizon.
|
* @returns Counts, bytes, generation range, and the compaction horizon.
|
||||||
*/
|
*/
|
||||||
/**
|
|
||||||
* @description D8 (gate-to-generation provenance): a deterministic content
|
|
||||||
* digest of the generation log THROUGH `g` — identical history ⇒ identical
|
|
||||||
* digest on any machine; any divergence (different records, different
|
|
||||||
* order, reclaimed range) ⇒ different digest. Composed from the packed
|
|
||||||
* tier's sealed-segment checksum chain (O(segments)) plus the live tier's
|
|
||||||
* per-generation delta digests (O(live window at most)). Release gates pin
|
|
||||||
* {generation, digest} and verify both at execution time.
|
|
||||||
* @param g - The generation to digest through (≤ committed).
|
|
||||||
* @returns A hex digest string, stable across reopen and repacking states
|
|
||||||
* ONLY for fully-packed prefixes — repacking changes representation, so
|
|
||||||
* the composed digest is defined over CONTENT: live-tier gens hash their
|
|
||||||
* delta + record ids, packed gens hash via frame CRCs. A gate should pin
|
|
||||||
* after a repack pass for long-term stability, or re-pin on repack.
|
|
||||||
*/
|
|
||||||
async generationDigest(g: number): Promise<string> {
|
|
||||||
if (!Number.isInteger(g) || g < 1 || g > this.committed) {
|
|
||||||
throw new RangeError(
|
|
||||||
`generationDigest(): generation ${g} is out of range [1, ${this.committed}]`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
if (g <= this.horizonGen) {
|
|
||||||
throw new GenerationCompactedError(g, this.horizonGen)
|
|
||||||
}
|
|
||||||
let digest = 0
|
|
||||||
const enc = new TextEncoder()
|
|
||||||
if (this.segments) {
|
|
||||||
const packed = await this.segments.digestThroughPacked(g)
|
|
||||||
if (packed !== null) digest = packed
|
|
||||||
}
|
|
||||||
// Live-tier composition: every committed gen ≤ g not covered by a sealed
|
|
||||||
// segment hashes its delta content in ascending order.
|
|
||||||
for (const gen of this.committedGensAsc()) {
|
|
||||||
if (gen > g) break
|
|
||||||
if (this.segments?.hasGeneration(gen)) continue
|
|
||||||
const delta = await this.getDelta(gen)
|
|
||||||
digest = crc32c(
|
|
||||||
enc.encode(
|
|
||||||
`${digest}:${gen}:${delta.timestamp}:${[...delta.nouns].sort().join(',')}:${[...delta.verbs].sort().join(',')}`
|
|
||||||
)
|
|
||||||
)
|
|
||||||
}
|
|
||||||
return digest.toString(16).padStart(8, '0')
|
|
||||||
}
|
|
||||||
|
|
||||||
async historyStats(): Promise<{
|
async historyStats(): Promise<{
|
||||||
generations: number
|
generations: number
|
||||||
bytes: number
|
bytes: number
|
||||||
|
|
@ -627,17 +538,14 @@ export class GenerationStore {
|
||||||
try {
|
try {
|
||||||
paths = await this.storage.listRawObjects(`${GENERATIONS_PREFIX}/${gen}/prev`)
|
paths = await this.storage.listRawObjects(`${GENERATIONS_PREFIX}/${gen}/prev`)
|
||||||
} catch {
|
} catch {
|
||||||
paths = []
|
return []
|
||||||
}
|
}
|
||||||
const records: GenerationRecord[] = []
|
const records: GenerationRecord[] = []
|
||||||
for (const p of paths) {
|
for (const p of paths) {
|
||||||
const record = (await this.storage.readRawObject(p)) as GenerationRecord | null
|
const record = (await this.storage.readRawObject(p)) as GenerationRecord | null
|
||||||
if (record) records.push(record)
|
if (record) records.push(record)
|
||||||
}
|
}
|
||||||
if (records.length > 0) return records
|
return records
|
||||||
// Two-tier: folded generations serve their record-set from the segment.
|
|
||||||
const packed = await this.segments?.readRecords(gen)
|
|
||||||
return packed ? (packed.map((r) => r.record) as GenerationRecord[]) : []
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -1875,15 +1783,9 @@ export class GenerationStore {
|
||||||
if (pending) {
|
if (pending) {
|
||||||
return (kind === 'noun' ? pending.nouns : pending.verbs).get(id) ?? null
|
return (kind === 'noun' ? pending.nouns : pending.verbs).get(id) ?? null
|
||||||
}
|
}
|
||||||
const live = (await this.storage.readRawObject(
|
return (await this.storage.readRawObject(
|
||||||
`${GENERATIONS_PREFIX}/${gen}/prev/${id}.json`
|
`${GENERATIONS_PREFIX}/${gen}/prev/${id}.json`
|
||||||
)) as GenerationRecord | null
|
)) as GenerationRecord | null
|
||||||
if (live) return live
|
|
||||||
// Two-tier: the packed tier serves folded generations (live-tier-wins).
|
|
||||||
if (this.segments?.hasGeneration(gen)) {
|
|
||||||
return (await this.segments.readRecord(gen, kind, id)) as GenerationRecord | null
|
|
||||||
}
|
|
||||||
return null
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -2230,21 +2132,6 @@ export class GenerationStore {
|
||||||
`${GENERATIONS_PREFIX}/${gen}/tx.json`
|
`${GENERATIONS_PREFIX}/${gen}/tx.json`
|
||||||
)) as GenerationDelta | null
|
)) as GenerationDelta | null
|
||||||
if (delta === null) {
|
if (delta === null) {
|
||||||
// Two-tier read (D1+D3): not in the live tier → the packed tier.
|
|
||||||
// Live-tier-wins ordering (a crash mid-fold leaves a duplicate, never
|
|
||||||
// a gap), so the segment lookup runs only after the live miss.
|
|
||||||
const packed = await this.segments?.readDelta(gen)
|
|
||||||
if (packed) {
|
|
||||||
const d = packed.delta as GenerationDelta
|
|
||||||
const entry = {
|
|
||||||
nouns: new Set(d.nouns),
|
|
||||||
verbs: new Set(d.verbs),
|
|
||||||
timestamp: packed.timestamp,
|
|
||||||
bytes: d.bytes ?? 0
|
|
||||||
}
|
|
||||||
this.setDelta(gen, entry)
|
|
||||||
return entry
|
|
||||||
}
|
|
||||||
throw new Error(
|
throw new Error(
|
||||||
`Generation delta missing: ${GENERATIONS_PREFIX}/${gen}/tx.json ` +
|
`Generation delta missing: ${GENERATIONS_PREFIX}/${gen}/tx.json ` +
|
||||||
`(store corrupted or records removed outside compactHistory())`
|
`(store corrupted or records removed outside compactHistory())`
|
||||||
|
|
@ -2326,94 +2213,6 @@ export class GenerationStore {
|
||||||
* @param options - Retention caps (see {@link CompactHistoryOptions}).
|
* @param options - Retention caps (see {@link CompactHistoryOptions}).
|
||||||
* @returns Count of removed record-sets and the new horizon.
|
* @returns Count of removed record-sets and the new horizon.
|
||||||
*/
|
*/
|
||||||
/**
|
|
||||||
* @description The REPACKER (D1+D3+repacking): fold cold live-tier
|
|
||||||
* generations into sealed segments — re-representation, never deletion.
|
|
||||||
* Every record and delta stays readable (asOf/chains unchanged); the
|
|
||||||
* per-generation directories are deleted only AFTER their segment is
|
|
||||||
* durable (crash between = duplicate representation, resolved
|
|
||||||
* live-tier-wins by every reader; never a gap). This is the transform that
|
|
||||||
* takes a 70k-file history to tens of segment files, and the ONLY history
|
|
||||||
* transform permitted under the archival profile.
|
|
||||||
*
|
|
||||||
* Folds oldest-first, contiguous from the packed boundary, in batches, and
|
|
||||||
* stops at the live window ({@link GenerationStore.REPACK_LIVE_WINDOW})
|
|
||||||
* or when `timeBudgetMs` is spent — an early stop is a consistent prefix;
|
|
||||||
* the next pass resumes.
|
|
||||||
*/
|
|
||||||
async repackHistory(options?: { timeBudgetMs?: number; batchGenerations?: number }): Promise<{
|
|
||||||
foldedGenerations: number
|
|
||||||
segmentsCreated: number
|
|
||||||
}> {
|
|
||||||
if (!this.segments) return { foldedGenerations: 0, segmentsCreated: 0 }
|
|
||||||
const segments = this.segments
|
|
||||||
return this.withMutex(async () => {
|
|
||||||
const deadline =
|
|
||||||
options?.timeBudgetMs !== undefined ? Date.now() + options.timeBudgetMs : undefined
|
|
||||||
const batchSize = options?.batchGenerations ?? 512
|
|
||||||
const coldCeiling = this.committed - GenerationStore.REPACK_LIVE_WINDOW
|
|
||||||
const packedThrough =
|
|
||||||
segments.segments().length > 0
|
|
||||||
? segments.segments()[segments.segments().length - 1].lastGeneration
|
|
||||||
: 0
|
|
||||||
|
|
||||||
// Cold, unpacked, committed generations — ascending, contiguous scan.
|
|
||||||
const eligible: number[] = []
|
|
||||||
for (const gen of this.committedGensAsc()) {
|
|
||||||
if (gen > coldCeiling) break
|
|
||||||
if (gen <= packedThrough) continue // already packed (dup fold barred)
|
|
||||||
if (this.pendingBuffer.has(gen)) continue // un-flushed = live by definition
|
|
||||||
eligible.push(gen)
|
|
||||||
}
|
|
||||||
|
|
||||||
let folded = 0
|
|
||||||
let segmentsCreated = 0
|
|
||||||
for (let i = 0; i < eligible.length; i += batchSize) {
|
|
||||||
if (deadline !== undefined && Date.now() >= deadline) break
|
|
||||||
const batch = eligible.slice(i, i + batchSize)
|
|
||||||
const foldInput: FoldGeneration[] = []
|
|
||||||
for (const gen of batch) {
|
|
||||||
const delta = (await this.storage.readRawObject(
|
|
||||||
`${GENERATIONS_PREFIX}/${gen}/tx.json`
|
|
||||||
)) as GenerationDelta | null
|
|
||||||
if (delta === null) {
|
|
||||||
// Already folded by a prior crashed pass whose dirs were removed,
|
|
||||||
// or damage — getDelta's two-tier read decides which, loudly,
|
|
||||||
// when someone asks. Skip; never fold a generation we cannot read.
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
const records: FoldGeneration['records'] = []
|
|
||||||
for (const [kind, ids] of [
|
|
||||||
['noun', delta.nouns] as const,
|
|
||||||
['verb', delta.verbs] as const
|
|
||||||
]) {
|
|
||||||
for (const id of ids) {
|
|
||||||
const record = await this.storage.readRawObject(
|
|
||||||
`${GENERATIONS_PREFIX}/${gen}/prev/${id}.json`
|
|
||||||
)
|
|
||||||
if (record) records.push({ kind, id, record })
|
|
||||||
}
|
|
||||||
}
|
|
||||||
foldInput.push({ generation: gen, timestamp: delta.timestamp, delta, records })
|
|
||||||
}
|
|
||||||
if (foldInput.length === 0) continue
|
|
||||||
await segments.fold(foldInput)
|
|
||||||
segmentsCreated++
|
|
||||||
// Segment + manifest durable → the live copies retire.
|
|
||||||
for (const g of foldInput) {
|
|
||||||
await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${g.generation}`)
|
|
||||||
}
|
|
||||||
folded += foldInput.length
|
|
||||||
}
|
|
||||||
if (folded > 0) {
|
|
||||||
prodLog.info(
|
|
||||||
`[GenerationStore] repacked ${folded} cold generation(s) into ${segmentsCreated} segment(s) — history preserved, file count reduced`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
return { foldedGenerations: folded, segmentsCreated }
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
async compact(options?: CompactHistoryOptions): Promise<CompactHistoryResult> {
|
async compact(options?: CompactHistoryOptions): Promise<CompactHistoryResult> {
|
||||||
return this.withMutex(async () => {
|
return this.withMutex(async () => {
|
||||||
const minPinned = this.minPinnedGeneration()
|
const minPinned = this.minPinnedGeneration()
|
||||||
|
|
@ -2421,11 +2220,6 @@ export class GenerationStore {
|
||||||
const maxAge = options?.maxAge
|
const maxAge = options?.maxAge
|
||||||
const maxBytes = options?.maxBytes
|
const maxBytes = options?.maxBytes
|
||||||
const ageCutoff = maxAge !== undefined ? Date.now() - maxAge : undefined
|
const ageCutoff = maxAge !== undefined ? Date.now() - maxAge : undefined
|
||||||
// Bounded maintenance pass (8.9.0): stop reclaiming once the budget is
|
|
||||||
// spent. Safe mid-loop — reclamation is oldest-first, so an early stop
|
|
||||||
// leaves a consistent contiguous prefix and the next pass resumes.
|
|
||||||
const deadline =
|
|
||||||
options?.timeBudgetMs !== undefined ? Date.now() + options.timeBudgetMs : undefined
|
|
||||||
const noCaps =
|
const noCaps =
|
||||||
maxGenerations === undefined && maxAge === undefined && maxBytes === undefined
|
maxGenerations === undefined && maxAge === undefined && maxBytes === undefined
|
||||||
|
|
||||||
|
|
@ -2439,7 +2233,6 @@ export class GenerationStore {
|
||||||
for (const gen of [...this.committedGensAsc()]) {
|
for (const gen of [...this.committedGensAsc()]) {
|
||||||
// Pins are always exempt: never reclaim a generation a live pin needs.
|
// Pins are always exempt: never reclaim a generation a live pin needs.
|
||||||
if (gen > minPinned) break // committedGensAsc ascending → nothing newer is eligible either
|
if (gen > minPinned) break // committedGensAsc ascending → nothing newer is eligible either
|
||||||
if (deadline !== undefined && Date.now() >= deadline) break // budget spent — resume next pass
|
|
||||||
const delta = await this.getDelta(gen)
|
const delta = await this.getDelta(gen)
|
||||||
if (!noCaps) {
|
if (!noCaps) {
|
||||||
const violatesCount = maxGenerations !== undefined && remainingCount > maxGenerations
|
const violatesCount = maxGenerations !== undefined && remainingCount > maxGenerations
|
||||||
|
|
@ -2505,16 +2298,6 @@ export class GenerationStore {
|
||||||
// Reclaimed generations leave the per-id chains stale → rebuild on next read.
|
// Reclaimed generations leave the per-id chains stale → rebuild on next read.
|
||||||
this.invalidateChains()
|
this.invalidateChains()
|
||||||
this.horizonGen = Math.max(this.horizonGen, highestRemoved)
|
this.horizonGen = Math.max(this.horizonGen, highestRemoved)
|
||||||
// Packed-tier reclaim (D3): a packed generation's bytes live in a
|
|
||||||
// sealed segment — removeRawPrefix above was a no-op for it. Drop
|
|
||||||
// WHOLE segments now fully below the horizon; a partially-reclaimed
|
|
||||||
// segment keeps its bytes until the boundary passes it (the frozen
|
|
||||||
// partial-segments-wait rule; logical reclamation above still holds —
|
|
||||||
// the generations left committedRanges and asOf below the horizon
|
|
||||||
// throws regardless).
|
|
||||||
if (this.segments) {
|
|
||||||
await this.segments.dropSegmentsBelow(this.horizonGen + 1)
|
|
||||||
}
|
|
||||||
const manifest: GenerationManifest = {
|
const manifest: GenerationManifest = {
|
||||||
version: 1,
|
version: 1,
|
||||||
generation: this.committed,
|
generation: this.committed,
|
||||||
|
|
|
||||||
|
|
@ -176,15 +176,6 @@ export interface CompactHistoryOptions {
|
||||||
* of each surviving generation's serialized record set (`GenerationDelta.bytes`).
|
* of each surviving generation's serialized record set (`GenerationDelta.bytes`).
|
||||||
*/
|
*/
|
||||||
maxBytes?: number
|
maxBytes?: number
|
||||||
/**
|
|
||||||
* Stop reclaiming after this many milliseconds even if caps are still
|
|
||||||
* exceeded (8.9.0). Compaction is maintenance — a bounded pass keeps
|
|
||||||
* `close()` (and any explicit maintenance window) from stalling on a large
|
|
||||||
* backlog; the next pass resumes where this one stopped (reclamation is
|
|
||||||
* oldest-first, so an early stop is always a consistent prefix). Unset =
|
|
||||||
* run to completion.
|
|
||||||
*/
|
|
||||||
timeBudgetMs?: number
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -54,9 +54,6 @@ export class HnswFlushError extends Error {
|
||||||
* acceleration provider).
|
* acceleration provider).
|
||||||
*/
|
*/
|
||||||
export class JsHnswVectorIndex implements VectorIndexProvider {
|
export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
/** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.name}. */
|
|
||||||
readonly name = 'hnsw-js'
|
|
||||||
|
|
||||||
private nouns: Map<string, HNSWNoun> = new Map()
|
private nouns: Map<string, HNSWNoun> = new Map()
|
||||||
/**
|
/**
|
||||||
* Reverse adjacency: `target id → (level → set of node ids that link TO target)`.
|
* Reverse adjacency: `target id → (level → set of node ids that link TO target)`.
|
||||||
|
|
|
||||||
|
|
@ -28,9 +28,6 @@ export type { FileVersion } from './vfs/types.js'
|
||||||
|
|
||||||
// Export diagnostics result type
|
// Export diagnostics result type
|
||||||
export type { DiagnosticsResult } from './brainy.js'
|
export type { DiagnosticsResult } from './brainy.js'
|
||||||
// brain.warm() — eager index/storage readiness report (per-surface honest
|
|
||||||
// outcome + timing). See the WarmReport JSDoc in brainy.ts.
|
|
||||||
export type { WarmReport, WarmOutcome } from './brainy.js'
|
|
||||||
export type {
|
export type {
|
||||||
GraphAuditReport,
|
GraphAuditReport,
|
||||||
GraphAuditDiscrepancy
|
GraphAuditDiscrepancy
|
||||||
|
|
@ -216,7 +213,6 @@ export type {
|
||||||
CommitFact,
|
CommitFact,
|
||||||
FactOp,
|
FactOp,
|
||||||
FactScanBatch,
|
FactScanBatch,
|
||||||
SCANFACTS_FIRST_BATCH_MS,
|
|
||||||
FactScanHandle
|
FactScanHandle
|
||||||
} from './db/factLog.js'
|
} from './db/factLog.js'
|
||||||
// The generalized family stamp — which source generation a projection
|
// The generalized family stamp — which source generation a projection
|
||||||
|
|
|
||||||
|
|
@ -381,20 +381,6 @@ export interface GraphIndexProvider {
|
||||||
*/
|
*/
|
||||||
init?(): Promise<void>
|
init?(): Promise<void>
|
||||||
|
|
||||||
/**
|
|
||||||
* @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
|
|
||||||
* pretouch) so first operations run at steady-state cost. Optional;
|
|
||||||
* absence means the provider demand-loads. Distinct from `init?()`: `init`
|
|
||||||
* is called once automatically during brain startup as part of the
|
|
||||||
* readiness contract (cold-load + rebuild gating); `warm` is an explicit,
|
|
||||||
* separate readiness step a caller opts into via `brain.warm()` (or
|
|
||||||
* `warmOnOpen`) specifically to pre-pay demand-load cost that `init` left
|
|
||||||
* lazy. A provider that already loads everything eagerly in `init?()` may
|
|
||||||
* implement `warm` as a no-op or omit it — `brain.warm()` falls back to a
|
|
||||||
* best-effort read-through probe when absent.
|
|
||||||
*/
|
|
||||||
warm?(): Promise<void>
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description OPTIONAL. A native provider returns true from the moment its
|
* @description OPTIONAL. A native provider returns true from the moment its
|
||||||
* `init()` detects a large epoch-drift until its background
|
* `init()` detects a large epoch-drift until its background
|
||||||
|
|
@ -961,26 +947,6 @@ export interface AtGenerationVectors {
|
||||||
* are retired and never looked up.
|
* are retired and never looked up.
|
||||||
*/
|
*/
|
||||||
export interface VectorIndexProvider {
|
export interface VectorIndexProvider {
|
||||||
/**
|
|
||||||
* @description REQUIRED self-reported implementation identity, rendered as
|
|
||||||
* `[vector-index:<name>]` in prose log lines and stamped into the
|
|
||||||
* transaction op-name strings that surface in journals/timings (e.g.
|
|
||||||
* `AddToVectorIndex(hnsw-js)`) — see
|
|
||||||
* `src/transaction/operations/IndexOperations.ts`. A provider must name
|
|
||||||
* itself TRUTHFULLY (its own algorithm/engine, e.g. its plugin/package
|
|
||||||
* name) and must never inherit a default — guessing a backend name would
|
|
||||||
* resurrect the exact bug this field exists to prevent: an operator
|
|
||||||
* hunting an index that isn't the one actually running. The built-in JS
|
|
||||||
* index always sets this to `'hnsw-js'`; a native provider picks its own
|
|
||||||
* string. At the TypeScript level this field is required; a runtime
|
|
||||||
* instance from an older provider compiled against the previous optional
|
|
||||||
* `providerId` contract is tolerated (never crashes, never silently
|
|
||||||
* mislabeled) — see `resolveVectorProviderId` in
|
|
||||||
* `src/transaction/operations/IndexOperations.ts`, which stamps
|
|
||||||
* `'unknown-provider'` and emits one loud warning for that case.
|
|
||||||
*/
|
|
||||||
readonly name: string
|
|
||||||
|
|
||||||
addItem(item: VectorDocument): Promise<string>
|
addItem(item: VectorDocument): Promise<string>
|
||||||
removeItem(id: string): Promise<boolean>
|
removeItem(id: string): Promise<boolean>
|
||||||
search(
|
search(
|
||||||
|
|
@ -1043,20 +1009,6 @@ export interface VectorIndexProvider {
|
||||||
*/
|
*/
|
||||||
init?(): Promise<void>
|
init?(): Promise<void>
|
||||||
|
|
||||||
/**
|
|
||||||
* @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
|
|
||||||
* pretouch) so first operations run at steady-state cost. Optional;
|
|
||||||
* absence means the provider demand-loads. Distinct from `init?()`: `init`
|
|
||||||
* runs automatically once during brain startup (the cold-load + rebuild
|
|
||||||
* readiness contract above); `warm` is a separate, explicit step a caller
|
|
||||||
* opts into via `brain.warm()` (or `warmOnOpen`) to pre-pay demand-load
|
|
||||||
* cost `init` left lazy — e.g. touching every mmap page rather than just
|
|
||||||
* opening the file. A provider that already loads everything eagerly in
|
|
||||||
* `init?()` may implement `warm` as a no-op or omit it — `brain.warm()`
|
|
||||||
* falls back to a best-effort probe `search()` when absent.
|
|
||||||
*/
|
|
||||||
warm?(): Promise<void>
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description OPTIONAL honest durability signal (readiness contract,
|
* @description OPTIONAL honest durability signal (readiness contract,
|
||||||
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
|
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
|
||||||
|
|
|
||||||
|
|
@ -96,14 +96,6 @@ export class FileSystemStorage extends BaseStorage {
|
||||||
private static readonly WRITER_STALE_THRESHOLD_MS = 60_000
|
private static readonly WRITER_STALE_THRESHOLD_MS = 60_000
|
||||||
private writerLockHeartbeat?: NodeJS.Timeout
|
private writerLockHeartbeat?: NodeJS.Timeout
|
||||||
private writerLockInfo?: WriterLockInfo
|
private writerLockInfo?: WriterLockInfo
|
||||||
/**
|
|
||||||
* The currently-executing heartbeat refresh, if any. `releaseWriterLock()`
|
|
||||||
* awaits it before unlinking: clearInterval() stops FUTURE ticks but not a
|
|
||||||
* tick already in flight, and a straggler landing after the unlink would
|
|
||||||
* RE-CREATE the lock file — a phantom lock blocking the next writer until
|
|
||||||
* the stale TTL expires (the pool-eviction reopen case).
|
|
||||||
*/
|
|
||||||
private writerHeartbeatInFlight?: Promise<void>
|
|
||||||
|
|
||||||
// Flush-request RPC state. The writer polls `locks/_flush_requests/` for
|
// Flush-request RPC state. The writer polls `locks/_flush_requests/` for
|
||||||
// new `.req` files and emits `.ack` files in `locks/_flush_responses/` after
|
// new `.req` files and emits `.ack` files in `locks/_flush_responses/` after
|
||||||
|
|
@ -1888,18 +1880,8 @@ export class FileSystemStorage extends BaseStorage {
|
||||||
// Heartbeat — rewrite lastHeartbeat every WRITER_HEARTBEAT_MS so other
|
// Heartbeat — rewrite lastHeartbeat every WRITER_HEARTBEAT_MS so other
|
||||||
// processes can tell a live writer from one that crashed without releasing.
|
// processes can tell a live writer from one that crashed without releasing.
|
||||||
this.writerLockHeartbeat = setInterval(() => {
|
this.writerLockHeartbeat = setInterval(() => {
|
||||||
const tick = this.refreshWriterLockHeartbeat().catch((err) => {
|
this.refreshWriterLockHeartbeat().catch((err) => {
|
||||||
// ENOENT = the lock (or its directory) vanished mid-refresh — the
|
|
||||||
// store was released or removed under us; the next acquire recreates
|
|
||||||
// it. Benign by construction; anything else stays loud.
|
|
||||||
if ((err as NodeJS.ErrnoException)?.code !== 'ENOENT') {
|
|
||||||
console.warn('[brainy] Failed to refresh writer lock heartbeat:', err)
|
console.warn('[brainy] Failed to refresh writer lock heartbeat:', err)
|
||||||
}
|
|
||||||
})
|
|
||||||
this.writerHeartbeatInFlight = tick.finally(() => {
|
|
||||||
if (this.writerHeartbeatInFlight === tick) {
|
|
||||||
this.writerHeartbeatInFlight = undefined
|
|
||||||
}
|
|
||||||
})
|
})
|
||||||
}, FileSystemStorage.WRITER_HEARTBEAT_MS)
|
}, FileSystemStorage.WRITER_HEARTBEAT_MS)
|
||||||
if (typeof this.writerLockHeartbeat.unref === 'function') {
|
if (typeof this.writerLockHeartbeat.unref === 'function') {
|
||||||
|
|
@ -1931,15 +1913,6 @@ export class FileSystemStorage extends BaseStorage {
|
||||||
clearInterval(this.writerLockHeartbeat)
|
clearInterval(this.writerLockHeartbeat)
|
||||||
this.writerLockHeartbeat = undefined
|
this.writerLockHeartbeat = undefined
|
||||||
}
|
}
|
||||||
// Drain an in-flight heartbeat tick BEFORE unlinking: clearInterval stops
|
|
||||||
// future ticks only, and a straggler write landing after the unlink would
|
|
||||||
// re-create the lock as a phantom (blocking the next writer until the
|
|
||||||
// stale TTL). After the drain, any refresh is either fully landed (we
|
|
||||||
// unlink its output below) or not started (it sees writerLockInfo
|
|
||||||
// undefined and returns).
|
|
||||||
if (this.writerHeartbeatInFlight) {
|
|
||||||
await this.writerHeartbeatInFlight
|
|
||||||
}
|
|
||||||
if (!this.writerLockInfo) {
|
if (!this.writerLockInfo) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -133,11 +133,6 @@ export class MemoryStorage extends BaseStorage {
|
||||||
*/
|
*/
|
||||||
protected async deleteObjectFromPath(path: string): Promise<void> {
|
protected async deleteObjectFromPath(path: string): Promise<void> {
|
||||||
this.objectStore.delete(path)
|
this.objectStore.delete(path)
|
||||||
// Filesystem parity: on disk, objects and raw BYTE files are both just
|
|
||||||
// files — unlink removes whichever exists. Without this, deleteRawObject
|
|
||||||
// on a raw-bytes path (fact-log/generation segments) silently no-ops on
|
|
||||||
// memory storage: the delete "succeeds" and the bytes remain.
|
|
||||||
this.rawBytesStore.delete(path)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
|
|
@ -37,47 +37,22 @@ const DEFAULT_OPTIONS: Required<TransactionOptions> = {
|
||||||
maxRollbackRetries: 3
|
maxRollbackRetries: 3
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* The floor term of {@link transactTimeoutBudget}'s scaling formula when the
|
|
||||||
* caller supplies neither an explicit override nor `config.transactionBudgetFloorMs`.
|
|
||||||
*/
|
|
||||||
const DEFAULT_BUDGET_FLOOR_MS = 30_000
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The apply-phase budget for a batch of `opCount` operations.
|
* The apply-phase budget for a batch of `opCount` operations.
|
||||||
*
|
*
|
||||||
* An explicit `override` wins untouched (a caller-specified deadline for this
|
* An explicit override wins untouched. Otherwise the budget SCALES with the
|
||||||
* one batch). Otherwise the budget SCALES with the batch:
|
* batch: `max(30 000 ms, opCount × 2 000 ms)`. The per-op term is calibrated
|
||||||
* `max(floorMs, opCount × 2 000)`, where `floorMs` defaults to `30 000` and
|
* from field data — bulk imports on network-attached disks measure ~2 s per
|
||||||
* is overridable via `BrainyConfig.transactionBudgetFloorMs` for the whole
|
* operation (each op pays canonical writes + fsync + index maintenance) — so
|
||||||
* brain. The per-op term is calibrated from field data — bulk imports on
|
* a flat 30 s budget silently capped honest work at ~15 operations while
|
||||||
* network-attached disks measure ~2 s per operation (each op pays canonical
|
* looking generous for small batches. Scaling keeps small transacts
|
||||||
* writes + fsync + index maintenance) — so a flat 30 s budget silently capped
|
* fast-failing and gives bulk ones a budget proportional to the work they
|
||||||
* honest work at ~15 operations while looking generous for small batches. A
|
* actually asked for; a trip still rolls back atomically and throws a
|
||||||
* 16-op batch, for example, scales to `max(30 000, 16 × 2 000) = 32 000`.
|
* retryable, fully-labeled TransactionTimeoutError.
|
||||||
* Scaling keeps small transacts fast-failing and gives bulk ones a budget
|
|
||||||
* proportional to the work they actually asked for.
|
|
||||||
*
|
|
||||||
* This is a **mid-batch** budget only: it governs whether the transaction's
|
|
||||||
* NEXT operation may start (see {@link Transaction.execute}), never whether
|
|
||||||
* already-completed work is rolled back after the fact. A trip mid-batch
|
|
||||||
* still rolls back every operation applied so far, atomically, and throws a
|
|
||||||
* retryable, fully-labeled TransactionTimeoutError — that zero-loss guarantee
|
|
||||||
* doesn't change; only the point at which the clock stops mattering does (at
|
|
||||||
* the last operation, not one check later).
|
|
||||||
*
|
|
||||||
* @param opCount - Number of operations in the batch.
|
|
||||||
* @param override - A full override for this call; wins over everything else.
|
|
||||||
* @param floorMs - The scaling floor for this call (e.g. `config.transactionBudgetFloorMs`).
|
|
||||||
* Ignored when `override` is set. Defaults to `30 000` when omitted.
|
|
||||||
*/
|
*/
|
||||||
export function transactTimeoutBudget(
|
export function transactTimeoutBudget(opCount: number, override?: number): number {
|
||||||
opCount: number,
|
|
||||||
override?: number,
|
|
||||||
floorMs?: number
|
|
||||||
): number {
|
|
||||||
if (override !== undefined) return override
|
if (override !== undefined) return override
|
||||||
return Math.max(floorMs ?? DEFAULT_BUDGET_FLOOR_MS, opCount * 2_000)
|
return Math.max(30_000, opCount * 2_000)
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -123,20 +98,7 @@ export class Transaction implements TransactionContext {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Execute all operations atomically.
|
* Execute all operations atomically
|
||||||
*
|
|
||||||
* Budget semantics: the configured budget gates starting the next
|
|
||||||
* operation; completed work is never rolled back for elapsed time. Elapsed
|
|
||||||
* time is checked ONLY before starting operation `i` for `i ≥ 1` — never
|
|
||||||
* before operation 0 (an operation always gets to start) and never again
|
|
||||||
* after the final operation completes. Concretely: a single-op transaction
|
|
||||||
* can never time out post-hoc — it either runs (and its result stands, no
|
|
||||||
* matter how long it took) or a `TransactionTimeoutError` is thrown before
|
|
||||||
* it starts, which cannot happen since there is no operation before it. A
|
|
||||||
* multi-op transaction whose op `i` overruns the budget stops op `i+1` from
|
|
||||||
* starting, rolls back operations `0..i` (reverse order), and throws
|
|
||||||
* `TransactionTimeoutError` — that zero-loss rollback guarantee for
|
|
||||||
* mid-batch trips is unchanged; only the post-completion check is gone.
|
|
||||||
*/
|
*/
|
||||||
async execute(): Promise<void> {
|
async execute(): Promise<void> {
|
||||||
if (this.state !== 'pending') {
|
if (this.state !== 'pending') {
|
||||||
|
|
@ -166,14 +128,10 @@ export class Transaction implements TransactionContext {
|
||||||
// already-applied writes as torn, generation-less state in canonical
|
// already-applied writes as torn, generation-less state in canonical
|
||||||
// storage.
|
// storage.
|
||||||
for (let i = 0; i < this.operations.length; i++) {
|
for (let i = 0; i < this.operations.length; i++) {
|
||||||
// Budget gates STARTING the next operation; completed work is never
|
// Budget check BEFORE starting the next operation. A trip here throws
|
||||||
// rolled back for elapsed time. Skipped for i === 0 (operation 0
|
// into the catch below and rolls back like any other failure — it must
|
||||||
// always gets to start — nothing has run yet to have overrun) and
|
// never bypass rollback.
|
||||||
// never re-checked after the final operation completes (there is no
|
if (Date.now() - this.startTime > this.options.timeout) {
|
||||||
// "next operation" left to gate). A trip here throws into the catch
|
|
||||||
// below and rolls back like any other failure — it must never bypass
|
|
||||||
// rollback.
|
|
||||||
if (i > 0 && Date.now() - this.startTime > this.options.timeout) {
|
|
||||||
throw new TransactionTimeoutError(this.options.timeout, i, {
|
throw new TransactionTimeoutError(this.options.timeout, i, {
|
||||||
elapsedMs: Date.now() - this.startTime,
|
elapsedMs: Date.now() - this.startTime,
|
||||||
totalOperations: this.operations.length,
|
totalOperations: this.operations.length,
|
||||||
|
|
|
||||||
|
|
@ -2,73 +2,34 @@
|
||||||
* Index Operations with Rollback Support
|
* Index Operations with Rollback Support
|
||||||
*
|
*
|
||||||
* Provides transactional operations for all indexes:
|
* Provides transactional operations for all indexes:
|
||||||
* - VectorIndexProvider (the JS HNSW fallback, or a native acceleration provider)
|
* - JsHnswVectorIndex (unified vector index)
|
||||||
* - MetadataIndexManager (roaring bitmap filtering)
|
* - MetadataIndexManager (roaring bitmap filtering)
|
||||||
* - GraphAdjacencyIndex (LSM-tree graph storage)
|
* - GraphAdjacencyIndex (LSM-tree graph storage)
|
||||||
*
|
*
|
||||||
* Each operation can be executed and rolled back atomically.
|
* Each operation can be executed and rolled back atomically.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js'
|
import type { JsHnswVectorIndex } from '../../hnsw/hnswIndex.js'
|
||||||
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
|
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
|
||||||
|
import type { GraphIndexProvider } from '../../plugin.js'
|
||||||
import type { GraphVerb } from '../../coreTypes.js'
|
import type { GraphVerb } from '../../coreTypes.js'
|
||||||
import type { Operation, RollbackAction } from '../types.js'
|
import type { Operation, RollbackAction } from '../types.js'
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Backend identity stamped into an operation's emitted `name` string (e.g.
|
* Add to HNSW index with rollback support
|
||||||
* `AddToVectorIndex(hnsw-js)`), resolved from the provider's own
|
|
||||||
* {@link VectorIndexProvider.name} self-report.
|
|
||||||
*
|
|
||||||
* These operation classes are backend-neutral (the vector index they wrap may
|
|
||||||
* be Brainy's own JS HNSW fallback OR a native acceleration provider), but
|
|
||||||
* their names surface directly in consumer-visible transaction journals and
|
|
||||||
* timings. `name` is REQUIRED at the TypeScript level — but a native provider
|
|
||||||
* instance compiled against the previous (pre-required) contract can still
|
|
||||||
* reach this function at runtime without it. That legacy case is tolerated,
|
|
||||||
* never crashed on and never silently mislabeled: it resolves to
|
|
||||||
* `'unknown-provider'` and emits ONE loud warning naming the missing contract
|
|
||||||
* field, so the fix (implement `name`) is discoverable rather than a silent
|
|
||||||
* fossil label in every journal line thereafter.
|
|
||||||
*/
|
|
||||||
const warnedMissingName = new WeakSet<VectorIndexProvider>()
|
|
||||||
|
|
||||||
function resolveVectorProviderId(index: VectorIndexProvider): string {
|
|
||||||
const name = (index as { name?: unknown }).name
|
|
||||||
if (typeof name === 'string') return name
|
|
||||||
if (!warnedMissingName.has(index)) {
|
|
||||||
warnedMissingName.add(index)
|
|
||||||
console.warn(
|
|
||||||
'[vector-index] provider is missing the required `name` field (VectorIndexProvider.name, ' +
|
|
||||||
'required since 8.10.0) — stamping "unknown-provider" in transaction op names until the ' +
|
|
||||||
'provider declares its own identity.'
|
|
||||||
)
|
|
||||||
}
|
|
||||||
return 'unknown-provider'
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Add to the vector index with rollback support.
|
|
||||||
*
|
|
||||||
* Backend-neutral: `index` is whatever the `'vector'` provider factory
|
|
||||||
* returns — Brainy's own JS HNSW fallback, or a native acceleration provider
|
|
||||||
* (e.g. DiskANN). The emitted `name` stamps the active backend (see
|
|
||||||
* {@link resolveVectorProviderId}) so operators reading a transaction journal
|
|
||||||
* or timing trace see which engine actually ran, never a fossil name from
|
|
||||||
* whichever engine happened to be active when this op class was written.
|
|
||||||
*
|
*
|
||||||
* Rollback strategy:
|
* Rollback strategy:
|
||||||
* - Remove item from index
|
* - Remove item from index
|
||||||
|
|
||||||
*/
|
*/
|
||||||
export class AddToVectorIndexOperation implements Operation {
|
export class AddToHNSWOperation implements Operation {
|
||||||
readonly name: string
|
readonly name = 'AddToHNSW'
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private readonly index: VectorIndexProvider,
|
private readonly index: JsHnswVectorIndex,
|
||||||
private readonly id: string,
|
private readonly id: string,
|
||||||
private readonly vector: number[]
|
private readonly vector: number[]
|
||||||
) {
|
) {}
|
||||||
this.name = `AddToVectorIndex(${resolveVectorProviderId(index)})`
|
|
||||||
}
|
|
||||||
|
|
||||||
async execute(): Promise<RollbackAction> {
|
async execute(): Promise<RollbackAction> {
|
||||||
// Check if item already exists (for rollback decision)
|
// Check if item already exists (for rollback decision)
|
||||||
|
|
@ -98,12 +59,11 @@ export class AddToVectorIndexOperation implements Operation {
|
||||||
* pre-existence as "existed" made every rollback skip removeItem, leaving
|
* pre-existence as "existed" made every rollback skip removeItem, leaving
|
||||||
* phantom entries in the index after a failed transaction. The safe default
|
* phantom entries in the index after a failed transaction. The safe default
|
||||||
* is to remove what this operation added — update flows pair this op with a
|
* is to remove what this operation added — update flows pair this op with a
|
||||||
* RemoveFromVectorIndexOperation whose own rollback restores the prior
|
* RemoveFromHNSWOperation whose own rollback restores the prior vector, so
|
||||||
* vector, so reverse-order rollback reconstructs the original state either
|
* reverse-order rollback reconstructs the original state either way.
|
||||||
* way.
|
|
||||||
*/
|
*/
|
||||||
private async itemExists(id: string): Promise<boolean> {
|
private async itemExists(id: string): Promise<boolean> {
|
||||||
const index = this.index as VectorIndexProvider & {
|
const index = this.index as JsHnswVectorIndex & {
|
||||||
getItem?: (id: string) => Promise<unknown>
|
getItem?: (id: string) => Promise<unknown>
|
||||||
}
|
}
|
||||||
if (typeof index.getItem !== 'function') return false
|
if (typeof index.getItem !== 'function') return false
|
||||||
|
|
@ -117,27 +77,21 @@ export class AddToVectorIndexOperation implements Operation {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Remove from the vector index with rollback support.
|
* Remove from HNSW index with rollback support
|
||||||
*
|
|
||||||
* Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the
|
|
||||||
* JS HNSW fallback or a native acceleration provider; the emitted `name`
|
|
||||||
* stamps the active backend.
|
|
||||||
*
|
*
|
||||||
* Rollback strategy:
|
* Rollback strategy:
|
||||||
* - Re-add item to index with original vector
|
* - Re-add item to index with original vector
|
||||||
*
|
*
|
||||||
* Note: Requires storing the vector for rollback
|
* Note: Requires storing the vector for rollback
|
||||||
*/
|
*/
|
||||||
export class RemoveFromVectorIndexOperation implements Operation {
|
export class RemoveFromHNSWOperation implements Operation {
|
||||||
readonly name: string
|
readonly name = 'RemoveFromHNSW'
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private readonly index: VectorIndexProvider,
|
private readonly index: JsHnswVectorIndex,
|
||||||
private readonly id: string,
|
private readonly id: string,
|
||||||
private readonly vector: number[] // Required for rollback
|
private readonly vector: number[] // Required for rollback
|
||||||
) {
|
) {}
|
||||||
this.name = `RemoveFromVectorIndex(${resolveVectorProviderId(index)})`
|
|
||||||
}
|
|
||||||
|
|
||||||
async execute(): Promise<RollbackAction> {
|
async execute(): Promise<RollbackAction> {
|
||||||
// Remove from index
|
// Remove from index
|
||||||
|
|
@ -331,24 +285,22 @@ export class RemoveFromGraphIndexOperation implements Operation {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Batch operation: Add multiple items to the vector index (backend-neutral —
|
* Batch operation: Add multiple items to HNSW index
|
||||||
* see {@link AddToVectorIndexOperation}).
|
|
||||||
*
|
*
|
||||||
* Useful for bulk imports with transaction support.
|
* Useful for bulk imports with transaction support.
|
||||||
* Rolls back all items if any fail.
|
* Rolls back all items if any fail.
|
||||||
*/
|
*/
|
||||||
export class BatchAddToVectorIndexOperation implements Operation {
|
export class BatchAddToHNSWOperation implements Operation {
|
||||||
readonly name: string
|
readonly name = 'BatchAddToHNSW'
|
||||||
|
|
||||||
private operations: AddToVectorIndexOperation[]
|
private operations: AddToHNSWOperation[]
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
index: VectorIndexProvider,
|
index: JsHnswVectorIndex,
|
||||||
items: Array<{ id: string; vector: number[] }>
|
items: Array<{ id: string; vector: number[] }>
|
||||||
) {
|
) {
|
||||||
this.name = `BatchAddToVectorIndex(${resolveVectorProviderId(index)})`
|
|
||||||
this.operations = items.map(
|
this.operations = items.map(
|
||||||
item => new AddToVectorIndexOperation(index, item.id, item.vector)
|
item => new AddToHNSWOperation(index, item.id, item.vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -21,12 +21,12 @@ export {
|
||||||
|
|
||||||
// Index Operations
|
// Index Operations
|
||||||
export {
|
export {
|
||||||
AddToVectorIndexOperation,
|
AddToHNSWOperation,
|
||||||
RemoveFromVectorIndexOperation,
|
RemoveFromHNSWOperation,
|
||||||
AddToMetadataIndexOperation,
|
AddToMetadataIndexOperation,
|
||||||
RemoveFromMetadataIndexOperation,
|
RemoveFromMetadataIndexOperation,
|
||||||
AddToGraphIndexOperation,
|
AddToGraphIndexOperation,
|
||||||
RemoveFromGraphIndexOperation,
|
RemoveFromGraphIndexOperation,
|
||||||
BatchAddToVectorIndexOperation,
|
BatchAddToHNSWOperation,
|
||||||
BatchAddToMetadataIndexOperation
|
BatchAddToMetadataIndexOperation
|
||||||
} from './IndexOperations.js'
|
} from './IndexOperations.js'
|
||||||
|
|
|
||||||
|
|
@ -1651,25 +1651,6 @@ export interface BrainyConfig {
|
||||||
*/
|
*/
|
||||||
disableAutoRebuild?: boolean
|
disableAutoRebuild?: boolean
|
||||||
|
|
||||||
/**
|
|
||||||
* The floor (ms) of the apply-phase transaction budget's scaling formula:
|
|
||||||
* `max(transactionBudgetFloorMs, opCount × 2 000)`. The budget governs
|
|
||||||
* whether a `transact()` / single-op write's NEXT internal operation may
|
|
||||||
* **start** — never whether already-completed work gets rolled back after
|
|
||||||
* the fact (a single-op write can never time out post-hoc: it either runs
|
|
||||||
* or it commits). A trip mid-batch still rolls back every applied operation
|
|
||||||
* atomically and throws a retryable `TransactionTimeoutError`; only the
|
|
||||||
* floor of the formula is configurable here.
|
|
||||||
*
|
|
||||||
* Raise this when a cold store's first writes after a restart legitimately
|
|
||||||
* take longer than 30s per operation (e.g. page-cache-cold canonical writes
|
|
||||||
* on a large brain) so a bulk `transact()` batch gets a proportionally
|
|
||||||
* larger runway instead of tripping mid-batch. Lower it to fail faster on a
|
|
||||||
* latency-sensitive write path. Default: `30000` (30 s) — unchanged from
|
|
||||||
* pre-8.9 behavior.
|
|
||||||
*/
|
|
||||||
transactionBudgetFloorMs?: number
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration
|
* How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration
|
||||||
* before throwing a retryable `MigrationInProgressError`.
|
* before throwing a retryable `MigrationInProgressError`.
|
||||||
|
|
@ -1756,10 +1737,7 @@ export interface BrainyConfig {
|
||||||
* Under Model-B EVERY write (`transact()` AND single-op `add`/`update`/
|
* Under Model-B EVERY write (`transact()` AND single-op `add`/`update`/
|
||||||
* `remove`/`relate`) produces an immutable generation record-set serving
|
* `remove`/`relate`) produces an immutable generation record-set serving
|
||||||
* historical reads (`asOf()`, pinned `Db` values). Without compaction those
|
* historical reads (`asOf()`, pinned `Db` values). Without compaction those
|
||||||
* accumulate, so Brainy **auto-compacts at `close()`** (time-bounded per
|
* accumulate, so Brainy **auto-compacts on every `flush()` and `close()`**.
|
||||||
* pass; 8.9.0 removed compaction from `flush()` — flush is durability work
|
|
||||||
* and never pays maintenance costs). A long-lived writer that never closes
|
|
||||||
* accumulates history until its next explicit `compactHistory()` call.
|
|
||||||
* Live `Db` pins are ALWAYS exempt from reclamation, in every mode.
|
* Live `Db` pins are ALWAYS exempt from reclamation, in every mode.
|
||||||
*
|
*
|
||||||
* Modes:
|
* Modes:
|
||||||
|
|
@ -1776,7 +1754,7 @@ export interface BrainyConfig {
|
||||||
* the oldest unpinned generations while ANY supplied cap is exceeded
|
* the oldest unpinned generations while ANY supplied cap is exceeded
|
||||||
* (predictable ops). `maxAge` in ms; `maxBytes` total history bytes.
|
* (predictable ops). `maxAge` in ms; `maxBytes` total history bytes.
|
||||||
*
|
*
|
||||||
* `autoCompact: false` disables the automatic close() compaction (manage
|
* `autoCompact: false` disables the automatic flush/close compaction (manage
|
||||||
* manually via `brain.compactHistory()`). `budgetBytes` is the settable
|
* manually via `brain.compactHistory()`). `budgetBytes` is the settable
|
||||||
* adaptive byte budget a coordinator drives (also via `brain.setRetentionBudget()`).
|
* adaptive byte budget a coordinator drives (also via `brain.setRetentionBudget()`).
|
||||||
* Long-term archives belong in `db.persist(path)` snapshots, which compaction
|
* Long-term archives belong in `db.persist(path)` snapshots, which compaction
|
||||||
|
|
@ -1794,7 +1772,7 @@ export interface BrainyConfig {
|
||||||
maxBytes?: number
|
maxBytes?: number
|
||||||
/** Adaptive byte budget for this brain, driven by a coordinator (e.g. cor). */
|
/** Adaptive byte budget for this brain, driven by a coordinator (e.g. cor). */
|
||||||
budgetBytes?: number
|
budgetBytes?: number
|
||||||
/** Run compaction automatically at close() (default: true; 8.9.0 — flush() never compacts). */
|
/** Run compaction automatically on flush()/close() (default: true). */
|
||||||
autoCompact?: boolean
|
autoCompact?: boolean
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1825,23 +1803,6 @@ export interface BrainyConfig {
|
||||||
*/
|
*/
|
||||||
eagerEmbeddings?: boolean
|
eagerEmbeddings?: boolean
|
||||||
|
|
||||||
/**
|
|
||||||
* When `true`, `init()` **awaits `warm()`** (see {@link Brainy.warm}) before
|
|
||||||
* it resolves — blocking startup until the vector index, metadata index,
|
|
||||||
* and graph adjacency have all faulted their backing storage in. This is a
|
|
||||||
* deliberate operator opt-in trade: startup takes longer, but the FIRST
|
|
||||||
* request after a cold restart runs at steady-state cost instead of paying
|
|
||||||
* page-cache-miss / demand-load latency on the critical path.
|
|
||||||
*
|
|
||||||
* Runs AFTER `init()`'s own sequence completes (index construction, crash
|
|
||||||
* recovery, migrations, VFS bootstrap) — never in place of any of it — so
|
|
||||||
* `warm()` always operates on a fully-initialized brain.
|
|
||||||
*
|
|
||||||
* Default: `false` (lazy — the pre-8.9 behavior: indexes demand-load on
|
|
||||||
* first access).
|
|
||||||
*/
|
|
||||||
warmOnOpen?: boolean
|
|
||||||
|
|
||||||
// Plugin configuration
|
// Plugin configuration
|
||||||
// Controls which plugins are loaded during init().
|
// Controls which plugins are loaded during init().
|
||||||
// - undefined (default): guarded auto-detection of the first-party
|
// - undefined (default): guarded auto-detection of the first-party
|
||||||
|
|
|
||||||
|
|
@ -423,46 +423,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
prodLog.debug('✅ Type-aware cache warming completed')
|
prodLog.debug('✅ Type-aware cache warming completed')
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Full hydration — the {@link Brainy.warm} readiness seam for the metadata
|
|
||||||
* index. Unlike {@link warmCache} / {@link warmCacheForTopTypes} (which
|
|
||||||
* warm only a heuristic subset: common fields plus the top-N types' top
|
|
||||||
* fields), this loads EVERY field's sparse index the field registry knows
|
|
||||||
* about — a real read through {@link loadSparseIndex} into the unified
|
|
||||||
* cache for each field, not a stat/existence check. Idempotent: an
|
|
||||||
* already-cached field's `loadSparseIndex` call is a cheap cache hit.
|
|
||||||
*
|
|
||||||
* Re-reads the field registry first when `fieldIndexes` is empty (a warm()
|
|
||||||
* call issued before `init()` populated it would otherwise hydrate
|
|
||||||
* nothing), then loads every discovered field in parallel.
|
|
||||||
*/
|
|
||||||
async hydrateAll(): Promise<void> {
|
|
||||||
if (this.fieldIndexes.size === 0) {
|
|
||||||
await this.loadFieldRegistry()
|
|
||||||
}
|
|
||||||
|
|
||||||
const fields = Array.from(this.fieldIndexes.keys())
|
|
||||||
if (fields.length === 0) {
|
|
||||||
prodLog.debug('[MetadataIndex] hydrateAll: no persisted fields to hydrate')
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
prodLog.debug(`[MetadataIndex] hydrateAll: loading ${fields.length} field(s) — ${fields.join(', ')}`)
|
|
||||||
|
|
||||||
await Promise.all(
|
|
||||||
fields.map(async field => {
|
|
||||||
try {
|
|
||||||
await this.loadSparseIndex(field)
|
|
||||||
} catch (error) {
|
|
||||||
// A single field's load failure doesn't abort the rest of the
|
|
||||||
// hydration — warm() is a best-effort readiness step, never a
|
|
||||||
// correctness gate (queries still demand-load on miss).
|
|
||||||
prodLog.debug(`[MetadataIndex] hydrateAll: field '${field}' failed to load:`, error)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Acquire an in-memory lock for coordinating concurrent metadata index writes
|
* Acquire an in-memory lock for coordinating concurrent metadata index writes
|
||||||
* Uses in-memory locks since MetadataIndexManager doesn't have direct file system access
|
* Uses in-memory locks since MetadataIndexManager doesn't have direct file system access
|
||||||
|
|
|
||||||
|
|
@ -1280,32 +1280,24 @@ describe('8.0 Db API — generational MVCC', () => {
|
||||||
await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
|
await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('Model-B retention — flush() NEVER compacts (8.9.0); adaptive reclaim runs at close()', async () => {
|
it('Model-B retention — setRetentionBudget drives adaptive reclaim on flush; live data intact', async () => {
|
||||||
// Default brain → ADAPTIVE retention with a driven byte budget far below
|
// Default brain → ADAPTIVE retention. A coordinator (e.g. cor's ResourceManager)
|
||||||
// the accumulated history (~13 generations of full-vector before-images).
|
// pushes a byte budget via setRetentionBudget(); auto-compaction on flush() reclaims
|
||||||
// The 8.9.0 law: flush() is durability-only — it must not reclaim even
|
// oldest history down toward it. Each update's before-image carries the full prior
|
||||||
// when the budget is exceeded (reclaim-on-flush blocked production writes
|
// 384-dim vector (~KBs), so ~13 generations far exceed a few-KB budget.
|
||||||
// for 25-191s). Maintenance runs at close(), time-bounded.
|
const { brain } = await openFsBrain()
|
||||||
const { brain, dir } = await openFsBrain()
|
|
||||||
const a = uid('ret-budget')
|
const a = uid('ret-budget')
|
||||||
await brain.add({ id: a, type: NounType.Document, data: 'v0', vector: vec(1), metadata: { v: 0 } })
|
await brain.add({ id: a, type: NounType.Document, data: 'v0', vector: vec(1), metadata: { v: 0 } })
|
||||||
for (let v = 1; v <= 12; v++) await brain.update({ id: a, metadata: { v } })
|
for (let v = 1; v <= 12; v++) await brain.update({ id: a, metadata: { v } })
|
||||||
|
|
||||||
brain.setRetentionBudget(6000) // ~6 KB — well below the accumulated history
|
brain.setRetentionBudget(6000) // ~6 KB — well below the accumulated history
|
||||||
await brain.flush()
|
await brain.flush() // group-commit + adaptive auto-compaction under the budget
|
||||||
|
|
||||||
// flush() paid durability only: nothing reclaimed, all history readable.
|
// History was reclaimed (the horizon advanced past the oldest generations)…
|
||||||
expect(generationStoreOf(brain).horizon()).toBe(0)
|
expect(generationStoreOf(brain).horizon()).toBeGreaterThan(0)
|
||||||
const probe = await brain.asOf(1) // readable proves nothing was reclaimed…
|
await expect(brain.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
|
||||||
await probe.release() // …and MUST be released: a held pin would (correctly)
|
// …but the budget reclaims HISTORY only — the live record is untouched.
|
||||||
// protect every newer generation through the close() compaction below.
|
expect((await brain.get(a))?.metadata?.v).toBe(12)
|
||||||
await brain.close() // ← THE auto-compaction site now
|
|
||||||
|
|
||||||
// close() reclaimed under the budget; live record intact; horizon durable.
|
|
||||||
const { brain: reopened } = await openFsBrain(dir)
|
|
||||||
expect(generationStoreOf(reopened).horizon()).toBeGreaterThan(0)
|
|
||||||
await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
|
|
||||||
expect((await reopened.get(a))?.metadata?.v).toBe(12)
|
|
||||||
})
|
})
|
||||||
|
|
||||||
// ==========================================================================
|
// ==========================================================================
|
||||||
|
|
|
||||||
|
|
@ -1,186 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/integration/history-repacking
|
|
||||||
* @description The D1+D3 two-tier history lifecycle end-to-end on a real
|
|
||||||
* brain. Laws: (1) repacking is RE-REPRESENTATION — after folding, every
|
|
||||||
* asOf() read below the fold boundary answers exactly as before, across a
|
|
||||||
* cold reopen; (2) folded per-generation directories are physically gone
|
|
||||||
* (the file-count cure is real, not cosmetic); (3) repack + reclaim compose:
|
|
||||||
* bounded retention after repacking drops whole segments and asOf below the
|
|
||||||
* horizon throws GenerationCompactedError; (4) repackHistory is explicit
|
|
||||||
* API and time-bounded (spent budget = consistent no-op).
|
|
||||||
*
|
|
||||||
* Uses a tiny REPACK_LIVE_WINDOW override so a small history has a cold
|
|
||||||
* tier at all (the production window is 1024).
|
|
||||||
*/
|
|
||||||
import { describe, it, expect, afterEach } from 'vitest'
|
|
||||||
import * as fs from 'node:fs'
|
|
||||||
import * as path from 'node:path'
|
|
||||||
import * as os from 'node:os'
|
|
||||||
import { Brainy } from '../../src/brainy.js'
|
|
||||||
import { NounType } from '../../src/types/graphTypes.js'
|
|
||||||
import { GenerationStore } from '../../src/db/generationStore.js'
|
|
||||||
import { GenerationCompactedError } from '../../src/db/errors.js'
|
|
||||||
import { SEGMENTS_PREFIX } from '../../src/db/generationSegments.js'
|
|
||||||
|
|
||||||
const stub = async (text: string): Promise<number[]> => {
|
|
||||||
const h = text.split('').reduce((a, c) => a + c.charCodeAt(0), 0)
|
|
||||||
return new Array(384).fill(0).map((_, i) => Math.sin(h + i))
|
|
||||||
}
|
|
||||||
|
|
||||||
const openBrain = async (dir: string): Promise<Brainy> => {
|
|
||||||
const brain = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'filesystem', path: dir },
|
|
||||||
embeddingFunction: stub
|
|
||||||
})
|
|
||||||
await brain.init()
|
|
||||||
return brain
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('history repacking — the two-tier lifecycle', () => {
|
|
||||||
const dirs: string[] = []
|
|
||||||
const tempDir = (): string => {
|
|
||||||
const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-repack-'))
|
|
||||||
dirs.push(d)
|
|
||||||
return d
|
|
||||||
}
|
|
||||||
const originalWindow = GenerationStore.REPACK_LIVE_WINDOW
|
|
||||||
|
|
||||||
afterEach(() => {
|
|
||||||
;(GenerationStore as any).REPACK_LIVE_WINDOW = originalWindow
|
|
||||||
for (const d of dirs.splice(0)) {
|
|
||||||
try {
|
|
||||||
fs.rmSync(d, { recursive: true, force: true })
|
|
||||||
} catch {
|
|
||||||
/* best effort */
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('repack preserves every historical read across cold reopen; folded dirs are gone', async () => {
|
|
||||||
;(GenerationStore as any).REPACK_LIVE_WINDOW = 3
|
|
||||||
const dir = tempDir()
|
|
||||||
const brain = await openBrain(dir)
|
|
||||||
|
|
||||||
const id = await brain.add({
|
|
||||||
data: 'versioned-entity',
|
|
||||||
type: NounType.Document,
|
|
||||||
metadata: { v: 0 }
|
|
||||||
})
|
|
||||||
for (let v = 1; v <= 10; v++) await brain.update({ id, metadata: { v } })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
// Ground truth BEFORE repacking: capture asOf views for early generations.
|
|
||||||
const before: Record<number, number> = {}
|
|
||||||
for (const g of [2, 4, 6]) {
|
|
||||||
const db = await brain.asOf(g)
|
|
||||||
before[g] = (await db.get(id))?.metadata?.v as number
|
|
||||||
await db.release()
|
|
||||||
}
|
|
||||||
|
|
||||||
const result = await brain.repackHistory()
|
|
||||||
expect(result.foldedGenerations).toBeGreaterThan(0)
|
|
||||||
expect(result.segmentsCreated).toBeGreaterThan(0)
|
|
||||||
|
|
||||||
// The folded per-generation directories are PHYSICALLY gone…
|
|
||||||
const genDirs = fs
|
|
||||||
.readdirSync(path.join(dir, '_generations'), { withFileTypes: true })
|
|
||||||
.filter((e) => e.isDirectory() && /^\d+$/.test(e.name)).length
|
|
||||||
expect(genDirs).toBeLessThanOrEqual(4) // live window (3) + at most the newest
|
|
||||||
// …and the segment tier exists (the filesystem adapter stores objects
|
|
||||||
// gzipped, so the manifest may live at either spelling).
|
|
||||||
const segDir = path.join(dir, SEGMENTS_PREFIX)
|
|
||||||
expect(
|
|
||||||
fs.existsSync(path.join(segDir, 'manifest.json')) ||
|
|
||||||
fs.existsSync(path.join(segDir, 'manifest.json.gz'))
|
|
||||||
).toBe(true)
|
|
||||||
expect(fs.readdirSync(segDir).some((f) => f.endsWith('.bgs'))).toBe(true)
|
|
||||||
|
|
||||||
// Same asOf answers from the packed tier, same process…
|
|
||||||
for (const g of [2, 4, 6]) {
|
|
||||||
const db = await brain.asOf(g)
|
|
||||||
expect((await db.get(id))?.metadata?.v).toBe(before[g])
|
|
||||||
await db.release()
|
|
||||||
}
|
|
||||||
await brain.close()
|
|
||||||
|
|
||||||
// …and across a COLD REOPEN (manifest discovery, no live dirs to list).
|
|
||||||
const reopened = await openBrain(dir)
|
|
||||||
for (const g of [2, 4, 6]) {
|
|
||||||
const db = await reopened.asOf(g)
|
|
||||||
expect((await db.get(id))?.metadata?.v).toBe(before[g])
|
|
||||||
await db.release()
|
|
||||||
}
|
|
||||||
expect((await reopened.get(id))?.metadata?.v).toBe(10) // live state untouched
|
|
||||||
await reopened.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('repack + bounded reclaim compose: whole segments drop, horizon is loud', async () => {
|
|
||||||
;(GenerationStore as any).REPACK_LIVE_WINDOW = 2
|
|
||||||
const dir = tempDir()
|
|
||||||
const brain = await openBrain(dir)
|
|
||||||
const id = await brain.add({ data: 'reclaim-probe', type: NounType.Document, metadata: { v: 0 } })
|
|
||||||
for (let v = 1; v <= 8; v++) await brain.update({ id, metadata: { v } })
|
|
||||||
await brain.flush()
|
|
||||||
await brain.repackHistory()
|
|
||||||
|
|
||||||
// Reclaim down to the 3 newest generations — packed segments below the
|
|
||||||
// horizon drop whole; asOf below throws loudly.
|
|
||||||
const res = await brain.compactHistory({ maxGenerations: 3 })
|
|
||||||
expect(res.removedGenerations).toBeGreaterThan(0)
|
|
||||||
await expect(brain.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
|
|
||||||
expect((await brain.get(id))?.metadata?.v).toBe(8)
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('generationDigest: reopen-stable, divergence-sensitive, loud below the horizon', async () => {
|
|
||||||
;(GenerationStore as any).REPACK_LIVE_WINDOW = 2
|
|
||||||
const dir = tempDir()
|
|
||||||
const brain = await openBrain(dir)
|
|
||||||
const id = await brain.add({ data: 'digest-probe', type: NounType.Document, metadata: { v: 0 } })
|
|
||||||
for (let v = 1; v <= 6; v++) await brain.update({ id, metadata: { v } })
|
|
||||||
await brain.flush()
|
|
||||||
await brain.repackHistory()
|
|
||||||
|
|
||||||
const gen = brain.generation()
|
|
||||||
const atHead = await brain.generationDigest(gen)
|
|
||||||
const atMid = await brain.generationDigest(3)
|
|
||||||
expect(atHead).toMatch(/^[0-9a-f]{8}$/)
|
|
||||||
expect(atMid).not.toBe(atHead) // more history ⇒ different digest
|
|
||||||
await brain.close()
|
|
||||||
|
|
||||||
// Reopen-stable: same history, same digests (packed prefix stability).
|
|
||||||
const reopened = await openBrain(dir)
|
|
||||||
expect(await reopened.generationDigest(gen)).toBe(atHead)
|
|
||||||
expect(await reopened.generationDigest(3)).toBe(atMid)
|
|
||||||
|
|
||||||
// New history diverges the head digest.
|
|
||||||
await reopened.update({ id, metadata: { v: 7 } })
|
|
||||||
await reopened.flush()
|
|
||||||
expect(await reopened.generationDigest(reopened.generation())).not.toBe(atHead)
|
|
||||||
|
|
||||||
// Below the horizon: LOUD, never a silent pin of reclaimed history.
|
|
||||||
await reopened.compactHistory({ maxGenerations: 2 })
|
|
||||||
await expect(reopened.generationDigest(1)).rejects.toBeInstanceOf(GenerationCompactedError)
|
|
||||||
await reopened.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('a spent time budget is a consistent no-op; the next pass resumes', async () => {
|
|
||||||
;(GenerationStore as any).REPACK_LIVE_WINDOW = 2
|
|
||||||
const dir = tempDir()
|
|
||||||
const brain = await openBrain(dir)
|
|
||||||
const id = await brain.add({ data: 'budget-probe', type: NounType.Document, metadata: { v: 0 } })
|
|
||||||
for (let v = 1; v <= 6; v++) await brain.update({ id, metadata: { v } })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const bounded = await brain.repackHistory({ timeBudgetMs: 0 })
|
|
||||||
expect(bounded).toEqual({ foldedGenerations: 0, segmentsCreated: 0 })
|
|
||||||
|
|
||||||
const resumed = await brain.repackHistory()
|
|
||||||
expect(resumed.foldedGenerations).toBeGreaterThan(0)
|
|
||||||
const db = await brain.asOf(3)
|
|
||||||
expect((await db.get(id))?.metadata?.v).toBeDefined()
|
|
||||||
await db.release()
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -153,46 +153,6 @@ describe('Multi-process safety + read-only mode', () => {
|
||||||
expect(err.lockInfo?.pid).toBe(otherPid)
|
expect(err.lockInfo?.pid).toBe(otherPid)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('release drains an in-flight heartbeat — no phantom lock re-created after unlink', async () => {
|
|
||||||
// The race (8.9.0): clearInterval stops FUTURE heartbeat ticks, but a
|
|
||||||
// tick already in flight could land its lock rewrite AFTER release's
|
|
||||||
// unlink — re-creating the lock as a phantom that blocks the next
|
|
||||||
// writer until the stale TTL. Simulate the in-flight tick explicitly
|
|
||||||
// and prove release waits for it.
|
|
||||||
writer = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
|
|
||||||
await writer.init()
|
|
||||||
const storage: any = (writer as any).storage
|
|
||||||
|
|
||||||
// An in-flight refresh that is ALREADY PAST its ownership guards
|
|
||||||
// (captured the lock info before release ran) and lands its atomic
|
|
||||||
// rewrite slowly — the exact straggler shape; absent the drain it
|
|
||||||
// writes after the unlink.
|
|
||||||
const { join: joinPath } = await import('node:path')
|
|
||||||
const capturedInfo = { ...storage.writerLockInfo }
|
|
||||||
const lockPath = joinPath(dir, 'locks', '_writer.lock')
|
|
||||||
const slowTick = (async () => {
|
|
||||||
await new Promise((r) => setTimeout(r, 100))
|
|
||||||
await storage.writeFileAtomic(
|
|
||||||
lockPath,
|
|
||||||
JSON.stringify({ ...capturedInfo, lastHeartbeat: new Date().toISOString() })
|
|
||||||
)
|
|
||||||
})()
|
|
||||||
storage.writerHeartbeatInFlight = slowTick.catch(() => {})
|
|
||||||
|
|
||||||
await writer.close() // → releaseWriterLock must drain slowTick first
|
|
||||||
await slowTick.catch(() => {}) // both paths fully settled either way
|
|
||||||
writer = null
|
|
||||||
|
|
||||||
const { existsSync } = await import('node:fs')
|
|
||||||
const { join } = await import('node:path')
|
|
||||||
expect(existsSync(join(dir, 'locks', '_writer.lock'))).toBe(false)
|
|
||||||
|
|
||||||
// And the directory is immediately claimable — no stale-TTL wait.
|
|
||||||
const next = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } })
|
|
||||||
await expect(next.init()).resolves.toBeUndefined()
|
|
||||||
await next.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('allows a second in-process writer with a warning (same PID)', async () => {
|
it('allows a second in-process writer with a warning (same PID)', async () => {
|
||||||
// Two Brainy instances in the same Node process: not the dangerous
|
// Two Brainy instances in the same Node process: not the dangerous
|
||||||
// cross-process case. Should succeed (with a console warning).
|
// cross-process case. Should succeed (with a console warning).
|
||||||
|
|
|
||||||
|
|
@ -1,310 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/unit/brainy/warm
|
|
||||||
* @description Coverage for `brain.warm()` / `warmOnOpen` / the provider
|
|
||||||
* `warm?()` contract (the cold-restart readiness fix): first operations after
|
|
||||||
* a cold restart should run at steady-state cost instead of paying
|
|
||||||
* demand-load latency on the critical path.
|
|
||||||
*
|
|
||||||
* Uses a real, in-process fake plugin provider implementing the actual
|
|
||||||
* `VectorIndexProvider` contract from `src/plugin.ts` — the real seam a
|
|
||||||
* native provider (e.g. a disk-native accelerator) plugs into. The real
|
|
||||||
* built-in JS metadata index and graph adjacency index run against real
|
|
||||||
* (filesystem or in-memory) storage with pre-existing data, so their
|
|
||||||
* hydration paths are exercised for real, not mocked.
|
|
||||||
*/
|
|
||||||
import { describe, it, expect, afterEach } from 'vitest'
|
|
||||||
import * as fs from 'node:fs'
|
|
||||||
import * as os from 'node:os'
|
|
||||||
import * as path from 'node:path'
|
|
||||||
import { Brainy } from '../../../src/brainy.js'
|
|
||||||
import { NounType, VerbType } from '../../../src/types/graphTypes.js'
|
|
||||||
import type { VectorIndexProvider } from '../../../src/plugin.js'
|
|
||||||
import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
|
|
||||||
import { MetadataIndexManager } from '../../../src/utils/metadataIndex.js'
|
|
||||||
import { GraphAdjacencyIndex } from '../../../src/graph/graphAdjacencyIndex.js'
|
|
||||||
|
|
||||||
const tmpDirs: string[] = []
|
|
||||||
function mkTmp(): string {
|
|
||||||
const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-warm-'))
|
|
||||||
tmpDirs.push(d)
|
|
||||||
return d
|
|
||||||
}
|
|
||||||
afterEach(() => {
|
|
||||||
for (const d of tmpDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true })
|
|
||||||
})
|
|
||||||
|
|
||||||
// Brainy's ValidationConfig fixes vectors at exactly 384 dimensions
|
|
||||||
// (src/utils/paramValidation.ts) — match it so `add()` doesn't reject test data.
|
|
||||||
const DIM = 384
|
|
||||||
const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin(seed + i))
|
|
||||||
|
|
||||||
/**
|
|
||||||
* A real (not mocked) VectorIndexProvider implementation, backed by a plain
|
|
||||||
* Map, that optionally implements `warm()` — the exact seam
|
|
||||||
* `AddToVectorIndexOperation` / `brain.warm()` call through.
|
|
||||||
*/
|
|
||||||
class FakeVectorProvider implements VectorIndexProvider {
|
|
||||||
readonly name = 'fake-vector-provider'
|
|
||||||
readonly items = new Map<string, Vector>()
|
|
||||||
warmCalls = 0
|
|
||||||
searchCalls: Array<{ k?: number }> = []
|
|
||||||
// Only present on the instance when the constructor is told to — mirrors a
|
|
||||||
// real provider that may or may not implement the optional hook. Assigned
|
|
||||||
// in the constructor BODY (not as a field initializer): under native ES
|
|
||||||
// class fields, field initializers run before constructor-body statements
|
|
||||||
// — including the parameter-property assignment — so referencing a
|
|
||||||
// parameter property from a field initializer would see it as still
|
|
||||||
// `undefined`.
|
|
||||||
warm?: () => Promise<void>
|
|
||||||
|
|
||||||
constructor(hasWarm: boolean) {
|
|
||||||
if (hasWarm) {
|
|
||||||
this.warm = async (): Promise<void> => {
|
|
||||||
this.warmCalls++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
async addItem(item: VectorDocument): Promise<string> {
|
|
||||||
this.items.set(item.id, item.vector)
|
|
||||||
return item.id
|
|
||||||
}
|
|
||||||
async removeItem(id: string): Promise<boolean> {
|
|
||||||
return this.items.delete(id)
|
|
||||||
}
|
|
||||||
async search(_queryVector: Vector, k?: number): Promise<Array<[string, number]>> {
|
|
||||||
this.searchCalls.push({ k })
|
|
||||||
return [...this.items.keys()].slice(0, k ?? 10).map((id) => [id, 0])
|
|
||||||
}
|
|
||||||
size(): number {
|
|
||||||
return this.items.size
|
|
||||||
}
|
|
||||||
clear(): void {
|
|
||||||
this.items.clear()
|
|
||||||
}
|
|
||||||
async rebuild(): Promise<void> {}
|
|
||||||
async flush(): Promise<number> {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
getPersistMode(): 'immediate' | 'deferred' {
|
|
||||||
return 'deferred'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Registers `provider` under the `'vector'` plugin key, before `init()`. */
|
|
||||||
function useFakeVectorProvider(brain: Brainy<any>, provider: FakeVectorProvider): void {
|
|
||||||
brain.use({
|
|
||||||
name: 'fake-vector-provider',
|
|
||||||
activate: async (ctx: any) => {
|
|
||||||
ctx.registerProvider('vector', () => provider)
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('brain.warm()', () => {
|
|
||||||
it('(a) calls the vector provider\'s warm() when present and reports "warmed"', async () => {
|
|
||||||
const brain = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'memory' },
|
|
||||||
silent: true
|
|
||||||
})
|
|
||||||
const provider = new FakeVectorProvider(true)
|
|
||||||
useFakeVectorProvider(brain, provider)
|
|
||||||
await brain.init()
|
|
||||||
await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
|
|
||||||
|
|
||||||
const report = await brain.warm()
|
|
||||||
|
|
||||||
expect(provider.warmCalls).toBe(1)
|
|
||||||
expect(provider.searchCalls.length).toBe(0) // warm() ran — no probe fallback
|
|
||||||
expect(report.vector.outcome).toBe('warmed')
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(b) falls back to a probe search() when warm() is absent and reports "probed", never "warmed"', async () => {
|
|
||||||
const brain = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'memory' },
|
|
||||||
silent: true
|
|
||||||
})
|
|
||||||
const provider = new FakeVectorProvider(false)
|
|
||||||
useFakeVectorProvider(brain, provider)
|
|
||||||
await brain.init()
|
|
||||||
await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
|
|
||||||
await brain.add({ data: 'b', type: NounType.Thing, vector: V(2) })
|
|
||||||
|
|
||||||
const report = await brain.warm()
|
|
||||||
|
|
||||||
expect(provider.warmCalls).toBe(0) // no warm() on this provider
|
|
||||||
expect(provider.searchCalls.length).toBe(1) // the probe ran
|
|
||||||
expect(provider.searchCalls[0].k).toBe(Math.min(100, provider.size())) // k = min(100, size)
|
|
||||||
expect(report.vector.outcome).toBe('probed')
|
|
||||||
expect(report.vector.outcome).not.toBe('warmed') // never conflated
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('vector: reports "unavailable" when there is nothing to probe (empty index, unknown dimension)', async () => {
|
|
||||||
const brain = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'memory' },
|
|
||||||
silent: true
|
|
||||||
})
|
|
||||||
// A disk-native provider MAY honestly report 0 resident entries while
|
|
||||||
// durable data exists on disk (the same posture documented on
|
|
||||||
// `VectorIndexProvider.isReady` — "an mmap/disk-native index may
|
|
||||||
// legitimately report 0 resident entries"). warm()'s probe fallback
|
|
||||||
// reads `size()`, so this is the real trigger for "nothing to probe":
|
|
||||||
// never a k=0 search, an honest skip.
|
|
||||||
const provider = new FakeVectorProvider(false)
|
|
||||||
provider.size = () => 0
|
|
||||||
useFakeVectorProvider(brain, provider)
|
|
||||||
await brain.init() // the VFS root bootstrap write sets `dimensions`, but size() still reports 0
|
|
||||||
|
|
||||||
const report = await brain.warm()
|
|
||||||
|
|
||||||
expect(provider.searchCalls.length).toBe(0) // nothing probed
|
|
||||||
expect(report.vector.outcome).toBe('unavailable')
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(c) metadata + graph hydration paths actually execute against filesystem storage with pre-existing data', async () => {
|
|
||||||
const dir = mkTmp()
|
|
||||||
|
|
||||||
// Build a brain with real data (including a field NOT in the metadata
|
|
||||||
// index's own common-fields warm subset — 'wave' — so hydrateAll()'s
|
|
||||||
// FULL hydration is distinguishable from init()'s partial warmCache()),
|
|
||||||
// and real graph edges, then close it (persisting everything).
|
|
||||||
const seed = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'filesystem', path: dir },
|
|
||||||
silent: true
|
|
||||||
})
|
|
||||||
await seed.init()
|
|
||||||
const ids: string[] = []
|
|
||||||
for (let i = 0; i < 6; i++) {
|
|
||||||
ids.push(
|
|
||||||
await seed.add({
|
|
||||||
data: `entity ${i}`,
|
|
||||||
type: NounType.Thing,
|
|
||||||
metadata: { wave: i % 3 },
|
|
||||||
vector: V(i + 1)
|
|
||||||
})
|
|
||||||
)
|
|
||||||
}
|
|
||||||
for (let i = 0; i + 1 < ids.length; i++) {
|
|
||||||
await seed.relate({ from: ids[i], to: ids[i + 1], type: VerbType.RelatedTo })
|
|
||||||
}
|
|
||||||
await seed.close()
|
|
||||||
|
|
||||||
// Cold-reopen a FRESH instance and spy on the real hydration seams before
|
|
||||||
// calling warm(), so we assert they actually ran (not just that the
|
|
||||||
// report claims they did).
|
|
||||||
const metaHydrateCalls: number[] = []
|
|
||||||
const loadedFields: string[] = []
|
|
||||||
const graphInitCalls: number[] = []
|
|
||||||
|
|
||||||
const origHydrateAll = MetadataIndexManager.prototype.hydrateAll
|
|
||||||
const origLoadSparseIndex = (MetadataIndexManager.prototype as any).loadSparseIndex
|
|
||||||
const origGraphInit = GraphAdjacencyIndex.prototype.init
|
|
||||||
|
|
||||||
MetadataIndexManager.prototype.hydrateAll = async function (...args: any[]) {
|
|
||||||
metaHydrateCalls.push(1)
|
|
||||||
return origHydrateAll.apply(this, args as any)
|
|
||||||
}
|
|
||||||
;(MetadataIndexManager.prototype as any).loadSparseIndex = async function (
|
|
||||||
field: string,
|
|
||||||
...args: any[]
|
|
||||||
) {
|
|
||||||
loadedFields.push(field)
|
|
||||||
return origLoadSparseIndex.apply(this, [field, ...args] as any)
|
|
||||||
}
|
|
||||||
GraphAdjacencyIndex.prototype.init = async function (...args: any[]) {
|
|
||||||
graphInitCalls.push(1)
|
|
||||||
return origGraphInit.apply(this, args as any)
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
const brain = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'filesystem', path: dir },
|
|
||||||
silent: true
|
|
||||||
})
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
const report = await brain.warm()
|
|
||||||
|
|
||||||
expect(metaHydrateCalls.length).toBe(1) // hydrateAll() actually ran
|
|
||||||
expect(loadedFields).toContain('wave') // a NON-common field was loaded — full hydration, not the heuristic subset
|
|
||||||
expect(report.metadata.outcome).toBe('warmed')
|
|
||||||
|
|
||||||
expect(graphInitCalls.length).toBeGreaterThanOrEqual(1) // graph's full-load seam ran (once at brain init, once via warm())
|
|
||||||
expect(report.graph.outcome).toBe('warmed')
|
|
||||||
|
|
||||||
// Correctness survives: the hydrated data still answers queries.
|
|
||||||
const byWhere = await brain.find({ type: NounType.Thing, where: { wave: 1 } })
|
|
||||||
expect(byWhere.length).toBe(2) // waves 1,4 of 0..5
|
|
||||||
|
|
||||||
await brain.close()
|
|
||||||
} finally {
|
|
||||||
MetadataIndexManager.prototype.hydrateAll = origHydrateAll
|
|
||||||
;(MetadataIndexManager.prototype as any).loadSparseIndex = origLoadSparseIndex
|
|
||||||
GraphAdjacencyIndex.prototype.init = origGraphInit
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(d) warmOnOpen: true runs warm() during init() — observable via the fake provider', async () => {
|
|
||||||
const brain = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'memory' },
|
|
||||||
warmOnOpen: true,
|
|
||||||
silent: true
|
|
||||||
})
|
|
||||||
const provider = new FakeVectorProvider(true)
|
|
||||||
useFakeVectorProvider(brain, provider)
|
|
||||||
|
|
||||||
// No explicit brain.warm() call — warmOnOpen must have run it as part of init().
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
expect(provider.warmCalls).toBe(1)
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('warmOnOpen defaults to false — init() does NOT run warm() unless opted in', async () => {
|
|
||||||
const brain = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'memory' },
|
|
||||||
silent: true
|
|
||||||
})
|
|
||||||
const provider = new FakeVectorProvider(true)
|
|
||||||
useFakeVectorProvider(brain, provider)
|
|
||||||
|
|
||||||
await brain.init()
|
|
||||||
|
|
||||||
expect(provider.warmCalls).toBe(0)
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(e) WarmReport shape: outcome literal + durationMs per surface, plus totalDurationMs', async () => {
|
|
||||||
const brain = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'memory' },
|
|
||||||
silent: true
|
|
||||||
})
|
|
||||||
const provider = new FakeVectorProvider(true)
|
|
||||||
useFakeVectorProvider(brain, provider)
|
|
||||||
await brain.init()
|
|
||||||
await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
|
|
||||||
|
|
||||||
const report = await brain.warm()
|
|
||||||
|
|
||||||
for (const surface of ['vector', 'metadata', 'graph'] as const) {
|
|
||||||
expect(['warmed', 'probed', 'unavailable']).toContain(report[surface].outcome)
|
|
||||||
expect(typeof report[surface].durationMs).toBe('number')
|
|
||||||
expect(report[surface].durationMs).toBeGreaterThanOrEqual(0)
|
|
||||||
}
|
|
||||||
expect(typeof report.totalDurationMs).toBe('number')
|
|
||||||
expect(report.totalDurationMs).toBeGreaterThanOrEqual(0)
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -186,52 +186,4 @@ describe('fact log — round-trip, framing, reconcile, rotation, scan', () => {
|
||||||
await log.sync()
|
await log.sync()
|
||||||
expect(log.segmentPaths()).toEqual([]) // only a tail exists — nothing sealed
|
expect(log.segmentPaths()).toEqual([]) // only a tail exists — nothing sealed
|
||||||
})
|
})
|
||||||
|
|
||||||
describe('scanFacts liveness contract (Stage-2 D1)', () => {
|
|
||||||
it('a wedged store fails LOUDLY within the first-batch bound — never a silent hang', async () => {
|
|
||||||
// Force a sealed segment (tiny rotateBytes) so the scan must READ from
|
|
||||||
// storage, then wedge that read: the exact production shape (a
|
|
||||||
// backlogged brain whose segment read never returned).
|
|
||||||
const mem: any = new MemoryStorage()
|
|
||||||
await mem.init()
|
|
||||||
const wedgeable = new FactLog(mem, { rotateBytes: 1 })
|
|
||||||
await wedgeable.open(0)
|
|
||||||
await wedgeable.append(fact(1))
|
|
||||||
await wedgeable.append(fact(2)) // second append rotates → seg 1 sealed
|
|
||||||
await wedgeable.sync()
|
|
||||||
|
|
||||||
const realRead = mem.readRawBytes.bind(mem)
|
|
||||||
mem.readRawBytes = (p: string) =>
|
|
||||||
p.includes('facts/seg-') ? new Promise(() => {}) : realRead(p) // hangs forever
|
|
||||||
|
|
||||||
const scan = wedgeable.scanFacts({ firstBatchTimeoutMs: 200 })
|
|
||||||
const started = Date.now()
|
|
||||||
await expect(scan.batches().next()).rejects.toThrow(/no first batch within 200ms/)
|
|
||||||
expect(Date.now() - started).toBeLessThan(5_000) // bound held, not a hang
|
|
||||||
})
|
|
||||||
|
|
||||||
it('a healthy scan is unaffected — first batch well inside the bound, all facts delivered', async () => {
|
|
||||||
for (let g = 1; g <= 5; g++) await log.append(fact(g))
|
|
||||||
await log.sync()
|
|
||||||
const scan = log.scanFacts({ batchSize: 2 })
|
|
||||||
const all: CommitFact[] = []
|
|
||||||
for await (const b of scan.batches()) all.push(...b.facts)
|
|
||||||
expect(all.map((f) => f.generation)).toEqual([1, 2, 3, 4, 5])
|
|
||||||
expect(scan.summary().factsYielded).toBe(5)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('consumer think-time between pulls never counts against the producer', async () => {
|
|
||||||
for (let g = 1; g <= 4; g++) await log.append(fact(g))
|
|
||||||
await log.sync()
|
|
||||||
// Bound tighter than the consumer's pause: only the FIRST pull is
|
|
||||||
// raced, so a slow consumer after batch 1 must not trip the deadline.
|
|
||||||
const gen = log.scanFacts({ batchSize: 2, firstBatchTimeoutMs: 150 }).batches()
|
|
||||||
const first = await gen.next()
|
|
||||||
expect(first.done).toBe(false)
|
|
||||||
await new Promise((r) => setTimeout(r, 400)) // dawdle past the bound
|
|
||||||
const second = await gen.next()
|
|
||||||
expect(second.done).toBe(false)
|
|
||||||
expect((await gen.next()).done).toBe(true)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -1,150 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/unit/db/generation-segments
|
|
||||||
* @description The generation-segment store (Stage-2 D1+D3 file format).
|
|
||||||
* Laws: (1) fold → read round-trips deltas and records byte-faithfully via
|
|
||||||
* sidecar point-reads; (2) the manifest is the ONLY discovery path — reopen
|
|
||||||
* reads one file, never a listing; (3) a lost/corrupt sidecar rebuilds from
|
|
||||||
* its segment loudly, a damaged SEGMENT fails loudly (never silent wrong
|
|
||||||
* data); (4) D3 reclaim drops whole segments only and bumps compactedBelow;
|
|
||||||
* (5) the packed digest is deterministic across reopen; (6) immutability —
|
|
||||||
* fold refuses overlap with sealed ranges.
|
|
||||||
*/
|
|
||||||
import { describe, it, expect, beforeEach } from 'vitest'
|
|
||||||
import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js'
|
|
||||||
import {
|
|
||||||
GenerationSegmentStore,
|
|
||||||
SEGMENTS_PREFIX,
|
|
||||||
type FoldGeneration
|
|
||||||
} from '../../../src/db/generationSegments.js'
|
|
||||||
|
|
||||||
const UUID = (n: number): string => `00000000-0000-4000-8000-${String(n).padStart(12, '0')}`
|
|
||||||
|
|
||||||
const gen = (g: number, recordCount = 2): FoldGeneration => ({
|
|
||||||
generation: g,
|
|
||||||
timestamp: 1_700_000_000_000 + g,
|
|
||||||
delta: { generation: g, nouns: [UUID(g)], verbs: [], bytes: 123 + g },
|
|
||||||
records: Array.from({ length: recordCount }, (_, i) => ({
|
|
||||||
kind: (i % 2 === 0 ? 'noun' : 'verb') as 'noun' | 'verb',
|
|
||||||
id: UUID(g * 100 + i),
|
|
||||||
record: { metadata: { noun: 'document', v: g }, vector: { v: [g, i] } }
|
|
||||||
}))
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('db/GenerationSegmentStore — the D1+D3 packed tier', () => {
|
|
||||||
let storage: MemoryStorage
|
|
||||||
let store: GenerationSegmentStore
|
|
||||||
|
|
||||||
beforeEach(async () => {
|
|
||||||
storage = new MemoryStorage()
|
|
||||||
await storage.init()
|
|
||||||
store = new GenerationSegmentStore(storage as any)
|
|
||||||
await store.open()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('fold → read round-trips deltas and records via sidecar point-reads', async () => {
|
|
||||||
const meta = await store.fold([gen(1), gen(2), gen(3)])
|
|
||||||
expect(meta).toMatchObject({ firstGeneration: 1, lastGeneration: 3, frames: 3 })
|
|
||||||
expect(meta.checksum).toBeGreaterThan(0)
|
|
||||||
|
|
||||||
expect(store.hasGeneration(2)).toBe(true)
|
|
||||||
expect(store.hasGeneration(4)).toBe(false)
|
|
||||||
|
|
||||||
const d2 = await store.readDelta(2)
|
|
||||||
expect(d2?.delta).toEqual({ generation: 2, nouns: [UUID(2)], verbs: [], bytes: 125 })
|
|
||||||
expect(d2?.timestamp).toBe(1_700_000_000_002)
|
|
||||||
|
|
||||||
const records = await store.readRecords(3)
|
|
||||||
expect(records).toHaveLength(2)
|
|
||||||
expect(records![0]).toEqual({
|
|
||||||
kind: 'noun',
|
|
||||||
id: UUID(300),
|
|
||||||
record: { metadata: { noun: 'document', v: 3 }, vector: { v: [3, 0] } }
|
|
||||||
})
|
|
||||||
// Point read by id, both kinds.
|
|
||||||
expect(await store.readRecord(3, 'verb', UUID(301))).toEqual({
|
|
||||||
metadata: { noun: 'document', v: 3 },
|
|
||||||
vector: { v: [3, 1] }
|
|
||||||
})
|
|
||||||
expect(await store.readRecord(3, 'noun', UUID(999))).toBeNull()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('reopen discovers everything from the manifest alone — no listing', async () => {
|
|
||||||
await store.fold([gen(1), gen(2)])
|
|
||||||
await store.fold([gen(3), gen(4)])
|
|
||||||
|
|
||||||
const reopened = new GenerationSegmentStore(storage as any)
|
|
||||||
await reopened.open()
|
|
||||||
expect(reopened.segments()).toHaveLength(2)
|
|
||||||
expect(reopened.hasGeneration(4)).toBe(true)
|
|
||||||
expect((await reopened.readDelta(1))?.timestamp).toBe(1_700_000_000_001)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('a lost sidecar rebuilds from its segment; a damaged segment fails LOUDLY', async () => {
|
|
||||||
const meta = await store.fold([gen(1), gen(2)])
|
|
||||||
const idxPath = `${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.idx`
|
|
||||||
await storage.deleteRawObject(idxPath)
|
|
||||||
|
|
||||||
const reopened = new GenerationSegmentStore(storage as any)
|
|
||||||
await reopened.open()
|
|
||||||
// Rebuild path: still serves correct data.
|
|
||||||
expect((await reopened.readRecords(2))!).toHaveLength(2)
|
|
||||||
|
|
||||||
// Now damage the SEGMENT itself: flip a payload byte → CRC mismatch, loud.
|
|
||||||
const segPath = `${SEGMENTS_PREFIX}/${meta.file}`
|
|
||||||
const bytes = (await storage.readRawBytes(segPath))!
|
|
||||||
bytes[bytes.length - 3] ^= 0xff
|
|
||||||
await storage.writeRawBytes(segPath, bytes)
|
|
||||||
const damaged = new GenerationSegmentStore(storage as any)
|
|
||||||
await damaged.open()
|
|
||||||
;(damaged as any).sidecars.clear()
|
|
||||||
await storage.deleteRawObject(idxPath) // force the sequential rebuild over damaged bytes
|
|
||||||
await expect(damaged.readRecords(2)).rejects.toThrow(/CRC mismatch|damaged/)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('D3 reclaim drops whole segments only and bumps compactedBelow', async () => {
|
|
||||||
await store.fold([gen(1), gen(2)])
|
|
||||||
await store.fold([gen(3), gen(4)])
|
|
||||||
await store.fold([gen(5), gen(6)])
|
|
||||||
|
|
||||||
// Horizon mid-segment-2 (below 4): only segment 1 is FULLY below → drops.
|
|
||||||
const r1 = await store.dropSegmentsBelow(4)
|
|
||||||
expect(r1).toEqual({ dropped: 1, compactedBelow: 3 })
|
|
||||||
expect(store.hasGeneration(1)).toBe(false)
|
|
||||||
expect(store.hasGeneration(3)).toBe(true) // partial segment survives whole
|
|
||||||
|
|
||||||
// Bytes actually gone.
|
|
||||||
expect(await storage.readRawBytes(`${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.bgs`)).toBeNull()
|
|
||||||
|
|
||||||
// Horizon past everything: the rest drop; compactedBelow is durable.
|
|
||||||
const r2 = await store.dropSegmentsBelow(7)
|
|
||||||
expect(r2.dropped).toBe(2)
|
|
||||||
const reopened = new GenerationSegmentStore(storage as any)
|
|
||||||
await reopened.open()
|
|
||||||
expect(reopened.compactedBelow()).toBe(7)
|
|
||||||
expect(reopened.segments()).toHaveLength(0)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('the packed digest is deterministic across reopen and changes with history', async () => {
|
|
||||||
await store.fold([gen(1), gen(2), gen(3)])
|
|
||||||
const atSeal = await store.digestThroughPacked(3)
|
|
||||||
const midSegment = await store.digestThroughPacked(2)
|
|
||||||
expect(atSeal).not.toBeNull()
|
|
||||||
expect(midSegment).not.toBeNull()
|
|
||||||
expect(midSegment).not.toBe(atSeal)
|
|
||||||
|
|
||||||
const reopened = new GenerationSegmentStore(storage as any)
|
|
||||||
await reopened.open()
|
|
||||||
expect(await reopened.digestThroughPacked(3)).toBe(atSeal)
|
|
||||||
expect(await reopened.digestThroughPacked(2)).toBe(midSegment)
|
|
||||||
|
|
||||||
await reopened.fold([gen(4)])
|
|
||||||
expect(await reopened.digestThroughPacked(4)).not.toBe(atSeal)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('sealed segments are immutable — fold refuses overlap, requires ascending input', async () => {
|
|
||||||
await store.fold([gen(1), gen(2)])
|
|
||||||
await expect(store.fold([gen(2), gen(3)])).rejects.toThrow(/overlaps the packed tier/)
|
|
||||||
await expect(store.fold([gen(4), gen(4)])).rejects.toThrow(/strictly ascending/)
|
|
||||||
await expect(store.fold([])).rejects.toThrow(/at least one generation/)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -488,20 +488,6 @@ describe('db/GenerationStore', () => {
|
||||||
expect(result.removedGenerations).toBe(2)
|
expect(result.removedGenerations).toBe(2)
|
||||||
store.release(2)
|
store.release(2)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('timeBudgetMs bounds a pass; the next pass resumes the same prefix', async () => {
|
|
||||||
await manyGens(4)
|
|
||||||
// A spent budget (0ms) stops before reclaiming anything — an early stop
|
|
||||||
// is a consistent prefix, never a partial generation.
|
|
||||||
const bounded = await store.compact({ timeBudgetMs: 0 })
|
|
||||||
expect(bounded.removedGenerations).toBe(0)
|
|
||||||
expect(bounded.horizon).toBe(0)
|
|
||||||
// The next (unbounded) pass picks up exactly where the bounded one
|
|
||||||
// stopped and completes the same work.
|
|
||||||
const resumed = await store.compact()
|
|
||||||
expect(resumed.removedGenerations).toBe(4)
|
|
||||||
expect(resumed.horizon).toBe(4)
|
|
||||||
})
|
|
||||||
})
|
})
|
||||||
|
|
||||||
// ==========================================================================
|
// ==========================================================================
|
||||||
|
|
|
||||||
|
|
@ -1,149 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/unit/transaction/budget-commit-completed-work
|
|
||||||
* @description Regression coverage for the "commit-completed-work" transaction
|
|
||||||
* budget semantics: the budget gates STARTING the next operation; it never
|
|
||||||
* converts already-completed work into a rollback. Concretely:
|
|
||||||
*
|
|
||||||
* - A single-op transaction can never time out post-hoc — its one operation
|
|
||||||
* either runs (and the transaction commits, however long it took) or it
|
|
||||||
* never gets to start (which cannot happen: there is no operation before it
|
|
||||||
* to have overrun the budget).
|
|
||||||
* - A multi-op transaction whose op `i` overruns the budget stops op `i+1`
|
|
||||||
* from starting: everything applied so far rolls back atomically and a
|
|
||||||
* `TransactionTimeoutError` is thrown — the mid-batch zero-loss guarantee is
|
|
||||||
* unchanged.
|
|
||||||
* - `transactTimeoutBudget`'s scaling floor (`max(floorMs, opCount * 2000)`)
|
|
||||||
* is overridable per call, the seam `BrainyConfig.transactionBudgetFloorMs`
|
|
||||||
* feeds at the brainy.ts read sites.
|
|
||||||
*
|
|
||||||
* Uses a real class implementing the `Operation` interface (not an anonymous
|
|
||||||
* object literal, not a mock of Transaction internals) so the exercised path
|
|
||||||
* is exactly what a real caller's operation looks like.
|
|
||||||
*/
|
|
||||||
import { describe, it, expect } from 'vitest'
|
|
||||||
import { Transaction, transactTimeoutBudget } from '../../../src/transaction/Transaction.js'
|
|
||||||
import type { Operation, RollbackAction } from '../../../src/transaction/types.js'
|
|
||||||
import { TransactionTimeoutError } from '../../../src/transaction/errors.js'
|
|
||||||
|
|
||||||
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms))
|
|
||||||
|
|
||||||
/**
|
|
||||||
* A real `Operation` implementation that sleeps for `delayMs` before applying
|
|
||||||
* a write to `log`, and returns a rollback action that removes it again. Used
|
|
||||||
* to deterministically make one operation "slow" relative to a transaction's
|
|
||||||
* budget without touching any Transaction internals.
|
|
||||||
*/
|
|
||||||
class SlowOperation implements Operation {
|
|
||||||
readonly name: string
|
|
||||||
executed = false
|
|
||||||
rolledBack = false
|
|
||||||
|
|
||||||
constructor(
|
|
||||||
private readonly log: string[],
|
|
||||||
label: string,
|
|
||||||
private readonly delayMs: number
|
|
||||||
) {
|
|
||||||
this.name = label
|
|
||||||
}
|
|
||||||
|
|
||||||
async execute(): Promise<RollbackAction | undefined> {
|
|
||||||
this.executed = true
|
|
||||||
if (this.delayMs > 0) await sleep(this.delayMs)
|
|
||||||
this.log.push(this.name)
|
|
||||||
return async () => {
|
|
||||||
this.rolledBack = true
|
|
||||||
const idx = this.log.indexOf(this.name)
|
|
||||||
if (idx >= 0) this.log.splice(idx, 1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('Transaction budget — commit-completed-work semantics', () => {
|
|
||||||
it('(a) single-op transaction whose op overruns the budget COMMITS — no rollback, no throw', async () => {
|
|
||||||
const log: string[] = []
|
|
||||||
const op = new SlowOperation(log, 'slow-single-op', 40)
|
|
||||||
|
|
||||||
// Budget (5ms) is far smaller than the op's 40ms — under the OLD
|
|
||||||
// (post-completion-check) semantics this would have thrown and rolled
|
|
||||||
// back after the op finished. Under the new semantics there is no
|
|
||||||
// operation after it to gate, so it commits.
|
|
||||||
const tx = new Transaction({ timeout: 5 })
|
|
||||||
tx.addOperation(op)
|
|
||||||
|
|
||||||
await expect(tx.execute()).resolves.toBeUndefined()
|
|
||||||
|
|
||||||
expect(tx.getState()).toBe('committed')
|
|
||||||
expect(op.executed).toBe(true)
|
|
||||||
expect(op.rolledBack).toBe(false)
|
|
||||||
expect(log).toEqual(['slow-single-op'])
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(b) two-op transaction: op0 overruns the budget → op1 never starts, op0 rolls back, throws TransactionTimeoutError', async () => {
|
|
||||||
const log: string[] = []
|
|
||||||
const op0 = new SlowOperation(log, 'op0-overruns', 40)
|
|
||||||
const op1 = new SlowOperation(log, 'op1-never-starts', 0)
|
|
||||||
|
|
||||||
const tx = new Transaction({ timeout: 5 })
|
|
||||||
tx.addOperation(op0)
|
|
||||||
tx.addOperation(op1)
|
|
||||||
|
|
||||||
const error = await tx.execute().catch((e) => e)
|
|
||||||
|
|
||||||
expect(error).toBeInstanceOf(TransactionTimeoutError)
|
|
||||||
expect(op0.executed).toBe(true)
|
|
||||||
expect(op0.rolledBack).toBe(true)
|
|
||||||
expect(op1.executed).toBe(false)
|
|
||||||
expect(op1.rolledBack).toBe(false)
|
|
||||||
expect(log).toEqual([]) // op0's write was undone; op1 never wrote
|
|
||||||
expect(tx.getState()).toBe('rolled_back')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('a single-op transaction never checks the budget before its only operation starts', async () => {
|
|
||||||
// Budget of 0ms: under the old "check before every op including 0" code
|
|
||||||
// this could theoretically trip before op 0 even started (if any
|
|
||||||
// measurable time elapsed between startTime capture and the check). The
|
|
||||||
// documented contract is stronger: operation 0 ALWAYS gets to start.
|
|
||||||
const log: string[] = []
|
|
||||||
const op = new SlowOperation(log, 'only-op', 5)
|
|
||||||
|
|
||||||
const tx = new Transaction({ timeout: 0 })
|
|
||||||
tx.addOperation(op)
|
|
||||||
|
|
||||||
await expect(tx.execute()).resolves.toBeUndefined()
|
|
||||||
expect(tx.getState()).toBe('committed')
|
|
||||||
expect(op.executed).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(c) transactTimeoutBudget: an explicit floorMs overrides the 30s default floor', () => {
|
|
||||||
// Below the per-op scaling term, a raised floor wins.
|
|
||||||
expect(transactTimeoutBudget(1, undefined, 5_000)).toBe(5_000) // 1 op × 2000 = 2000 < 5000 floor
|
|
||||||
expect(transactTimeoutBudget(1)).toBe(30_000) // unchanged default when floorMs omitted
|
|
||||||
|
|
||||||
// Above the floor, opCount * 2000 still wins over a smaller floor.
|
|
||||||
expect(transactTimeoutBudget(10, undefined, 5_000)).toBe(20_000) // 10 × 2000 = 20000 > 5000 floor
|
|
||||||
|
|
||||||
// A full `override` always wins, regardless of floorMs.
|
|
||||||
expect(transactTimeoutBudget(10, 999, 5_000)).toBe(999)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(c) a configured floor changes real Transaction behavior end-to-end via TransactionManager-style options', async () => {
|
|
||||||
// Simulates how brainy.ts wires `this.config.transactionBudgetFloorMs`
|
|
||||||
// into `transactTimeoutBudget()`: opCount 0 isolates the floor term
|
|
||||||
// (0 × 2000 = 0), so a lowered floor makes a normally-generous budget
|
|
||||||
// fail-fast for a real 2-op Transaction.
|
|
||||||
const lowFloorBudget = transactTimeoutBudget(0, undefined, 10) // 10ms floor
|
|
||||||
expect(lowFloorBudget).toBe(10)
|
|
||||||
|
|
||||||
const log: string[] = []
|
|
||||||
const op0 = new SlowOperation(log, 'op0', 30)
|
|
||||||
const op1 = new SlowOperation(log, 'op1-gated-by-lowered-floor', 0)
|
|
||||||
|
|
||||||
const tx = new Transaction({ timeout: lowFloorBudget })
|
|
||||||
tx.addOperation(op0)
|
|
||||||
tx.addOperation(op1)
|
|
||||||
|
|
||||||
const error = await tx.execute().catch((e) => e)
|
|
||||||
expect(error).toBeInstanceOf(TransactionTimeoutError)
|
|
||||||
expect(op1.executed).toBe(false)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -1,176 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/unit/transaction/vectorIndexOperations-rename
|
|
||||||
* @description Coverage for the backend-neutral vector-index transaction
|
|
||||||
* operations (formerly `AddToHNSWOperation` / `RemoveFromHNSWOperation`).
|
|
||||||
* These op-name strings surface directly in consumer-visible transaction
|
|
||||||
* journals and timings — a fossil "HNSW" name misdirects an operator running
|
|
||||||
* a native (non-HNSW) vector provider. Verifies:
|
|
||||||
* 1. rollback wiring stayed byte-identical under the rename,
|
|
||||||
* 2. the emitted `name` stamps the active backend — `'hnsw-js'` for
|
|
||||||
* Brainy's own JS index, a provider's own required `name` when it
|
|
||||||
* self-identifies, and the tolerant-loud `'unknown-provider'` literal
|
|
||||||
* (plus one console.warn) when a runtime instance lacks `name`
|
|
||||||
* altogether (an older native provider compiled against the previous,
|
|
||||||
* optional `providerId` contract) — never a silently-wrong guess.
|
|
||||||
*/
|
|
||||||
import { describe, it, expect, vi, afterEach } from 'vitest'
|
|
||||||
import {
|
|
||||||
AddToVectorIndexOperation,
|
|
||||||
RemoveFromVectorIndexOperation,
|
|
||||||
BatchAddToVectorIndexOperation
|
|
||||||
} from '../../../src/transaction/operations/IndexOperations.js'
|
|
||||||
import type { VectorIndexProvider } from '../../../src/plugin.js'
|
|
||||||
import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
|
|
||||||
import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* A minimal, real (not mocked) VectorIndexProvider implementation backed by
|
|
||||||
* a plain Map — exercises the exact contract the operations call through,
|
|
||||||
* without any Brainy internals.
|
|
||||||
*/
|
|
||||||
class FakeVectorProvider implements VectorIndexProvider {
|
|
||||||
readonly items = new Map<string, Vector>()
|
|
||||||
constructor(readonly name: string) {}
|
|
||||||
|
|
||||||
async addItem(item: VectorDocument): Promise<string> {
|
|
||||||
this.items.set(item.id, item.vector)
|
|
||||||
return item.id
|
|
||||||
}
|
|
||||||
async removeItem(id: string): Promise<boolean> {
|
|
||||||
return this.items.delete(id)
|
|
||||||
}
|
|
||||||
async getItem(id: string): Promise<VectorDocument | undefined> {
|
|
||||||
const vector = this.items.get(id)
|
|
||||||
return vector ? { id, vector } : undefined
|
|
||||||
}
|
|
||||||
async search(): Promise<Array<[string, number]>> {
|
|
||||||
return []
|
|
||||||
}
|
|
||||||
size(): number {
|
|
||||||
return this.items.size
|
|
||||||
}
|
|
||||||
clear(): void {
|
|
||||||
this.items.clear()
|
|
||||||
}
|
|
||||||
async rebuild(): Promise<void> {}
|
|
||||||
async flush(): Promise<number> {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
getPersistMode(): 'immediate' | 'deferred' {
|
|
||||||
return 'immediate'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('Vector index transaction operations — backend-neutral rename', () => {
|
|
||||||
describe('rollback wiring (byte-identical behavior under the rename)', () => {
|
|
||||||
it('AddToVectorIndexOperation rollback removes a newly-added item', async () => {
|
|
||||||
const provider = new FakeVectorProvider('fake-provider')
|
|
||||||
const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
|
||||||
|
|
||||||
const rollback = await op.execute()
|
|
||||||
expect(provider.items.has('id-1')).toBe(true)
|
|
||||||
|
|
||||||
await rollback()
|
|
||||||
expect(provider.items.has('id-1')).toBe(false)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('AddToVectorIndexOperation rollback is a no-op when the item pre-existed (update semantics)', async () => {
|
|
||||||
const provider = new FakeVectorProvider('fake-provider')
|
|
||||||
await provider.addItem({ id: 'id-1', vector: [9, 9, 9] })
|
|
||||||
|
|
||||||
const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
|
||||||
const rollback = await op.execute()
|
|
||||||
expect(provider.items.get('id-1')).toEqual([1, 2, 3])
|
|
||||||
|
|
||||||
await rollback()
|
|
||||||
// Pre-existing item is NOT removed by rollback — it stays (the update
|
|
||||||
// itself is not undone by this op; that's RemoveFromVectorIndexOperation's job).
|
|
||||||
expect(provider.items.has('id-1')).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('RemoveFromVectorIndexOperation rollback re-adds the item with its original vector', async () => {
|
|
||||||
const provider = new FakeVectorProvider('fake-provider')
|
|
||||||
await provider.addItem({ id: 'id-1', vector: [4, 5, 6] })
|
|
||||||
|
|
||||||
const op = new RemoveFromVectorIndexOperation(provider, 'id-1', [4, 5, 6])
|
|
||||||
const rollback = await op.execute()
|
|
||||||
expect(provider.items.has('id-1')).toBe(false)
|
|
||||||
|
|
||||||
await rollback()
|
|
||||||
expect(provider.items.get('id-1')).toEqual([4, 5, 6])
|
|
||||||
})
|
|
||||||
|
|
||||||
it('BatchAddToVectorIndexOperation rolls back every item in reverse order on undo', async () => {
|
|
||||||
const provider = new FakeVectorProvider('fake-provider')
|
|
||||||
const op = new BatchAddToVectorIndexOperation(provider, [
|
|
||||||
{ id: 'a', vector: [1] },
|
|
||||||
{ id: 'b', vector: [2] },
|
|
||||||
{ id: 'c', vector: [3] }
|
|
||||||
])
|
|
||||||
|
|
||||||
const rollback = await op.execute()
|
|
||||||
expect([...provider.items.keys()].sort()).toEqual(['a', 'b', 'c'])
|
|
||||||
|
|
||||||
await rollback()
|
|
||||||
expect(provider.items.size).toBe(0)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('backend stamping in the emitted op name', () => {
|
|
||||||
afterEach(() => {
|
|
||||||
vi.restoreAllMocks()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('stamps the real JS HNSW index as hnsw-js (self-identifying name)', () => {
|
|
||||||
const index = new JsHnswVectorIndex()
|
|
||||||
const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3])
|
|
||||||
const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3])
|
|
||||||
|
|
||||||
expect(addOp.name).toBe('AddToVectorIndex(hnsw-js)')
|
|
||||||
expect(removeOp.name).toBe('RemoveFromVectorIndex(hnsw-js)')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('stamps a self-identifying provider\'s own name, never "HNSW"', () => {
|
|
||||||
const provider = new FakeVectorProvider('acme-diskann')
|
|
||||||
const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
|
||||||
const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
|
||||||
const batchOp = new BatchAddToVectorIndexOperation(provider, [{ id: 'a', vector: [1] }])
|
|
||||||
|
|
||||||
expect(addOp.name).toBe('AddToVectorIndex(acme-diskann)')
|
|
||||||
expect(removeOp.name).toBe('RemoveFromVectorIndex(acme-diskann)')
|
|
||||||
expect(batchOp.name).toBe('BatchAddToVectorIndex(acme-diskann)')
|
|
||||||
expect(addOp.name).not.toContain('HNSW')
|
|
||||||
expect(removeOp.name).not.toContain('HNSW')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('tolerant-loud: stamps "unknown-provider" and warns exactly once when a runtime provider lacks the required `name` (an older native provider compiled against the previous optional `providerId` contract)', () => {
|
|
||||||
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
|
|
||||||
// Simulate a native provider compiled before `name` was required — the
|
|
||||||
// TypeScript contract requires `name`, but `as unknown as` mirrors what
|
|
||||||
// actually reaches this code at runtime from an un-rebuilt native addon.
|
|
||||||
const legacyProvider = {
|
|
||||||
addItem: async (item: VectorDocument) => item.id,
|
|
||||||
removeItem: async () => true,
|
|
||||||
search: async () => [],
|
|
||||||
size: () => 0,
|
|
||||||
clear: () => {},
|
|
||||||
rebuild: async () => {},
|
|
||||||
flush: async () => 0,
|
|
||||||
getPersistMode: () => 'immediate' as const
|
|
||||||
} as unknown as VectorIndexProvider
|
|
||||||
|
|
||||||
const addOp = new AddToVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
|
|
||||||
const removeOp = new RemoveFromVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
|
|
||||||
|
|
||||||
expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)')
|
|
||||||
expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)')
|
|
||||||
// Never silently falls back to the JS engine's name for a provider it
|
|
||||||
// knows nothing about — that's the exact fossil-naming bug being fixed.
|
|
||||||
expect(addOp.name).not.toContain('hnsw-js')
|
|
||||||
// Loud, not silent — but exactly once per provider instance, not once
|
|
||||||
// per op stamped against it.
|
|
||||||
expect(warnSpy).toHaveBeenCalledTimes(1)
|
|
||||||
expect(warnSpy.mock.calls[0][0]).toContain('name')
|
|
||||||
})
|
|
||||||
})
|
|
||||||
})
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue