diff --git a/.forgejo/workflows/ci.yml b/.github/workflows/ci.yml similarity index 100% rename from .forgejo/workflows/ci.yml rename to .github/workflows/ci.yml diff --git a/CHANGELOG.md b/CHANGELOG.md index b6dd91fd..fd0de54e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,15 +2,6 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. -### [8.10.0](https://github.com/soulcraftlabs/brainy/compare/v8.9.0...v8.10.0) (2026-07-23) - -- docs: adoption storefront — contributing guide, security policy, README support + cor section (9a99a7b) -- fix(release): push the public mirror explicitly and verify the tag lands at the right commit before publishing (6ba94c8) -- docs: project guide version line points at npm instead of a hardcoded stale number (3a1efc9) -- feat: vector provider identity is a required name field (hnsw-js), rendered [vector-index:] (3be4ba9) -- feat: warm contract (warm/warmOnOpen/provider warm hook), configurable transact budget floor, backend-neutral vector index op names (55b867c) - - ### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19) - docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78) diff --git a/CLAUDE.md b/CLAUDE.md index c7336a18..568c10db 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md` **Brainy's current open actions:** None. MIT open-source — no platform-specific actions. -**Current version:** run `npm view @soulcraft/brainy version` (never trust a hardcoded number here — this line went stale for months); consumer-facing changes tracked in `RELEASES.md` +**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`) --- diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d277091d..ab8a0246 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,66 +1,298 @@ # Contributing to Brainy -Brainy is MIT-licensed and genuinely open to outside contributions. This page -is the honest, current path — please don't rely on older instructions you -may find elsewhere in the repo's history. +Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for contributing to the project. -## Where the project lives +## Code of Conduct -The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/brainy**. -It's anonymously readable and cloneable — no account needed to browse, clone, -or build. +By participating in this project, you agree to abide by our Code of Conduct: +- Be respectful and inclusive +- Welcome newcomers and help them get started +- Focus on constructive criticism +- Respect differing viewpoints and experiences -## How to contribute +## How to Contribute -**Found a bug, or have an idea?** Email **brainy@soulcraft.com**. No account, -no ceremony — you'll get a receipt, and it goes to a human. +### Reporting Issues -**Want to send a patch?** Two ways, both first-class: +Before creating an issue, please check existing issues to avoid duplicates. -- **Email a patch.** Run `git format-patch` against your change and email the - output to **brainy@soulcraft.com**. This is a genuinely supported path, not - a fallback — plenty of good contributions arrive this way. -- **Open a pull request on the forge.** Request an account at - **source.soulcraft.com** (registration is request-with-approval, so allow - a little lag), clone, push a branch, and open a PR there. Maintainers - review and land it. +When creating an issue, include: +- Clear, descriptive title +- Detailed description of the problem +- Steps to reproduce +- Expected vs actual behavior +- System information (OS, Node version, Brainy version) +- Code examples if applicable -Either way, for anything beyond a small fix, opening an issue first (email is -fine) to talk through the approach saves everyone rework. +### Suggesting Features -## Development setup +Feature requests are welcome! Please provide: +- Clear use case +- Proposed API/interface +- Examples of how it would work +- Any potential challenges or considerations +### Pull Requests + +#### Before Starting + +1. Check existing issues and PRs +2. Open an issue to discuss significant changes +3. Fork the repository +4. Create a feature branch from `main` + +#### Development Setup + +**Quick Setup (Recommended):** ```bash -git clone https://source.soulcraft.com/soulcraft/brainy.git +# Clone your fork +git clone https://github.com/your-username/brainy.git cd brainy + +# Run setup script (installs all dependencies including Rust) +./scripts/setup-dev.sh +``` + +**Manual Setup:** +```bash +# Clone your fork +git clone https://github.com/your-username/brainy.git +cd brainy + +# Install system dependencies (Ubuntu/Debian) +sudo apt-get install -y build-essential pkg-config libssl-dev + +# Install Rust (for WASM embedding engine) +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source ~/.cargo/env +rustup target add wasm32-unknown-unknown +cargo install wasm-pack + +# Install Node.js dependencies npm install + +# Build Candle WASM embedding engine +npm run build:candle + +# Build TypeScript npm run build + +# Run tests npm test ``` -Tests run on [Vitest](https://vitest.dev/). `npm test` runs the unit suite; -see `package.json` for `test:integration`, `test:coverage`, and friends. +#### Making Changes -## Standards +1. **Follow the code style** + - TypeScript for all source code + - Clear variable and function names + - Comments for complex logic + - JSDoc for public APIs -- **Strict TypeScript.** No `any` escape hatches to dodge the type checker. -- **Tests exercise real behavior.** No mocking away the thing you're supposed - to be testing. -- **No stubs, no TODO-code.** If something can't be finished, say so and - leave it out — don't merge a placeholder. -- **JSDoc on every exported function, class, and type.** -- **[Conventional Commits](https://www.conventionalcommits.org/).** `feat:`, - `fix:`, `docs:`, `perf:`, `refactor:`, `test:`, `chore:`. Never - `BREAKING CHANGE` in a commit message — major version bumps are a separate, - deliberate decision. -- **Performance claims are measured or labeled projected.** If a PR or its - description states a number, cite the benchmark that produced it (see - [docs/performance-envelopes.md](docs/performance-envelopes.md) for the - pattern). Don't state an estimate as if it were measured. +2. **Write tests** + - Add tests for new features + - Update tests for changes + - Ensure all tests pass -## License +3. **Update documentation** + - Update README if needed + - Add/update API documentation + - Include examples -Brainy is [MIT licensed](LICENSE). Contributions are accepted under the same -license — there's no CLA to sign. +#### Commit Guidelines -Thank you for considering a contribution. +Follow conventional commits format: + +``` +type(scope): description + +[optional body] + +[optional footer] +``` + +Types: +- `feat`: New feature +- `fix`: Bug fix +- `docs`: Documentation changes +- `style`: Code style changes +- `refactor`: Code refactoring +- `perf`: Performance improvements +- `test`: Test changes +- `chore`: Build/tooling changes + +Examples: +```bash +feat(triple): add graph traversal depth limit +fix(storage): handle concurrent write conflicts +docs(api): update search method documentation +``` + +#### Submitting PR + +1. Push to your fork +2. Create PR against `main` branch +3. Fill out PR template +4. Ensure CI checks pass +5. Wait for review + +### Testing + +#### Running Tests + +```bash +# Run all tests +npm test + +# Run specific test file +npm test tests/core.test.ts + +# Run with coverage +npm run test:coverage + +# Watch mode +npm run test:watch +``` + +#### Writing Tests + +```typescript +import { describe, it, expect } from 'vitest' +import { Brainy } from '../src' + +describe('Feature Name', () => { + it('should do something specific', async () => { + const brain = new Brainy() + await brain.init() + + // Test implementation + const result = await brain.search("test") + + expect(result).toBeDefined() + expect(result.length).toBeGreaterThan(0) + }) +}) +``` + +## Architecture Guidelines + +### Adding New Features + +1. **Check existing functionality** + - Review `ARCHITECTURE.md` + - Check if similar features exist + - Consider if it should be an augmentation + +2. **Design considerations** + - Maintain backward compatibility + - Consider performance impact + - Think about all storage adapters + - Plan for extensibility + +3. **Implementation checklist** + - [ ] Core functionality + - [ ] Tests (unit and integration) + - [ ] Documentation + - [ ] TypeScript types + - [ ] Examples + - [ ] Performance benchmarks (if applicable) + +### Creating Augmentations + +Augmentations extend Brainy's functionality: + +```typescript +import { BrainyAugmentation } from '../types' + +export class MyAugmentation extends BrainyAugmentation { + name = 'MyAugmentation' + + async onInit(brain: Brainy): Promise { + // Initialize augmentation + } + + async onAdd(item: any, brain: Brainy): Promise { + // Process before adding + return item + } + + async onSearch(query: any, results: any[], brain: Brainy): Promise { + // Process search results + return results + } +} +``` + +### Performance Considerations + +- Use batch operations where possible +- Implement caching strategically +- Consider memory usage +- Profile performance impacts +- Add benchmarks for critical paths + +## Documentation + +### API Documentation + +Use JSDoc for all public APIs: + +```typescript +/** + * Searches for similar items using vector similarity + * @param query - Search query (text or vector) + * @param options - Search options + * @returns Array of search results with scores + * @example + * ```typescript + * const results = await brain.search("machine learning", { limit: 10 }) + * ``` + */ +async search(query: string | Vector, options?: SearchOptions): Promise { + // Implementation +} +``` + +### Examples + +Add examples for new features: + +```typescript +// examples/feature-name.ts +import { Brainy } from 'brainy' + +async function exampleUsage() { + const brain = new Brainy() + await brain.init() + + // Show feature usage + // Include comments explaining what's happening + // Handle errors appropriately +} + +exampleUsage().catch(console.error) +``` + +## Release Process + +1. **Version bump**: Follow semantic versioning +2. **Update CHANGELOG**: Document all changes +3. **Run tests**: Ensure all tests pass +4. **Build**: Generate distribution files +5. **Tag**: Create git tag for version +6. **Publish**: Release to npm + +## Getting Help + +- **Discord**: Join our community +- **Issues**: Ask questions on GitHub +- **Discussions**: Share ideas and get feedback + +## Recognition + +Contributors will be recognized in: +- CHANGELOG.md for their contributions +- README.md contributors section +- GitHub contributors page + +Thank you for contributing to Brainy! 🧠 \ No newline at end of file diff --git a/README.md b/README.md index 2caf6493..d1342f52 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@

npm version npm downloads - CI + CI Documentation MIT License TypeScript @@ -23,9 +23,8 @@ Quick start · One query · Features · - Scale with Cor · - Docs · - Support + Scale with Cor · + Docs

--- @@ -173,11 +172,9 @@ await brain.vfs.search('React components with hooks') // semantic file **[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)** -## When you outgrow Brainy +## From laptop to hundreds of millions -Brainy's pure-TypeScript engines carry real workloads a long way on their own — see the measured, per-operation numbers (not marketing figures) in **[docs/performance-envelopes.md](docs/performance-envelopes.md)** for what to expect, unaccelerated, on plain filesystem storage. - -When a deployment needs native-scale vector/graph performance — memory-mapped indexes that don't need your dataset in RAM, billion-scale ambitions — add the native engine. **The API doesn't change:** +Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**: ```bash npm install @soulcraft/cor @@ -190,14 +187,13 @@ await brain.init() // @soulcraft/cor detected — same code, native engines un Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate. -Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor is more headroom for when you need it, not capability held back to sell you later. Licensing and support: **cor@soulcraft.com**. +Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both. ## Performance -- Per-operation p50/p95 at 1k and 10k entities, pure-JS floor, measured and re-run every release that touches a measured path: **[docs/performance-envelopes.md](docs/performance-envelopes.md)**. - JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41). - Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan. -- Capacity planning and architecture: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)** +- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)** ## Use cases @@ -216,10 +212,6 @@ Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor **Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use. -## Support & community +## Contributing & license -- **Bugs and ideas** → **brainy@soulcraft.com** — no account needed, you'll get a receipt. -- **Security reports** → **security@soulcraft.com** — see **[SECURITY.md](SECURITY.md)**. -- **Contributing** → see **[CONTRIBUTING.md](CONTRIBUTING.md)**. - -MIT © Brainy Contributors. +Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors. diff --git a/RELEASES.md b/RELEASES.md index 89bf38e7..7799c6f6 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -31,72 +31,6 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the --- -## Unreleased (the warm contract: cold-restart writes stop paying demand-load latency) - -From a production deployment's cold-restart incident: the FIRST writes after every -restart on a large brain measured 33–35s each (page cache cold) against the transact -apply budget — Brainy's own op-count-scaled budget, `max(30s, opCount × 2s)`, e.g. -32,000ms for a 16-op batch. There is no external deadline in this story, and no -post-completion veto either: every write is itself a multi-operation transaction (a -single `add()` applies several operations — canonical writes, the vector-index insert, -the metadata-index update), so ONE cold operation that legitimately runs ~33s consumes -the whole budget, the gate before the NEXT operation trips, and the write rolls back -atomically (zero loss, by design) — refused, retried, refused again, until the page -cache warms passively (~30 minutes). The cure is not weaker atomicity; it is the two -new knobs below — a budget floor sized for cold stores, and a warm contract that pays -demand-load cost OFF the transaction path. - -- **The budget's start-gating contract is now explicit, documented, and pinned by - regression tests.** The budget gates STARTING the next operation — completed work is - never rolled back for elapsed time — and a transaction's first operation now - unconditionally starts by code, not merely because elapsed time happens to be ~0 when - it is checked. This has been the shipped schedule since 8.7.0 (no behavior change for - existing integrations); it is now stated in `Transaction.execute()`'s contract JSDoc - and enforced by tests so it cannot silently regress. Mid-batch atomicity is unchanged: - a trip before operation `i+1` still rolls back `0..i` and throws a retryable - `TransactionTimeoutError`. -- **The budget's 30s floor is now configurable**: `new Brainy({ transactionBudgetFloorMs })` - raises (or lowers) the floor of `max(transactionBudgetFloorMs, opCount × 2000)` for every - internal transact batch. Useful for a store whose cold writes legitimately run past 30s - per operation, so a bulk batch gets a proportionally larger runway instead of tripping - mid-batch on cold-cache latency. -- **New: `brain.warm()`** — eagerly loads/faults-in the vector index, metadata index, and - graph adjacency so the first real operation after a cold restart runs at steady-state - cost instead of paying demand-load latency on the critical path. Returns a `WarmReport` - with one honest outcome per surface — never conflate the first two: - - `'warmed'` — the surface's own provider `warm()` hook ran, or a full-hydration seam - loaded every shard/field/segment from storage. Steady-state cost is paid. - - `'probed'` — no `warm()` hook was available, so a best-effort read (one `search()` call - for the vector index) faulted in *some* backing storage as a side effect — real work, - but never reported as `'warmed'`. - - `'unavailable'` — nothing ran (no hook, no hydration seam, or nothing to probe). -- **New config: `warmOnOpen: true`** makes `init()` await `brain.warm()` before it resolves - — a deliberate blocking trade-off: startup takes longer, the first request doesn't. - Default `false` (unchanged lazy behavior). -- **New optional provider hook: `warm?(): Promise`** on the vector and graph - acceleration provider contracts (`src/plugin.ts`) — a native provider can implement it to - eagerly pretouch its own backing storage (e.g. mmap pretouch); absence means brainy falls - back to the probe/hydration behavior above. -- **Transaction op-name strings changed in journals/timings**: the vector-index - transaction operations were renamed from `AddToHNSW`/`RemoveFromHNSW` to backend-neutral - `AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an - algorithm that may not be the one actually running (a non-HNSW native vector provider - emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist). - The parenthesized suffix names the ACTIVE backend: `hnsw-js` for the built-in engine, or - the native provider's own identity when it self-identifies. **If you parse these op-name - strings** (log processors, journal tooling), update your matcher from - `AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`. -- **Provider identity is now a REQUIRED `name` field** on the vector provider contract - (`VectorIndexProvider.name`, `src/plugin.ts`) — every implementation self-reports its own - identity truthfully (its algorithm/engine), never inheriting a default. It renders as the - op-name suffix above and, wherever the vector index identifies itself in prose log lines, - as the tag `[vector-index:]`. **Native provider adoption is a one-line change**: - declare `readonly name = ''`. A provider instance that still lacks - `name` at runtime (an older native build compiled against the previous, optional field) is - never crashed on and never silently mislabeled: it stamps `unknown-provider` and emits one - loud warning naming the missing field, so the gap is discoverable instead of a permanent - fossil label in every journal line. - ## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close()) The write path stops paying maintenance costs — the last structural piece of the diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index 1f3c4732..00000000 --- a/SECURITY.md +++ /dev/null @@ -1,36 +0,0 @@ -# Security Policy - -## Reporting a vulnerability - -Email **security@soulcraft.com**. That's the one door for security reports -across the company, and it works the same way for Brainy: every report is -read by a human, you'll get a private receipt, and we'll work with you on -coordinated disclosure — please don't open a public issue for anything -that isn't already public. - -Include what you'd want if you were on the other end: affected version, -how to reproduce, and what you think the impact is. If you have a patch or -a suggested fix, send it along — it's welcome but not required. - -There is no bounty program today. We're saying that plainly so you know -what to expect going in. - -## Response time - -We respond as fast as truth allows. That means: no fixed SLA, no promise of -a reply within a specific number of hours — but a real report from a real -person gets read promptly and taken seriously. If you haven't heard anything -in a reasonable stretch, a follow-up email is completely fine. - -## Supported versions - -The latest `8.x` minor release line receives security fixes. If you're -running an older major version, please upgrade before reporting — we can't -commit to backporting fixes to unsupported lines. - -## Scope - -This policy covers the `@soulcraft/brainy` package itself — the code in -this repository. If you're evaluating a deployment that also uses -`@soulcraft/cor`, report issues in that package the same way, to the same -address; we'll route internally. diff --git a/package-lock.json b/package-lock.json index 37aeb81d..fb9262e9 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.10.0", + "version": "8.9.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.10.0", + "version": "8.9.0", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index a3ece83c..7366ce98 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.10.0", + "version": "8.9.0", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", @@ -128,13 +128,13 @@ "publishConfig": { "access": "public" }, - "homepage": "https://source.soulcraft.com/soulcraft/brainy", + "homepage": "https://github.com/soulcraftlabs/brainy", "bugs": { - "url": "https://source.soulcraft.com/soulcraft/brainy/issues" + "url": "https://github.com/soulcraftlabs/brainy/issues" }, "repository": { "type": "git", - "url": "git+https://source.soulcraft.com/soulcraft/brainy.git" + "url": "git+https://github.com/soulcraftlabs/brainy.git" }, "files": [ "dist/**/*.js", diff --git a/scripts/release.sh b/scripts/release.sh index 43fa50bd..7d860564 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -142,7 +142,7 @@ else fi # Create new changelog entry -CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraft/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d)) +CHANGELOG_ENTRY="### [${NEW_VERSION}](https://github.com/soulcraftlabs/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d)) ${COMMITS} " @@ -175,59 +175,26 @@ echo -e "${BLUE}7️⃣ Creating git tag v${NEW_VERSION}...${NC}" git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}" echo -e "${GREEN}✅ Tag created${NC}\n" -# Step 9: Push to origin — the forge is the one home (ruled 2026-07-23; the -# old public GitHub repo is archived history, no longer part of any release). -echo -e "${BLUE}8️⃣ Pushing to origin...${NC}" +# Step 9: Push to GitHub +echo -e "${BLUE}8️⃣ Pushing to GitHub...${NC}" git push --follow-tags origin "$CURRENT_BRANCH" -echo -e "${GREEN}✅ Pushed to origin${NC}\n" +echo -e "${GREEN}✅ Pushed to GitHub${NC}\n" -# Step 10: Publish — forge FIRST (home), npmjs second (the world's storefront). -# The fleet-wide ~/.npmrc maps the @soulcraft scope to the forge registry, and -# a scope mapping BEATS `--registry` on the command line — so each publish -# names its registry via the scope override explicitly. Nothing implicit. -FORGE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraft/npm/" -FORGE_NPM_TOKEN_FILE="$HOME/.config/soulcraft/npm-publish-brainy.token" -echo -e "${BLUE}9️⃣ Publishing to the forge registry (home)...${NC}" -if [ -f "$FORGE_NPM_TOKEN_FILE" ]; then - TMPRC="$(mktemp)" - chmod 600 "$TMPRC" - { - echo "@soulcraft:registry=${FORGE_NPM_REG}" - echo "//source.soulcraft.com/api/packages/soulcraft/npm/:_authToken=$(cat "$FORGE_NPM_TOKEN_FILE")" - } > "$TMPRC" - if npm publish --tag "$NPM_TAG" --userconfig "$TMPRC"; then - echo -e "${GREEN}✅ Published to the forge${NC}\n" - else - rm -f "$TMPRC" - echo -e "${RED}❌ Forge publish FAILED — aborting before npmjs so the pair never diverges. Fix and re-run.${NC}" - exit 1 - fi - rm -f "$TMPRC" -else - echo -e "${RED}❌ Forge publish token missing (${FORGE_NPM_TOKEN_FILE}) — aborting. The forge is home; publish it first or restage the token.${NC}" - exit 1 -fi - -echo -e "${BLUE}9️⃣½ Publishing to npmjs (storefront, dist-tag: ${NPM_TAG})...${NC}" -npm publish --tag "$NPM_TAG" "--@soulcraft:registry=https://registry.npmjs.org/" +# Step 10: Publish to npm +echo -e "${BLUE}9️⃣ Publishing to npm (dist-tag: ${NPM_TAG})...${NC}" +npm publish --tag "$NPM_TAG" # Brainy is the only PUBLIC @soulcraft package — verify visibility after every publish. -npm access get status @soulcraft/brainy "--@soulcraft:registry=https://registry.npmjs.org/" || true -echo -e "${GREEN}✅ Published to npmjs${NC}\n" +npm access get status @soulcraft/brainy || true +echo -e "${GREEN}✅ Published to npm${NC}\n" -# Step 11: Release object on the forge (presentational — the tag, CHANGELOG, -# and RELEASES.md are the record; this just gives the forge UI a release page). -echo -e "${BLUE}🔟 Creating forge release...${NC}" -if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then - if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \ - -H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \ - -d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then - echo -e "${GREEN}✅ Forge release created${NC}\n" - else - echo -e "${RED}⚠️ Forge release API call failed — tag + CHANGELOG remain the record; create the release page via the forge UI if wanted${NC}\n" - fi +# Step 11: Create GitHub release +echo -e "${BLUE}🔟 Creating GitHub release...${NC}" +if [ "$PRERELEASE" = true ]; then + gh release create "v${NEW_VERSION}" --generate-notes --prerelease else - echo -e "${RED}⚠️ FORGEJO_RELEASE_TOKEN unset — no release page created; tag + CHANGELOG remain the record${NC}\n" + gh release create "v${NEW_VERSION}" --generate-notes fi +echo -e "${GREEN}✅ GitHub release created${NC}\n" # Step 12: Push public docs to the soulcraft.com docs ingest door # (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when @@ -246,4 +213,4 @@ echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}" echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" echo "" echo -e "📦 npm: ${BLUE}https://www.npmjs.com/package/@soulcraft/brainy/v/${NEW_VERSION}${NC}" -echo -e "🏠 Forge: ${BLUE}https://source.soulcraft.com/soulcraft/brainy/releases/tag/v${NEW_VERSION}${NC}" +echo -e "🐙 GitHub: ${BLUE}https://github.com/soulcraftlabs/brainy/releases/tag/v${NEW_VERSION}${NC}" diff --git a/src/brainy.ts b/src/brainy.ts index fd9e1ffa..8ba991dd 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -63,9 +63,7 @@ import type { PathOptions, MetadataIndexProvider, OpaqueIdSet, - AtGenerationVectors, - VectorIndexProvider, - GraphIndexProvider + AtGenerationVectors } from './plugin.js' import type { BrainyPlugin, @@ -90,12 +88,12 @@ import { findCallerLocation } from './utils/callerLocation.js' import { SaveNounMetadataOperation, SaveNounOperation, - AddToVectorIndexOperation, + AddToHNSWOperation, AddToMetadataIndexOperation, SaveVerbMetadataOperation, SaveVerbOperation, AddToGraphIndexOperation, - RemoveFromVectorIndexOperation, + RemoveFromHNSWOperation, RemoveFromMetadataIndexOperation, RemoveFromGraphIndexOperation, UpdateNounMetadataOperation, @@ -268,7 +266,6 @@ type ResolvedBrainyConfig = Required< | 'retention' | 'eagerEmbeddings' | 'migrationWaitTimeoutMs' - | 'transactionBudgetFloorMs' > > & Pick< @@ -281,7 +278,6 @@ type ResolvedBrainyConfig = Required< | 'retention' | 'eagerEmbeddings' | 'migrationWaitTimeoutMs' - | 'transactionBudgetFloorMs' > /** @@ -392,38 +388,6 @@ class InsertPreconditionExistsSignal extends Error { */ export type IndexFamily = 'vector' | 'metadata' | 'graph' -/** - * @description Honest per-surface outcome for {@link Brainy.warm}. Literal - * meanings — never conflate the first two: - * - `'warmed'` — the provider's own `warm?()` hook ran (vector/graph), or the - * surface's full-hydration seam loaded EVERY shard/field/segment from - * storage (metadata; graph's fallback path). The surface is genuinely at - * steady-state cost for the next operation. - * - `'probed'` — no `warm?()` hook was available, so a best-effort read - * (e.g. one `search()` call) faulted in *some* backing storage as a side - * effect. Real work happened, but it is NOT the same guarantee as - * `'warmed'` — never reported as `'warmed'`. - * - `'unavailable'` — nothing ran: no hook, no hydration seam, and (for the - * vector probe fallback) nothing to probe (an empty index or unknown - * vector dimension). The surface is unchanged by this `warm()` call. - */ -export type WarmOutcome = 'warmed' | 'probed' | 'unavailable' - -/** - * @description Result of {@link Brainy.warm}: one {@link WarmOutcome} + - * elapsed time per index surface, plus the total wall-clock time for the - * whole call. `durationMs` is measured around exactly the work described by - * that surface's `outcome` (e.g. the vector entry's `durationMs` times the - * provider `warm()` call OR the probe `search()` call — whichever ran). - */ -export interface WarmReport { - vector: { outcome: WarmOutcome; durationMs: number } - metadata: { outcome: WarmOutcome; durationMs: number } - graph: { outcome: WarmOutcome; durationMs: number } - /** Total wall-clock time for the whole `warm()` call (all three surfaces). */ - totalDurationMs: number -} - /** * How long a failed aggregation-backfill walk suppresses fresh walk attempts. * Within the window, queries rethrow the recorded failure instantly (loud, @@ -1418,17 +1382,6 @@ export class Brainy implements BrainyInterface { }) } - // Eager index warm (operator opt-in, `warmOnOpen: true`). Runs AFTER - // every step above — index construction, crash recovery, migrations, - // VFS bootstrap — never in place of any of it, so `warm()` always - // operates on a fully-initialized brain. Blocking BY DESIGN: the - // operator traded a longer startup for a first-request that runs at - // steady-state cost instead of paying demand-load latency on the - // critical path. See the `warmOnOpen` JSDoc in brainy.types.ts. - if (this.config.warmOnOpen) { - await this.warm() - } - // Resolve ready Promise - consumers awaiting brain.ready will now proceed if (this._readyResolve) { this._readyResolve() @@ -1827,9 +1780,7 @@ export class Brainy implements BrainyInterface { await this.generationStore.runWithoutGeneration(() => this.transactionManager.executeTransaction(run, { timeout: transactTimeoutBudget( - (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0), - undefined, - this.config.transactionBudgetFloorMs + (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0) ) }) ) @@ -1846,9 +1797,7 @@ export class Brainy implements BrainyInterface { execute: () => this.transactionManager.executeTransaction(run, { timeout: transactTimeoutBudget( - (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0), - undefined, - this.config.transactionBudgetFloorMs + (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0) ) }) }) @@ -2160,7 +2109,7 @@ export class Brainy implements BrainyInterface { // Operation 3: Add to HNSW index (after entity saved) tx.addOperation( - new AddToVectorIndexOperation(this.index, id, vector) + new AddToHNSWOperation(this.index, id, vector) ) // Operation 4: Add to metadata index @@ -3149,10 +3098,10 @@ export class Brainy implements BrainyInterface { // Operation 3-4: Update HNSW index (remove and re-add if reindexing needed) if (needsReindexing) { tx.addOperation( - new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector) + new RemoveFromHNSWOperation(this.index, params.id, existing.vector) ) tx.addOperation( - new AddToVectorIndexOperation(this.index, params.id, vector) + new AddToHNSWOperation(this.index, params.id, vector) ) } @@ -3268,7 +3217,7 @@ export class Brainy implements BrainyInterface { // Operation 1: Remove from vector index if (noun) { tx.addOperation( - new RemoveFromVectorIndexOperation(this.index, id, noun.vector) + new RemoveFromHNSWOperation(this.index, id, noun.vector) ) } @@ -7076,7 +7025,7 @@ export class Brainy implements BrainyInterface { // Add delete operations to transaction if (noun) { tx.addOperation( - new RemoveFromVectorIndexOperation(this.index, id, noun.vector) + new RemoveFromHNSWOperation(this.index, id, noun.vector) ) } @@ -7939,13 +7888,7 @@ export class Brainy implements BrainyInterface { // Budget scales with the batch (or the caller's explicit // timeoutMs): a flat 30s cap silently limited honest bulk work // to ~15 ops on network disks (~2s/op measured in the field). - { - timeout: transactTimeoutBudget( - plan.operations.length, - options?.timeoutMs, - this.config.transactionBudgetFloorMs - ) - } + { timeout: transactTimeoutBudget(plan.operations.length, options?.timeoutMs) } ) } })) @@ -8322,27 +8265,6 @@ export class Brainy implements BrainyInterface { return this.generationStore.compact(options) } - /** - * @description Repack cold generation history into sealed segments — - * re-representation, never deletion: every record and delta stays readable - * (`asOf()` unchanged); the physical file count drops by orders of - * magnitude. Runs automatically (time-bounded) at `close()`; call this for - * explicit maintenance windows on long-lived writers. The ONLY history - * transform permitted under the archival profile (`retention: 'all'`). - * @param options - `timeBudgetMs` bounds the pass (early stop = consistent - * prefix, next pass resumes); `batchGenerations` sizes each fold. - * @returns Folded generation count and segments created. - */ - async repackHistory(options?: { - timeBudgetMs?: number - batchGenerations?: number - }): Promise<{ foldedGenerations: number; segmentsCreated: number }> { - this.assertWritable('repackHistory') - await this.ensureInitialized() - await this.generationStore.flushPendingSingleOps() - return this.generationStore.repackHistory(options) - } - /** * @description Read-only generational-history footprint for fleet audits: * generation count, total on-disk bytes, generation/timestamp range, the @@ -8370,24 +8292,6 @@ export class Brainy implements BrainyInterface { } } - /** - * @description A deterministic content digest of the generation log through - * `g` (D8 — gate-to-generation provenance): identical history produces the - * identical digest on any machine; divergence produces a different one. - * Release gates and suite verdicts pin `{generation, digest}` and verify - * both at execution time instead of pinning a git commit. O(segments + - * live-tier window), never O(all generations). Throws `RangeError` out of - * range and `GenerationCompactedError` below the horizon — a gate can - * never silently pin reclaimed history. - * @example - * const gate = { generation: brain.generation(), digest: await brain.generationDigest(brain.generation()) } - */ - async generationDigest(g: number): Promise { - await this.ensureInitialized() - await this.generationStore.flushPendingSingleOps() - return this.generationStore.generationDigest(g) - } - /** * @description Drive the adaptive retention byte budget at runtime — the * settable input a machine-level coordinator (e.g. cor's `ResourceManager`, @@ -9368,7 +9272,7 @@ export class Brainy implements BrainyInterface { plan.operations.push( new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew), new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew), - new AddToVectorIndexOperation(this.index, id, vector), + new AddToHNSWOperation(this.index, id, vector), new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing) ) plan.touchedNouns.push(id) @@ -9536,8 +9440,8 @@ export class Brainy implements BrainyInterface { ) if (needsReindexing) { plan.operations.push( - new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector), - new AddToVectorIndexOperation(this.index, params.id, vector) + new RemoveFromHNSWOperation(this.index, params.id, existing.vector), + new AddToHNSWOperation(this.index, params.id, vector) ) } plan.operations.push( @@ -9622,7 +9526,7 @@ export class Brainy implements BrainyInterface { } if (noun) { - plan.operations.push(new RemoveFromVectorIndexOperation(this.index, id, noun.vector)) + plan.operations.push(new RemoveFromHNSWOperation(this.index, id, noun.vector)) } if (metadata) { plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata)) @@ -14337,118 +14241,6 @@ export class Brainy implements BrainyInterface { return this.embedder(textToEmbed) } - /** - * Eagerly load/fault-in the vector index, metadata index, and graph - * adjacency so the FIRST real operation after this call runs at - * steady-state cost — no page-cache-miss / demand-load latency on the - * critical path. Complements {@link warmupEmbeddings} (which warms the - * embedding engine, not the storage indexes). - * - * Sequence per surface: - * - **Vector**: calls the provider's own `warm?()` when the active - * `'vector'` provider implements it (`'warmed'`). Otherwise runs a - * best-effort probe — one `search()` with a deterministic unit vector of - * the index's known dimension, `k = min(100, size())` — which faults in - * *some* backing storage as a side effect but is reported honestly as - * `'probed'`, never `'warmed'`. An empty index or unknown dimension has - * nothing to probe (`'unavailable'`). - * - **Metadata**: full hydration — every persisted field's sparse index is - * loaded from storage (`MetadataIndexManager.hydrateAll()`), not just the - * heuristic common-fields subset `init()` warms. - * - **Graph**: calls the provider's own `warm?()` when the active graph - * provider implements it; otherwise re-runs its existing eager cold-load - * `init()` seam (idempotent — the JS adjacency index's `init()` already - * loads the full LSM manifests + SSTables, so re-invoking it here is a - * genuine full hydration, not a stat call). - * - * A single surface's `'unavailable'`/probe outcome never fails the whole - * call — `warm()` is a best-effort readiness step, not a correctness gate; - * queries still demand-load normally regardless of what `warm()` achieved. - * - * @returns A {@link WarmReport}: honest per-surface outcome + timing. - * @example - * ```typescript - * const brain = new Brainy({ storage: { path: '/data' } }) - * await brain.init() - * const report = await brain.warm() // blocking, explicit control over timing - * console.log(report.vector.outcome, report.vector.durationMs) - * - * // Or opt into the same work automatically during init(): - * const eager = new Brainy({ storage: { path: '/data' }, warmOnOpen: true }) - * await eager.init() // warm() already ran before this resolves - * ``` - */ - async warm(): Promise { - await this.ensureInitialized({ needs: ['vector', 'metadata', 'graph'] }) - - const totalStart = Date.now() - - // --- Vector --------------------------------------------------------- - const vectorStart = Date.now() - let vectorOutcome: WarmOutcome - const vectorProvider = this.index as VectorIndexProvider & { warm?: () => Promise } - if (typeof vectorProvider.warm === 'function') { - await vectorProvider.warm() - vectorOutcome = 'warmed' - } else { - const size = this.index.size() - const dimension = this.dimensions - if (size > 0 && dimension && dimension > 0) { - // A deterministic unit vector (L2 norm 1) — same probe every call, - // no reliance on stored data shape. - const probeVector = new Array(dimension).fill(1 / Math.sqrt(dimension)) - await this.index.search(probeVector, Math.min(100, size)) - vectorOutcome = 'probed' - } else { - // Nothing to probe: an empty index, or the vector dimension is not - // yet known (no entity has ever been added). - vectorOutcome = 'unavailable' - } - } - const vectorDurationMs = Date.now() - vectorStart - - // --- Metadata -------------------------------------------------------- - const metadataStart = Date.now() - let metadataOutcome: WarmOutcome - const metadataWithHydrate = this.metadataIndex as unknown as { hydrateAll?: () => Promise } - if (typeof metadataWithHydrate.hydrateAll === 'function') { - await metadataWithHydrate.hydrateAll() - metadataOutcome = 'warmed' - } else { - // No hydration seam on this metadata provider — nothing to run. - metadataOutcome = 'unavailable' - } - const metadataDurationMs = Date.now() - metadataStart - - // --- Graph ------------------------------------------------------------- - const graphStart = Date.now() - let graphOutcome: WarmOutcome - const graphProvider = this.graphIndex as GraphIndexProvider & { - warm?: () => Promise - init?: () => Promise - } - if (typeof graphProvider.warm === 'function') { - await graphProvider.warm() - graphOutcome = 'warmed' - } else if (typeof graphProvider.init === 'function') { - // Existing eager cold-load seam (readiness contract): idempotent, and - // for the JS adjacency index it's already a genuine full load of the - // LSM manifests + SSTables — a real hydration, not a stat call. - await graphProvider.init() - graphOutcome = 'warmed' - } else { - graphOutcome = 'unavailable' - } - const graphDurationMs = Date.now() - graphStart - - return { - vector: { outcome: vectorOutcome, durationMs: vectorDurationMs }, - metadata: { outcome: metadataOutcome, durationMs: metadataDurationMs }, - graph: { outcome: graphOutcome, durationMs: graphDurationMs }, - totalDurationMs: Date.now() - totalStart - } - } - /** * Explicitly warm up the embedding engine * @@ -14967,9 +14759,6 @@ export class Brainy implements BrainyInterface { // Migration LOCK wait budget — left undefined when omitted so // awaitMigrationLock() applies its 30 s default at the read site. migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined, - // Apply-phase transaction budget floor — left undefined when omitted so - // transactTimeoutBudget() applies its 30 s default at the read site. - transactionBudgetFloorMs: config?.transactionBudgetFloorMs ?? undefined, // Pre-upgrade backup — default-on; opt out with `migrationBackup: false`. migrationBackup: config?.migrationBackup ?? true, // Vector index configuration (8.0) — algorithm-neutral surface with @@ -14985,9 +14774,6 @@ export class Brainy implements BrainyInterface { // is the active one on a non-reader writer outside unit tests). Explicit // true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts. eagerEmbeddings: config?.eagerEmbeddings ?? undefined, - // Eager index warm at init() — default off (lazy demand-load), the - // pre-8.9 behavior. See the warmOnOpen JSDoc in brainy.types.ts. - warmOnOpen: config?.warmOnOpen ?? false, // Plugin configuration - undefined = auto-detect plugins: config?.plugins ?? undefined, // Integration Hub - undefined/false = disabled @@ -16406,21 +16192,11 @@ export class Brainy implements BrainyInterface { await this.generationStore.flushPendingSingleOps() } - // Phase 0b: REPACK cold history into sealed segments (D1+D3 — - // re-representation, never deletion; the only history transform under the - // archival profile), then auto-compact per config.retention. Repack runs - // FIRST so bounded-retention reclaim can drop whole segments. Both are - // time-bounded maintenance passes (8.9.0 law: flush() never pays these); - // both are housekeeping — failures warn, never fail a clean shutdown. - if (!this.isReadOnly && this.generationStore) { - try { - await this.generationStore.repackHistory({ timeBudgetMs: 5_000 }) - } catch (error) { - console.warn( - `History repacking failed (non-fatal): ${error instanceof Error ? error.message : String(error)}` - ) - } - } + // Phase 0b: Auto-compact generational history per config.retention (default + // on) BEFORE the generation store closes below. This is THE auto-compaction + // site (8.9.0 — flush() never compacts): time-bounded per pass, respects + // live Db pins and an explicit autoCompact: false; no-op on read-only + // instances. await this.autoCompactHistory() // Phase 1: Flush ALL components in parallel to persist buffered data diff --git a/src/db/factLog.ts b/src/db/factLog.ts index 94e79700..4c5e95fd 100644 --- a/src/db/factLog.ts +++ b/src/db/factLog.ts @@ -102,26 +102,12 @@ export interface FactScanBatch { segmentId: string } -/** - * Liveness bound on a scan's FIRST batch (Stage-2 co-freeze, D1 contract): - * `batches()` must yield its first batch — or fail loudly — within this many - * ms of the first pull. A backlogged or damaged store may be SLOW, but it may - * never be SILENT: a consumer awaiting the first batch is otherwise - * indistinguishable from a wedge (the exact failure shape a production heal - * hit against a generations-backlogged brain). - */ -export const SCANFACTS_FIRST_BATCH_MS = 10_000 - /** The telemetry a scan OPEN returns (frozen shape). */ export interface FactScanHandle { headGeneration: number segmentCount: number approxFactCount: number - /** - * Ordered batches; a detected gap aborts LOUDLY, never a silent skip. - * Liveness contract: the FIRST batch resolves or rejects within - * {@link SCANFACTS_FIRST_BATCH_MS} of the first pull — never a silent hang. - */ + /** Ordered batches; a detected gap aborts LOUDLY, never a silent skip. */ batches: () => AsyncGenerator /** Close telemetry — the invariant cross-check, valid after iteration ends. */ summary: () => { factsYielded: number; segmentsRead: number } @@ -454,8 +440,6 @@ export class FactLog { toGeneration?: number kinds?: Array<'noun' | 'verb'> batchSize?: number - /** Test override for the first-batch liveness bound (default {@link SCANFACTS_FIRST_BATCH_MS}). */ - firstBatchTimeoutMs?: number }): FactScanHandle { const from = options?.fromGeneration ?? 1 const to = options?.toGeneration ?? this.head @@ -530,43 +514,11 @@ export class FactLog { } } - // Liveness wrapper: the FIRST pull races the contract deadline. Only the - // first — the bound is time-to-first-batch (proof the producer is alive), - // not per-batch pacing; and it runs only while a pull is actually pending, - // so consumer think-time between pulls never counts against the producer. - const firstBatchTimeoutMs = options?.firstBatchTimeoutMs ?? SCANFACTS_FIRST_BATCH_MS - async function* batchesWithLiveness(this: void): AsyncGenerator { - const inner = batches() - let timer: NodeJS.Timeout | undefined - try { - const deadline = new Promise((_, reject) => { - timer = setTimeout( - () => - reject( - new Error( - `fact log: scanFacts produced no first batch within ${firstBatchTimeoutMs}ms ` + - `(liveness contract) — the store is wedged or unreadably slow; aborting scan LOUDLY ` + - `instead of hanging the consumer.` - ) - ), - firstBatchTimeoutMs - ) - timer.unref?.() - }) - const first = await Promise.race([inner.next(), deadline]) - if (first.done) return - yield first.value - } finally { - clearTimeout(timer) - } - yield* inner - } - return { headGeneration: this.head, segmentCount: segments.length + (tailSnapshot.length > 0 ? 1 : 0), approxFactCount, - batches: batchesWithLiveness, + batches, summary: () => ({ factsYielded, segmentsRead }) } } diff --git a/src/db/generationSegments.ts b/src/db/generationSegments.ts deleted file mode 100644 index 0c14b60c..00000000 --- a/src/db/generationSegments.ts +++ /dev/null @@ -1,459 +0,0 @@ -/** - * @module db/generationSegments - * @description The generation-segment store — Stage-2 D1+D3+repacking's file - * format (co-frozen 2026-07-19; design: the d1-d3-repacking spec). - * - * Packs CONSECUTIVE cold generations' record-sets (before-images + delta) - * into append-once segment files with derived sidecar indexes, so history - * scales in SEGMENTS (tens) instead of FILES-PER-GENERATION (hundreds of - * thousands), and cold-open reads ONE manifest instead of listing the - * backlog. Layout under `_generations/segments/`: - * - * - `seg-.bgs` — magic "BGS1", then one frame per - * generation: `u32 payloadLen | u32 crc32c | msgpack payload`. Payload is - * POSITIONAL: `[generation, timestamp, delta, records[], flags]` with - * records `[kindByte, id, record]`. `flags` reserves encoding evolution - * (bit 0 = compressed payload — v1 always 0; a future writer upgrade, - * never a format break). Sealed segments are IMMUTABLE — the fact log's - * own law, generalized. - * - `seg-.idx` — DERIVED sidecar (msgpack): per-generation frame - * offsets (point reads = one ranged read, never a listing) + per-id - * generation postings (per-id chain rebuilds read only what they need). - * Corrupt/missing → rebuilt from its segment in one sequential read, - * loudly. - * - `manifest.json` — the segment catalogue + `compactedBelow` (D3's - * horizon marker). Cold-open reads THIS; the packed backlog is never - * listed. - * - * D3 semantics carried here: bounded-retention reclaim drops WHOLE segments - * at boundaries (O(1) per segment, no rewrite); under the archival profile - * (`retention: 'all'`) nothing here is ever dropped — folding is the only - * transform (re-representation, never deletion). - */ - -import { encode as msgpackEncode, decode as msgpackDecode } from '@msgpack/msgpack' -import { crc32c } from '../utils/crc32c.js' -import type { FactLogStorage } from './factLog.js' -import { prodLog } from '../utils/logger.js' - -/** Directory for segment files + manifest, under the generations prefix. */ -export const SEGMENTS_PREFIX = '_generations/segments' - -/** Target sealed-segment size (co-freeze proposal; tunable on evidence). */ -export const SEGMENT_TARGET_BYTES = 64 * 1024 * 1024 - -const MAGIC = new TextEncoder().encode('BGS1') -const FRAME_PREFIX_BYTES = 8 // u32 payloadLen + u32 crc32c -const MANIFEST_PATH = `${SEGMENTS_PREFIX}/manifest.json` - -/** One generation's fold input — exactly what the live tier holds for it. */ -export interface FoldGeneration { - generation: number - timestamp: number - /** The tx.json delta object, carried verbatim. */ - delta: unknown - /** The before-image record-set (empty for record-less generations). */ - records: Array<{ kind: 'noun' | 'verb'; id: string; record: unknown }> -} - -/** Manifest entry for one sealed segment. */ -export interface SegmentMeta { - file: string - firstGeneration: number - lastGeneration: number - frames: number - bytes: number - /** crc32c of the full segment byte stream — the digest chain's link. */ - checksum: number -} - -interface SegmentManifest { - version: 1 - compactedBelow: number - segments: SegmentMeta[] -} - -interface SidecarIndex { - version: 1 - /** [generation, frameOffset, frameLen] ascending by generation. */ - generations: Array<[number, number, number]> - /** `${kindByte}:${id}` → ascending generations holding a record for it. */ - ids: Record -} - -const segmentFileName = (firstGeneration: number): string => - `seg-${String(firstGeneration).padStart(20, '0')}.bgs` -const sidecarFileName = (firstGeneration: number): string => - `seg-${String(firstGeneration).padStart(20, '0')}.idx` - -/** - * The generation-segment store. Owns the packed tier ONLY — the live - * per-generation tier and the routing between tiers belong to - * `GenerationStore`. All mutating entry points here are called under the - * generation store's commit mutex. - */ -export class GenerationSegmentStore { - private readonly storage: FactLogStorage - private manifest: SegmentManifest = { version: 1, compactedBelow: 0, segments: [] } - /** Sidecar cache — segments are immutable, so entries never invalidate. */ - private readonly sidecars = new Map() - - constructor(storage: FactLogStorage) { - this.storage = storage - } - - /** Load the manifest (ONE read — never a directory listing). */ - async open(): Promise { - const raw = (await this.storage.readRawObject(MANIFEST_PATH)) as SegmentManifest | null - if (raw) { - if (raw.version !== 1) { - throw new Error( - `[GenerationSegments] manifest version ${String(raw.version)} is newer than this ` + - `engine understands — refusing to serve partial history. Upgrade the engine.` - ) - } - this.manifest = raw - } - } - - /** The packed tier's catalogue (ascending, immutable snapshot). */ - segments(): readonly SegmentMeta[] { - return this.manifest.segments - } - - /** D3's horizon marker: generations below this were reclaimed (bounded profiles only). */ - compactedBelow(): number { - return this.manifest.compactedBelow - } - - /** The covering sealed segment for `gen`, or null if it lives outside the packed tier. */ - private coveringSegment(gen: number): SegmentMeta | null { - // Manifest is ascending and ranges never overlap — binary search. - const segs = this.manifest.segments - let lo = 0 - let hi = segs.length - 1 - while (lo <= hi) { - const mid = (lo + hi) >> 1 - const s = segs[mid] - if (gen < s.firstGeneration) hi = mid - 1 - else if (gen > s.lastGeneration) lo = mid + 1 - else return s - } - return null - } - - /** True when `gen` is packed (readable from this tier). */ - hasGeneration(gen: number): boolean { - return this.coveringSegment(gen) !== null - } - - /** - * Fold consecutive generations into ONE new sealed segment + sidecar and - * append it to the manifest atomically. Caller guarantees: `gens` is - * ascending, contiguous with the packed tier (first = last packed + 1 when - * segments exist), and already durable in the live tier. Crash between the - * segment write and the caller's live-tier delete leaves a DUPLICATE - * representation — resolved live-tier-wins by the reader; never a gap. - */ - async fold(gens: FoldGeneration[]): Promise { - if (gens.length === 0) { - throw new Error('[GenerationSegments] fold() requires at least one generation') - } - for (let i = 1; i < gens.length; i++) { - if (gens[i].generation <= gens[i - 1].generation) { - throw new Error('[GenerationSegments] fold() input must be strictly ascending') - } - } - const last = this.manifest.segments[this.manifest.segments.length - 1] - if (last && gens[0].generation <= last.lastGeneration) { - throw new Error( - `[GenerationSegments] fold() overlaps the packed tier: ${gens[0].generation} ≤ ` + - `sealed ${last.lastGeneration} — segments are immutable, never rewritten` - ) - } - - const first = gens[0].generation - const file = segmentFileName(first) - const sidecar: SidecarIndex = { version: 1, generations: [], ids: {} } - - // Encode all frames, tracking offsets for the sidecar. - const parts: Uint8Array[] = [MAGIC] - let offset = MAGIC.length - for (const g of gens) { - const payload = msgpackEncode([ - g.generation, - g.timestamp, - g.delta, - g.records.map((r) => [r.kind === 'noun' ? 0 : 1, r.id, r.record]), - 0 // flags: v1 = uncompressed - ]) - const frame = new Uint8Array(FRAME_PREFIX_BYTES + payload.length) - const view = new DataView(frame.buffer) - view.setUint32(0, payload.length, true) - view.setUint32(4, crc32c(payload), true) - frame.set(payload, FRAME_PREFIX_BYTES) - sidecar.generations.push([g.generation, offset, frame.length]) - for (const r of g.records) { - const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}` - ;(sidecar.ids[key] ??= []).push(g.generation) - } - parts.push(frame) - offset += frame.length - } - const total = parts.reduce((n, p) => n + p.length, 0) - const bytes = new Uint8Array(total) - let at = 0 - for (const p of parts) { - bytes.set(p, at) - at += p.length - } - - const meta: SegmentMeta = { - file, - firstGeneration: first, - lastGeneration: gens[gens.length - 1].generation, - frames: gens.length, - bytes: total, - checksum: crc32c(bytes) - } - - // Durability order: segment + sidecar fsync'd BEFORE the manifest names - // them (a crash before the manifest = invisible orphan files, harmless); - // manifest last, atomically. - const segPath = `${SEGMENTS_PREFIX}/${file}` - const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(first)}` - await this.storage.writeRawBytes(segPath, bytes) - await this.storage.writeRawBytes(idxPath, msgpackEncode(sidecar)) - await this.storage.syncRawObjects([segPath, idxPath]) - const next: SegmentManifest = { - ...this.manifest, - segments: [...this.manifest.segments, meta] - } - await this.storage.writeRawObject(MANIFEST_PATH, next) - await this.storage.syncRawObjects([MANIFEST_PATH]) - this.manifest = next - this.sidecars.set(file, sidecar) - return meta - } - - /** Load (or rebuild, loudly) a segment's sidecar. */ - private async sidecarFor(meta: SegmentMeta): Promise { - const cached = this.sidecars.get(meta.file) - if (cached) return cached - const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(meta.firstGeneration)}` - const raw = await this.storage.readRawBytes(idxPath) - if (raw) { - try { - const idx = msgpackDecode(raw) as SidecarIndex - if (idx.version === 1) { - this.sidecars.set(meta.file, idx) - return idx - } - } catch { - // fall through to rebuild - } - } - // Sidecars are DERIVED: rebuild from the segment, loudly — never serve - // wrong offsets silently. - prodLog.warn( - `[GenerationSegments] sidecar for ${meta.file} missing or unreadable — rebuilding from the segment` - ) - const rebuilt = await this.rebuildSidecar(meta) - await this.storage.writeRawBytes(idxPath, msgpackEncode(rebuilt)) - this.sidecars.set(meta.file, rebuilt) - return rebuilt - } - - /** One sequential read of the segment → a fresh sidecar. Verifies every frame CRC. */ - private async rebuildSidecar(meta: SegmentMeta): Promise { - const frames = await this.readAllFrames(meta) - const idx: SidecarIndex = { version: 1, generations: [], ids: {} } - for (const f of frames) { - idx.generations.push([f.generation, f.offset, f.frameLen]) - for (const r of f.records) { - const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}` - ;(idx.ids[key] ??= []).push(f.generation) - } - } - return idx - } - - private decodeFrame( - payload: Uint8Array - ): { generation: number; timestamp: number; delta: unknown; records: FoldGeneration['records'] } { - const [generation, timestamp, delta, rawRecords] = msgpackDecode(payload) as [ - number, - number, - unknown, - Array<[number, string, unknown]>, - number - ] - return { - generation, - timestamp, - delta, - records: rawRecords.map(([kindByte, id, record]) => ({ - kind: kindByte === 0 ? ('noun' as const) : ('verb' as const), - id, - record - })) - } - } - - private async readAllFrames(meta: SegmentMeta): Promise< - Array & { offset: number; frameLen: number }> - > { - const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`) - if (!bytes) { - throw new Error( - `[GenerationSegments] sealed segment ${meta.file} is MISSING — packed history is damaged; ` + - `refusing to continue silently` - ) - } - const out: Array & { offset: number; frameLen: number }> = [] - let at = MAGIC.length - const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength) - while (at + FRAME_PREFIX_BYTES <= bytes.length) { - const payloadLen = view.getUint32(at, true) - const crc = view.getUint32(at + 4, true) - const payload = bytes.subarray(at + FRAME_PREFIX_BYTES, at + FRAME_PREFIX_BYTES + payloadLen) - if (payload.length !== payloadLen || crc32c(payload) !== crc) { - throw new Error( - `[GenerationSegments] frame CRC mismatch in ${meta.file} at offset ${at} — ` + - `packed history is damaged; refusing to serve it` - ) - } - out.push({ ...this.decodeFrame(payload), offset: at, frameLen: FRAME_PREFIX_BYTES + payloadLen }) - at += FRAME_PREFIX_BYTES + payloadLen - } - return out - } - - /** Read one packed generation's frame via its sidecar offset (one ranged read). */ - private async readFrame( - gen: number - ): Promise | null> { - const meta = this.coveringSegment(gen) - if (!meta) return null - const idx = await this.sidecarFor(meta) - // generations ascending → binary search. - const gens = idx.generations - let lo = 0 - let hi = gens.length - 1 - while (lo <= hi) { - const mid = (lo + hi) >> 1 - if (gens[mid][0] < gen) lo = mid + 1 - else if (gens[mid][0] > gen) hi = mid - 1 - else { - const [, offset, frameLen] = gens[mid] - const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`) - if (!bytes) { - throw new Error(`[GenerationSegments] sealed segment ${meta.file} is MISSING`) - } - const frame = bytes.subarray(offset, offset + frameLen) - const view = new DataView(frame.buffer, frame.byteOffset, frame.byteLength) - const payloadLen = view.getUint32(0, true) - const crc = view.getUint32(4, true) - const payload = frame.subarray(FRAME_PREFIX_BYTES, FRAME_PREFIX_BYTES + payloadLen) - if (payload.length !== payloadLen || crc32c(payload) !== crc) { - throw new Error( - `[GenerationSegments] frame CRC mismatch for generation ${gen} in ${meta.file} — ` + - `packed history is damaged; refusing to serve it` - ) - } - return this.decodeFrame(payload) - } - } - // In the covering range but not present: the packed tier is dense by - // construction (fold packs every generation it is handed, including - // record-less ones) — absence inside a sealed range is damage. - throw new Error( - `[GenerationSegments] generation ${gen} is inside sealed segment ${meta.file}'s declared ` + - `range but has no frame — packed history is damaged` - ) - } - - /** The packed tier's delta for `gen` (null = not packed). */ - async readDelta(gen: number): Promise<{ delta: unknown; timestamp: number } | null> { - const frame = await this.readFrame(gen) - return frame ? { delta: frame.delta, timestamp: frame.timestamp } : null - } - - /** The packed tier's full record-set for `gen` (null = not packed). */ - async readRecords(gen: number): Promise { - const frame = await this.readFrame(gen) - return frame ? frame.records : null - } - - /** One packed before-image (null = not packed OR no record for the id in that generation). */ - async readRecord(gen: number, kind: 'noun' | 'verb', id: string): Promise { - const frame = await this.readFrame(gen) - if (!frame) return null - const hit = frame.records.find((r) => r.kind === kind && r.id === id) - return hit ? hit.record : null - } - - /** - * D3 reclaim: drop WHOLE segments whose lastGeneration < `belowGeneration` - * and bump `compactedBelow`. Partial segments are never dropped — the - * boundary waits. NEVER called under the archival profile (the caller - * enforces retention semantics; this method only executes boundary drops). - */ - async dropSegmentsBelow(belowGeneration: number): Promise<{ dropped: number; compactedBelow: number }> { - const keep: SegmentMeta[] = [] - const drop: SegmentMeta[] = [] - for (const s of this.manifest.segments) { - ;(s.lastGeneration < belowGeneration ? drop : keep).push(s) - } - if (drop.length === 0) { - return { dropped: 0, compactedBelow: this.manifest.compactedBelow } - } - const compactedBelow = Math.max( - this.manifest.compactedBelow, - drop[drop.length - 1].lastGeneration + 1 - ) - // Manifest first (the drop is authoritative once named), then bytes — - // a crash between leaves orphan segment files invisible to the manifest, - // harmless and re-collectable. - const next: SegmentManifest = { ...this.manifest, compactedBelow, segments: keep } - await this.storage.writeRawObject(MANIFEST_PATH, next) - await this.storage.syncRawObjects([MANIFEST_PATH]) - this.manifest = next - for (const s of drop) { - await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${s.file}`) - await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${sidecarFileName(s.firstGeneration)}`) - this.sidecars.delete(s.file) - } - return { dropped: drop.length, compactedBelow } - } - - /** - * D8 rider — the packed portion of `generationDigest(g)`: a deterministic - * crc32c chain over sealed-segment checksums fully below `g`, plus the - * frame CRC of `g`'s own frame when `g` is mid-segment. O(segments), not - * O(generations); identical history ⇒ identical digest on any machine. - * The live-tier portion is composed by the caller. - */ - async digestThroughPacked(g: number): Promise { - let digest = 0 - let covered = false - for (const s of this.manifest.segments) { - if (s.lastGeneration <= g) { - digest = crc32c(new TextEncoder().encode(`${digest}:${s.checksum}`)) - if (s.lastGeneration === g) covered = true - } else if (s.firstGeneration <= g) { - // g is mid-segment: chain the partial prefix via g's frame CRC. - const frame = await this.readFrame(g) - if (frame === null) return null - const idx = await this.sidecarFor(s) - const upTo = idx.generations.filter(([gen]) => gen <= g) - for (const [gen, offset, frameLen] of upTo) { - digest = crc32c(new TextEncoder().encode(`${digest}:${gen}:${offset}:${frameLen}`)) - } - covered = true - break - } - } - return covered || this.manifest.segments.length > 0 ? digest : null - } -} diff --git a/src/db/generationStore.ts b/src/db/generationStore.ts index aede17a4..4e3738d9 100644 --- a/src/db/generationStore.ts +++ b/src/db/generationStore.ts @@ -46,8 +46,6 @@ import type { TxLogEntry } from './types.js' import { FactLog, storageSupportsFactLog, type CommitFact, type FactOp } from './factLog.js' -import { GenerationSegmentStore, type FoldGeneration } from './generationSegments.js' -import { crc32c } from '../utils/crc32c.js' /** * The byte-identical before-images of every id a commit touches, read UNDER @@ -268,21 +266,6 @@ export class GenerationStore { */ private historyBytesTotal: number | null = null - /** - * The packed tier (D1+D3): sealed segments holding folded cold - * generations. Null until {@link open} wires it (and on storage adapters - * without raw-byte primitives — the live tier then carries everything, - * exactly as before the packed tier existed). - */ - private segments: GenerationSegmentStore | null = null - - /** - * Live-tier window: generations newer than `committed - REPACK_LIVE_WINDOW` - * are never folded — the hot tail stays in the per-generation layout the - * write path owns. Matches the resident chain window's scale. - */ - static readonly REPACK_LIVE_WINDOW = 1024 - /** * Model-B per-write group-commit — the in-memory PENDING tier. * @@ -450,33 +433,6 @@ export class GenerationStore { this.factLog = null } - // PACKED TIER (D1+D3): same capability gate as the fact log. Opening - // reads ONE manifest — never a listing of the packed backlog — and seeds - // committedRanges with the sealed ranges so packed generations resolve - // exactly like live ones. - if (storageSupportsFactLog(this.storage)) { - this.segments = new GenerationSegmentStore(this.storage) - await this.segments.open() - const packedRanges = this.segments - .segments() - .map((s): [number, number] => [s.firstGeneration, Math.min(s.lastGeneration, this.committed)]) - .filter(([lo, hi]) => lo <= hi) - if (packedRanges.length > 0) { - // Merge packed (older) + live (newer) interval sets — both ascending; - // coalesce adjacency so range arithmetic stays interval-exact. - const merged: Array<[number, number]> = [] - for (const r of [...packedRanges, ...this.committedRanges].sort((a, b) => a[0] - b[0])) { - const last = merged[merged.length - 1] - if (last && r[0] <= last[1] + 1) last[1] = Math.max(last[1], r[1]) - else merged.push([r[0], r[1]]) - } - this.committedRanges = merged - } - this.horizonGen = Math.max(this.horizonGen, this.segments.compactedBelow() - 1) - } else { - this.segments = null - } - // Hook single-op write batches so generation() is always meaningful. // Suppressed while a transact batch executes (the batch is ONE generation). if (!options?.readOnly) { @@ -544,51 +500,6 @@ export class GenerationStore { * deltas (cache-bounded reads). * @returns Counts, bytes, generation range, and the compaction horizon. */ - /** - * @description D8 (gate-to-generation provenance): a deterministic content - * digest of the generation log THROUGH `g` — identical history ⇒ identical - * digest on any machine; any divergence (different records, different - * order, reclaimed range) ⇒ different digest. Composed from the packed - * tier's sealed-segment checksum chain (O(segments)) plus the live tier's - * per-generation delta digests (O(live window at most)). Release gates pin - * {generation, digest} and verify both at execution time. - * @param g - The generation to digest through (≤ committed). - * @returns A hex digest string, stable across reopen and repacking states - * ONLY for fully-packed prefixes — repacking changes representation, so - * the composed digest is defined over CONTENT: live-tier gens hash their - * delta + record ids, packed gens hash via frame CRCs. A gate should pin - * after a repack pass for long-term stability, or re-pin on repack. - */ - async generationDigest(g: number): Promise { - if (!Number.isInteger(g) || g < 1 || g > this.committed) { - throw new RangeError( - `generationDigest(): generation ${g} is out of range [1, ${this.committed}]` - ) - } - if (g <= this.horizonGen) { - throw new GenerationCompactedError(g, this.horizonGen) - } - let digest = 0 - const enc = new TextEncoder() - if (this.segments) { - const packed = await this.segments.digestThroughPacked(g) - if (packed !== null) digest = packed - } - // Live-tier composition: every committed gen ≤ g not covered by a sealed - // segment hashes its delta content in ascending order. - for (const gen of this.committedGensAsc()) { - if (gen > g) break - if (this.segments?.hasGeneration(gen)) continue - const delta = await this.getDelta(gen) - digest = crc32c( - enc.encode( - `${digest}:${gen}:${delta.timestamp}:${[...delta.nouns].sort().join(',')}:${[...delta.verbs].sort().join(',')}` - ) - ) - } - return digest.toString(16).padStart(8, '0') - } - async historyStats(): Promise<{ generations: number bytes: number @@ -627,17 +538,14 @@ export class GenerationStore { try { paths = await this.storage.listRawObjects(`${GENERATIONS_PREFIX}/${gen}/prev`) } catch { - paths = [] + return [] } const records: GenerationRecord[] = [] for (const p of paths) { const record = (await this.storage.readRawObject(p)) as GenerationRecord | null if (record) records.push(record) } - if (records.length > 0) return records - // Two-tier: folded generations serve their record-set from the segment. - const packed = await this.segments?.readRecords(gen) - return packed ? (packed.map((r) => r.record) as GenerationRecord[]) : [] + return records } /** @@ -1875,15 +1783,9 @@ export class GenerationStore { if (pending) { return (kind === 'noun' ? pending.nouns : pending.verbs).get(id) ?? null } - const live = (await this.storage.readRawObject( + return (await this.storage.readRawObject( `${GENERATIONS_PREFIX}/${gen}/prev/${id}.json` )) as GenerationRecord | null - if (live) return live - // Two-tier: the packed tier serves folded generations (live-tier-wins). - if (this.segments?.hasGeneration(gen)) { - return (await this.segments.readRecord(gen, kind, id)) as GenerationRecord | null - } - return null } /** @@ -2230,21 +2132,6 @@ export class GenerationStore { `${GENERATIONS_PREFIX}/${gen}/tx.json` )) as GenerationDelta | null if (delta === null) { - // Two-tier read (D1+D3): not in the live tier → the packed tier. - // Live-tier-wins ordering (a crash mid-fold leaves a duplicate, never - // a gap), so the segment lookup runs only after the live miss. - const packed = await this.segments?.readDelta(gen) - if (packed) { - const d = packed.delta as GenerationDelta - const entry = { - nouns: new Set(d.nouns), - verbs: new Set(d.verbs), - timestamp: packed.timestamp, - bytes: d.bytes ?? 0 - } - this.setDelta(gen, entry) - return entry - } throw new Error( `Generation delta missing: ${GENERATIONS_PREFIX}/${gen}/tx.json ` + `(store corrupted or records removed outside compactHistory())` @@ -2326,94 +2213,6 @@ export class GenerationStore { * @param options - Retention caps (see {@link CompactHistoryOptions}). * @returns Count of removed record-sets and the new horizon. */ - /** - * @description The REPACKER (D1+D3+repacking): fold cold live-tier - * generations into sealed segments — re-representation, never deletion. - * Every record and delta stays readable (asOf/chains unchanged); the - * per-generation directories are deleted only AFTER their segment is - * durable (crash between = duplicate representation, resolved - * live-tier-wins by every reader; never a gap). This is the transform that - * takes a 70k-file history to tens of segment files, and the ONLY history - * transform permitted under the archival profile. - * - * Folds oldest-first, contiguous from the packed boundary, in batches, and - * stops at the live window ({@link GenerationStore.REPACK_LIVE_WINDOW}) - * or when `timeBudgetMs` is spent — an early stop is a consistent prefix; - * the next pass resumes. - */ - async repackHistory(options?: { timeBudgetMs?: number; batchGenerations?: number }): Promise<{ - foldedGenerations: number - segmentsCreated: number - }> { - if (!this.segments) return { foldedGenerations: 0, segmentsCreated: 0 } - const segments = this.segments - return this.withMutex(async () => { - const deadline = - options?.timeBudgetMs !== undefined ? Date.now() + options.timeBudgetMs : undefined - const batchSize = options?.batchGenerations ?? 512 - const coldCeiling = this.committed - GenerationStore.REPACK_LIVE_WINDOW - const packedThrough = - segments.segments().length > 0 - ? segments.segments()[segments.segments().length - 1].lastGeneration - : 0 - - // Cold, unpacked, committed generations — ascending, contiguous scan. - const eligible: number[] = [] - for (const gen of this.committedGensAsc()) { - if (gen > coldCeiling) break - if (gen <= packedThrough) continue // already packed (dup fold barred) - if (this.pendingBuffer.has(gen)) continue // un-flushed = live by definition - eligible.push(gen) - } - - let folded = 0 - let segmentsCreated = 0 - for (let i = 0; i < eligible.length; i += batchSize) { - if (deadline !== undefined && Date.now() >= deadline) break - const batch = eligible.slice(i, i + batchSize) - const foldInput: FoldGeneration[] = [] - for (const gen of batch) { - const delta = (await this.storage.readRawObject( - `${GENERATIONS_PREFIX}/${gen}/tx.json` - )) as GenerationDelta | null - if (delta === null) { - // Already folded by a prior crashed pass whose dirs were removed, - // or damage — getDelta's two-tier read decides which, loudly, - // when someone asks. Skip; never fold a generation we cannot read. - continue - } - const records: FoldGeneration['records'] = [] - for (const [kind, ids] of [ - ['noun', delta.nouns] as const, - ['verb', delta.verbs] as const - ]) { - for (const id of ids) { - const record = await this.storage.readRawObject( - `${GENERATIONS_PREFIX}/${gen}/prev/${id}.json` - ) - if (record) records.push({ kind, id, record }) - } - } - foldInput.push({ generation: gen, timestamp: delta.timestamp, delta, records }) - } - if (foldInput.length === 0) continue - await segments.fold(foldInput) - segmentsCreated++ - // Segment + manifest durable → the live copies retire. - for (const g of foldInput) { - await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${g.generation}`) - } - folded += foldInput.length - } - if (folded > 0) { - prodLog.info( - `[GenerationStore] repacked ${folded} cold generation(s) into ${segmentsCreated} segment(s) — history preserved, file count reduced` - ) - } - return { foldedGenerations: folded, segmentsCreated } - }) - } - async compact(options?: CompactHistoryOptions): Promise { return this.withMutex(async () => { const minPinned = this.minPinnedGeneration() @@ -2505,16 +2304,6 @@ export class GenerationStore { // Reclaimed generations leave the per-id chains stale → rebuild on next read. this.invalidateChains() this.horizonGen = Math.max(this.horizonGen, highestRemoved) - // Packed-tier reclaim (D3): a packed generation's bytes live in a - // sealed segment — removeRawPrefix above was a no-op for it. Drop - // WHOLE segments now fully below the horizon; a partially-reclaimed - // segment keeps its bytes until the boundary passes it (the frozen - // partial-segments-wait rule; logical reclamation above still holds — - // the generations left committedRanges and asOf below the horizon - // throws regardless). - if (this.segments) { - await this.segments.dropSegmentsBelow(this.horizonGen + 1) - } const manifest: GenerationManifest = { version: 1, generation: this.committed, diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts index eb2acd71..605d2a0b 100644 --- a/src/hnsw/hnswIndex.ts +++ b/src/hnsw/hnswIndex.ts @@ -54,9 +54,6 @@ export class HnswFlushError extends Error { * acceleration provider). */ export class JsHnswVectorIndex implements VectorIndexProvider { - /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.name}. */ - readonly name = 'hnsw-js' - private nouns: Map = new Map() /** * Reverse adjacency: `target id → (level → set of node ids that link TO target)`. diff --git a/src/index.ts b/src/index.ts index bfab39da..ee01d885 100644 --- a/src/index.ts +++ b/src/index.ts @@ -28,9 +28,6 @@ export type { FileVersion } from './vfs/types.js' // Export diagnostics result type export type { DiagnosticsResult } from './brainy.js' -// brain.warm() — eager index/storage readiness report (per-surface honest -// outcome + timing). See the WarmReport JSDoc in brainy.ts. -export type { WarmReport, WarmOutcome } from './brainy.js' export type { GraphAuditReport, GraphAuditDiscrepancy @@ -216,7 +213,6 @@ export type { CommitFact, FactOp, FactScanBatch, - SCANFACTS_FIRST_BATCH_MS, FactScanHandle } from './db/factLog.js' // The generalized family stamp — which source generation a projection diff --git a/src/plugin.ts b/src/plugin.ts index 02639955..df7c7224 100644 --- a/src/plugin.ts +++ b/src/plugin.ts @@ -381,20 +381,6 @@ export interface GraphIndexProvider { */ init?(): Promise - /** - * @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap - * pretouch) so first operations run at steady-state cost. Optional; - * absence means the provider demand-loads. Distinct from `init?()`: `init` - * is called once automatically during brain startup as part of the - * readiness contract (cold-load + rebuild gating); `warm` is an explicit, - * separate readiness step a caller opts into via `brain.warm()` (or - * `warmOnOpen`) specifically to pre-pay demand-load cost that `init` left - * lazy. A provider that already loads everything eagerly in `init?()` may - * implement `warm` as a no-op or omit it — `brain.warm()` falls back to a - * best-effort read-through probe when absent. - */ - warm?(): Promise - /** * @description OPTIONAL. A native provider returns true from the moment its * `init()` detects a large epoch-drift until its background @@ -961,26 +947,6 @@ export interface AtGenerationVectors { * are retired and never looked up. */ export interface VectorIndexProvider { - /** - * @description REQUIRED self-reported implementation identity, rendered as - * `[vector-index:]` in prose log lines and stamped into the - * transaction op-name strings that surface in journals/timings (e.g. - * `AddToVectorIndex(hnsw-js)`) — see - * `src/transaction/operations/IndexOperations.ts`. A provider must name - * itself TRUTHFULLY (its own algorithm/engine, e.g. its plugin/package - * name) and must never inherit a default — guessing a backend name would - * resurrect the exact bug this field exists to prevent: an operator - * hunting an index that isn't the one actually running. The built-in JS - * index always sets this to `'hnsw-js'`; a native provider picks its own - * string. At the TypeScript level this field is required; a runtime - * instance from an older provider compiled against the previous optional - * `providerId` contract is tolerated (never crashes, never silently - * mislabeled) — see `resolveVectorProviderId` in - * `src/transaction/operations/IndexOperations.ts`, which stamps - * `'unknown-provider'` and emits one loud warning for that case. - */ - readonly name: string - addItem(item: VectorDocument): Promise removeItem(id: string): Promise search( @@ -1043,20 +1009,6 @@ export interface VectorIndexProvider { */ init?(): Promise - /** - * @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap - * pretouch) so first operations run at steady-state cost. Optional; - * absence means the provider demand-loads. Distinct from `init?()`: `init` - * runs automatically once during brain startup (the cold-load + rebuild - * readiness contract above); `warm` is a separate, explicit step a caller - * opts into via `brain.warm()` (or `warmOnOpen`) to pre-pay demand-load - * cost `init` left lazy — e.g. touching every mmap page rather than just - * opening the file. A provider that already loads everything eagerly in - * `init?()` may implement `warm` as a no-op or omit it — `brain.warm()` - * falls back to a best-effort probe `search()` when absent. - */ - warm?(): Promise - /** * @description OPTIONAL honest durability signal (readiness contract, * mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted diff --git a/src/storage/adapters/memoryStorage.ts b/src/storage/adapters/memoryStorage.ts index 1b1f412e..bab9d4d9 100644 --- a/src/storage/adapters/memoryStorage.ts +++ b/src/storage/adapters/memoryStorage.ts @@ -133,11 +133,6 @@ export class MemoryStorage extends BaseStorage { */ protected async deleteObjectFromPath(path: string): Promise { this.objectStore.delete(path) - // Filesystem parity: on disk, objects and raw BYTE files are both just - // files — unlink removes whichever exists. Without this, deleteRawObject - // on a raw-bytes path (fact-log/generation segments) silently no-ops on - // memory storage: the delete "succeeds" and the bytes remain. - this.rawBytesStore.delete(path) } /** diff --git a/src/transaction/Transaction.ts b/src/transaction/Transaction.ts index 79b53006..092a2a25 100644 --- a/src/transaction/Transaction.ts +++ b/src/transaction/Transaction.ts @@ -37,47 +37,22 @@ const DEFAULT_OPTIONS: Required = { maxRollbackRetries: 3 } -/** - * The floor term of {@link transactTimeoutBudget}'s scaling formula when the - * caller supplies neither an explicit override nor `config.transactionBudgetFloorMs`. - */ -const DEFAULT_BUDGET_FLOOR_MS = 30_000 - /** * The apply-phase budget for a batch of `opCount` operations. * - * An explicit `override` wins untouched (a caller-specified deadline for this - * one batch). Otherwise the budget SCALES with the batch: - * `max(floorMs, opCount × 2 000)`, where `floorMs` defaults to `30 000` and - * is overridable via `BrainyConfig.transactionBudgetFloorMs` for the whole - * brain. The per-op term is calibrated from field data — bulk imports on - * network-attached disks measure ~2 s per operation (each op pays canonical - * writes + fsync + index maintenance) — so a flat 30 s budget silently capped - * honest work at ~15 operations while looking generous for small batches. A - * 16-op batch, for example, scales to `max(30 000, 16 × 2 000) = 32 000`. - * Scaling keeps small transacts fast-failing and gives bulk ones a budget - * proportional to the work they actually asked for. - * - * This is a **mid-batch** budget only: it governs whether the transaction's - * NEXT operation may start (see {@link Transaction.execute}), never whether - * already-completed work is rolled back after the fact. A trip mid-batch - * still rolls back every operation applied so far, atomically, and throws a - * retryable, fully-labeled TransactionTimeoutError — that zero-loss guarantee - * doesn't change; only the point at which the clock stops mattering does (at - * the last operation, not one check later). - * - * @param opCount - Number of operations in the batch. - * @param override - A full override for this call; wins over everything else. - * @param floorMs - The scaling floor for this call (e.g. `config.transactionBudgetFloorMs`). - * Ignored when `override` is set. Defaults to `30 000` when omitted. + * An explicit override wins untouched. Otherwise the budget SCALES with the + * batch: `max(30 000 ms, opCount × 2 000 ms)`. The per-op term is calibrated + * from field data — bulk imports on network-attached disks measure ~2 s per + * operation (each op pays canonical writes + fsync + index maintenance) — so + * a flat 30 s budget silently capped honest work at ~15 operations while + * looking generous for small batches. Scaling keeps small transacts + * fast-failing and gives bulk ones a budget proportional to the work they + * actually asked for; a trip still rolls back atomically and throws a + * retryable, fully-labeled TransactionTimeoutError. */ -export function transactTimeoutBudget( - opCount: number, - override?: number, - floorMs?: number -): number { +export function transactTimeoutBudget(opCount: number, override?: number): number { if (override !== undefined) return override - return Math.max(floorMs ?? DEFAULT_BUDGET_FLOOR_MS, opCount * 2_000) + return Math.max(30_000, opCount * 2_000) } /** @@ -123,20 +98,7 @@ export class Transaction implements TransactionContext { } /** - * Execute all operations atomically. - * - * Budget semantics: the configured budget gates starting the next - * operation; completed work is never rolled back for elapsed time. Elapsed - * time is checked ONLY before starting operation `i` for `i ≥ 1` — never - * before operation 0 (an operation always gets to start) and never again - * after the final operation completes. Concretely: a single-op transaction - * can never time out post-hoc — it either runs (and its result stands, no - * matter how long it took) or a `TransactionTimeoutError` is thrown before - * it starts, which cannot happen since there is no operation before it. A - * multi-op transaction whose op `i` overruns the budget stops op `i+1` from - * starting, rolls back operations `0..i` (reverse order), and throws - * `TransactionTimeoutError` — that zero-loss rollback guarantee for - * mid-batch trips is unchanged; only the post-completion check is gone. + * Execute all operations atomically */ async execute(): Promise { if (this.state !== 'pending') { @@ -166,14 +128,10 @@ export class Transaction implements TransactionContext { // already-applied writes as torn, generation-less state in canonical // storage. for (let i = 0; i < this.operations.length; i++) { - // Budget gates STARTING the next operation; completed work is never - // rolled back for elapsed time. Skipped for i === 0 (operation 0 - // always gets to start — nothing has run yet to have overrun) and - // never re-checked after the final operation completes (there is no - // "next operation" left to gate). A trip here throws into the catch - // below and rolls back like any other failure — it must never bypass - // rollback. - if (i > 0 && Date.now() - this.startTime > this.options.timeout) { + // Budget check BEFORE starting the next operation. A trip here throws + // into the catch below and rolls back like any other failure — it must + // never bypass rollback. + if (Date.now() - this.startTime > this.options.timeout) { throw new TransactionTimeoutError(this.options.timeout, i, { elapsedMs: Date.now() - this.startTime, totalOperations: this.operations.length, diff --git a/src/transaction/operations/IndexOperations.ts b/src/transaction/operations/IndexOperations.ts index d130bb3f..97c39270 100644 --- a/src/transaction/operations/IndexOperations.ts +++ b/src/transaction/operations/IndexOperations.ts @@ -2,73 +2,34 @@ * Index Operations with Rollback Support * * Provides transactional operations for all indexes: - * - VectorIndexProvider (the JS HNSW fallback, or a native acceleration provider) + * - JsHnswVectorIndex (unified vector index) * - MetadataIndexManager (roaring bitmap filtering) * - GraphAdjacencyIndex (LSM-tree graph storage) * * Each operation can be executed and rolled back atomically. */ -import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js' +import type { JsHnswVectorIndex } from '../../hnsw/hnswIndex.js' import type { MetadataIndexManager } from '../../utils/metadataIndex.js' +import type { GraphIndexProvider } from '../../plugin.js' import type { GraphVerb } from '../../coreTypes.js' import type { Operation, RollbackAction } from '../types.js' /** - * Backend identity stamped into an operation's emitted `name` string (e.g. - * `AddToVectorIndex(hnsw-js)`), resolved from the provider's own - * {@link VectorIndexProvider.name} self-report. - * - * These operation classes are backend-neutral (the vector index they wrap may - * be Brainy's own JS HNSW fallback OR a native acceleration provider), but - * their names surface directly in consumer-visible transaction journals and - * timings. `name` is REQUIRED at the TypeScript level — but a native provider - * instance compiled against the previous (pre-required) contract can still - * reach this function at runtime without it. That legacy case is tolerated, - * never crashed on and never silently mislabeled: it resolves to - * `'unknown-provider'` and emits ONE loud warning naming the missing contract - * field, so the fix (implement `name`) is discoverable rather than a silent - * fossil label in every journal line thereafter. - */ -const warnedMissingName = new WeakSet() - -function resolveVectorProviderId(index: VectorIndexProvider): string { - const name = (index as { name?: unknown }).name - if (typeof name === 'string') return name - if (!warnedMissingName.has(index)) { - warnedMissingName.add(index) - console.warn( - '[vector-index] provider is missing the required `name` field (VectorIndexProvider.name, ' + - 'required since 8.10.0) — stamping "unknown-provider" in transaction op names until the ' + - 'provider declares its own identity.' - ) - } - return 'unknown-provider' -} - -/** - * Add to the vector index with rollback support. - * - * Backend-neutral: `index` is whatever the `'vector'` provider factory - * returns — Brainy's own JS HNSW fallback, or a native acceleration provider - * (e.g. DiskANN). The emitted `name` stamps the active backend (see - * {@link resolveVectorProviderId}) so operators reading a transaction journal - * or timing trace see which engine actually ran, never a fossil name from - * whichever engine happened to be active when this op class was written. + * Add to HNSW index with rollback support * * Rollback strategy: * - Remove item from index + */ -export class AddToVectorIndexOperation implements Operation { - readonly name: string +export class AddToHNSWOperation implements Operation { + readonly name = 'AddToHNSW' constructor( - private readonly index: VectorIndexProvider, + private readonly index: JsHnswVectorIndex, private readonly id: string, private readonly vector: number[] - ) { - this.name = `AddToVectorIndex(${resolveVectorProviderId(index)})` - } + ) {} async execute(): Promise { // Check if item already exists (for rollback decision) @@ -98,12 +59,11 @@ export class AddToVectorIndexOperation implements Operation { * pre-existence as "existed" made every rollback skip removeItem, leaving * phantom entries in the index after a failed transaction. The safe default * is to remove what this operation added — update flows pair this op with a - * RemoveFromVectorIndexOperation whose own rollback restores the prior - * vector, so reverse-order rollback reconstructs the original state either - * way. + * RemoveFromHNSWOperation whose own rollback restores the prior vector, so + * reverse-order rollback reconstructs the original state either way. */ private async itemExists(id: string): Promise { - const index = this.index as VectorIndexProvider & { + const index = this.index as JsHnswVectorIndex & { getItem?: (id: string) => Promise } if (typeof index.getItem !== 'function') return false @@ -117,27 +77,21 @@ export class AddToVectorIndexOperation implements Operation { } /** - * Remove from the vector index with rollback support. - * - * Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the - * JS HNSW fallback or a native acceleration provider; the emitted `name` - * stamps the active backend. + * Remove from HNSW index with rollback support * * Rollback strategy: * - Re-add item to index with original vector * * Note: Requires storing the vector for rollback */ -export class RemoveFromVectorIndexOperation implements Operation { - readonly name: string +export class RemoveFromHNSWOperation implements Operation { + readonly name = 'RemoveFromHNSW' constructor( - private readonly index: VectorIndexProvider, + private readonly index: JsHnswVectorIndex, private readonly id: string, private readonly vector: number[] // Required for rollback - ) { - this.name = `RemoveFromVectorIndex(${resolveVectorProviderId(index)})` - } + ) {} async execute(): Promise { // Remove from index @@ -331,24 +285,22 @@ export class RemoveFromGraphIndexOperation implements Operation { } /** - * Batch operation: Add multiple items to the vector index (backend-neutral — - * see {@link AddToVectorIndexOperation}). + * Batch operation: Add multiple items to HNSW index * * Useful for bulk imports with transaction support. * Rolls back all items if any fail. */ -export class BatchAddToVectorIndexOperation implements Operation { - readonly name: string +export class BatchAddToHNSWOperation implements Operation { + readonly name = 'BatchAddToHNSW' - private operations: AddToVectorIndexOperation[] + private operations: AddToHNSWOperation[] constructor( - index: VectorIndexProvider, + index: JsHnswVectorIndex, items: Array<{ id: string; vector: number[] }> ) { - this.name = `BatchAddToVectorIndex(${resolveVectorProviderId(index)})` this.operations = items.map( - item => new AddToVectorIndexOperation(index, item.id, item.vector) + item => new AddToHNSWOperation(index, item.id, item.vector) ) } diff --git a/src/transaction/operations/index.ts b/src/transaction/operations/index.ts index c5548e70..422f6dad 100644 --- a/src/transaction/operations/index.ts +++ b/src/transaction/operations/index.ts @@ -21,12 +21,12 @@ export { // Index Operations export { - AddToVectorIndexOperation, - RemoveFromVectorIndexOperation, + AddToHNSWOperation, + RemoveFromHNSWOperation, AddToMetadataIndexOperation, RemoveFromMetadataIndexOperation, AddToGraphIndexOperation, RemoveFromGraphIndexOperation, - BatchAddToVectorIndexOperation, + BatchAddToHNSWOperation, BatchAddToMetadataIndexOperation } from './IndexOperations.js' diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts index 8bede8d3..1ec4c4c3 100644 --- a/src/types/brainy.types.ts +++ b/src/types/brainy.types.ts @@ -1651,25 +1651,6 @@ export interface BrainyConfig { */ disableAutoRebuild?: boolean - /** - * The floor (ms) of the apply-phase transaction budget's scaling formula: - * `max(transactionBudgetFloorMs, opCount × 2 000)`. The budget governs - * whether a `transact()` / single-op write's NEXT internal operation may - * **start** — never whether already-completed work gets rolled back after - * the fact (a single-op write can never time out post-hoc: it either runs - * or it commits). A trip mid-batch still rolls back every applied operation - * atomically and throws a retryable `TransactionTimeoutError`; only the - * floor of the formula is configurable here. - * - * Raise this when a cold store's first writes after a restart legitimately - * take longer than 30s per operation (e.g. page-cache-cold canonical writes - * on a large brain) so a bulk `transact()` batch gets a proportionally - * larger runway instead of tripping mid-batch. Lower it to fail faster on a - * latency-sensitive write path. Default: `30000` (30 s) — unchanged from - * pre-8.9 behavior. - */ - transactionBudgetFloorMs?: number - /** * How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration * before throwing a retryable `MigrationInProgressError`. @@ -1825,23 +1806,6 @@ export interface BrainyConfig { */ eagerEmbeddings?: boolean - /** - * When `true`, `init()` **awaits `warm()`** (see {@link Brainy.warm}) before - * it resolves — blocking startup until the vector index, metadata index, - * and graph adjacency have all faulted their backing storage in. This is a - * deliberate operator opt-in trade: startup takes longer, but the FIRST - * request after a cold restart runs at steady-state cost instead of paying - * page-cache-miss / demand-load latency on the critical path. - * - * Runs AFTER `init()`'s own sequence completes (index construction, crash - * recovery, migrations, VFS bootstrap) — never in place of any of it — so - * `warm()` always operates on a fully-initialized brain. - * - * Default: `false` (lazy — the pre-8.9 behavior: indexes demand-load on - * first access). - */ - warmOnOpen?: boolean - // Plugin configuration // Controls which plugins are loaded during init(). // - undefined (default): guarded auto-detection of the first-party diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts index fdb17c22..c772bce4 100644 --- a/src/utils/metadataIndex.ts +++ b/src/utils/metadataIndex.ts @@ -423,46 +423,6 @@ export class MetadataIndexManager implements MetadataIndexProvider { prodLog.debug('✅ Type-aware cache warming completed') } - /** - * Full hydration — the {@link Brainy.warm} readiness seam for the metadata - * index. Unlike {@link warmCache} / {@link warmCacheForTopTypes} (which - * warm only a heuristic subset: common fields plus the top-N types' top - * fields), this loads EVERY field's sparse index the field registry knows - * about — a real read through {@link loadSparseIndex} into the unified - * cache for each field, not a stat/existence check. Idempotent: an - * already-cached field's `loadSparseIndex` call is a cheap cache hit. - * - * Re-reads the field registry first when `fieldIndexes` is empty (a warm() - * call issued before `init()` populated it would otherwise hydrate - * nothing), then loads every discovered field in parallel. - */ - async hydrateAll(): Promise { - if (this.fieldIndexes.size === 0) { - await this.loadFieldRegistry() - } - - const fields = Array.from(this.fieldIndexes.keys()) - if (fields.length === 0) { - prodLog.debug('[MetadataIndex] hydrateAll: no persisted fields to hydrate') - return - } - - prodLog.debug(`[MetadataIndex] hydrateAll: loading ${fields.length} field(s) — ${fields.join(', ')}`) - - await Promise.all( - fields.map(async field => { - try { - await this.loadSparseIndex(field) - } catch (error) { - // A single field's load failure doesn't abort the rest of the - // hydration — warm() is a best-effort readiness step, never a - // correctness gate (queries still demand-load on miss). - prodLog.debug(`[MetadataIndex] hydrateAll: field '${field}' failed to load:`, error) - } - }) - ) - } - /** * Acquire an in-memory lock for coordinating concurrent metadata index writes * Uses in-memory locks since MetadataIndexManager doesn't have direct file system access diff --git a/tests/integration/history-repacking.test.ts b/tests/integration/history-repacking.test.ts deleted file mode 100644 index 2bcee038..00000000 --- a/tests/integration/history-repacking.test.ts +++ /dev/null @@ -1,186 +0,0 @@ -/** - * @module tests/integration/history-repacking - * @description The D1+D3 two-tier history lifecycle end-to-end on a real - * brain. Laws: (1) repacking is RE-REPRESENTATION — after folding, every - * asOf() read below the fold boundary answers exactly as before, across a - * cold reopen; (2) folded per-generation directories are physically gone - * (the file-count cure is real, not cosmetic); (3) repack + reclaim compose: - * bounded retention after repacking drops whole segments and asOf below the - * horizon throws GenerationCompactedError; (4) repackHistory is explicit - * API and time-bounded (spent budget = consistent no-op). - * - * Uses a tiny REPACK_LIVE_WINDOW override so a small history has a cold - * tier at all (the production window is 1024). - */ -import { describe, it, expect, afterEach } from 'vitest' -import * as fs from 'node:fs' -import * as path from 'node:path' -import * as os from 'node:os' -import { Brainy } from '../../src/brainy.js' -import { NounType } from '../../src/types/graphTypes.js' -import { GenerationStore } from '../../src/db/generationStore.js' -import { GenerationCompactedError } from '../../src/db/errors.js' -import { SEGMENTS_PREFIX } from '../../src/db/generationSegments.js' - -const stub = async (text: string): Promise => { - const h = text.split('').reduce((a, c) => a + c.charCodeAt(0), 0) - return new Array(384).fill(0).map((_, i) => Math.sin(h + i)) -} - -const openBrain = async (dir: string): Promise => { - const brain = new Brainy({ - requireSubtype: false, - storage: { type: 'filesystem', path: dir }, - embeddingFunction: stub - }) - await brain.init() - return brain -} - -describe('history repacking — the two-tier lifecycle', () => { - const dirs: string[] = [] - const tempDir = (): string => { - const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-repack-')) - dirs.push(d) - return d - } - const originalWindow = GenerationStore.REPACK_LIVE_WINDOW - - afterEach(() => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = originalWindow - for (const d of dirs.splice(0)) { - try { - fs.rmSync(d, { recursive: true, force: true }) - } catch { - /* best effort */ - } - } - }) - - it('repack preserves every historical read across cold reopen; folded dirs are gone', async () => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = 3 - const dir = tempDir() - const brain = await openBrain(dir) - - const id = await brain.add({ - data: 'versioned-entity', - type: NounType.Document, - metadata: { v: 0 } - }) - for (let v = 1; v <= 10; v++) await brain.update({ id, metadata: { v } }) - await brain.flush() - - // Ground truth BEFORE repacking: capture asOf views for early generations. - const before: Record = {} - for (const g of [2, 4, 6]) { - const db = await brain.asOf(g) - before[g] = (await db.get(id))?.metadata?.v as number - await db.release() - } - - const result = await brain.repackHistory() - expect(result.foldedGenerations).toBeGreaterThan(0) - expect(result.segmentsCreated).toBeGreaterThan(0) - - // The folded per-generation directories are PHYSICALLY gone… - const genDirs = fs - .readdirSync(path.join(dir, '_generations'), { withFileTypes: true }) - .filter((e) => e.isDirectory() && /^\d+$/.test(e.name)).length - expect(genDirs).toBeLessThanOrEqual(4) // live window (3) + at most the newest - // …and the segment tier exists (the filesystem adapter stores objects - // gzipped, so the manifest may live at either spelling). - const segDir = path.join(dir, SEGMENTS_PREFIX) - expect( - fs.existsSync(path.join(segDir, 'manifest.json')) || - fs.existsSync(path.join(segDir, 'manifest.json.gz')) - ).toBe(true) - expect(fs.readdirSync(segDir).some((f) => f.endsWith('.bgs'))).toBe(true) - - // Same asOf answers from the packed tier, same process… - for (const g of [2, 4, 6]) { - const db = await brain.asOf(g) - expect((await db.get(id))?.metadata?.v).toBe(before[g]) - await db.release() - } - await brain.close() - - // …and across a COLD REOPEN (manifest discovery, no live dirs to list). - const reopened = await openBrain(dir) - for (const g of [2, 4, 6]) { - const db = await reopened.asOf(g) - expect((await db.get(id))?.metadata?.v).toBe(before[g]) - await db.release() - } - expect((await reopened.get(id))?.metadata?.v).toBe(10) // live state untouched - await reopened.close() - }) - - it('repack + bounded reclaim compose: whole segments drop, horizon is loud', async () => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = 2 - const dir = tempDir() - const brain = await openBrain(dir) - const id = await brain.add({ data: 'reclaim-probe', type: NounType.Document, metadata: { v: 0 } }) - for (let v = 1; v <= 8; v++) await brain.update({ id, metadata: { v } }) - await brain.flush() - await brain.repackHistory() - - // Reclaim down to the 3 newest generations — packed segments below the - // horizon drop whole; asOf below throws loudly. - const res = await brain.compactHistory({ maxGenerations: 3 }) - expect(res.removedGenerations).toBeGreaterThan(0) - await expect(brain.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError) - expect((await brain.get(id))?.metadata?.v).toBe(8) - await brain.close() - }) - - it('generationDigest: reopen-stable, divergence-sensitive, loud below the horizon', async () => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = 2 - const dir = tempDir() - const brain = await openBrain(dir) - const id = await brain.add({ data: 'digest-probe', type: NounType.Document, metadata: { v: 0 } }) - for (let v = 1; v <= 6; v++) await brain.update({ id, metadata: { v } }) - await brain.flush() - await brain.repackHistory() - - const gen = brain.generation() - const atHead = await brain.generationDigest(gen) - const atMid = await brain.generationDigest(3) - expect(atHead).toMatch(/^[0-9a-f]{8}$/) - expect(atMid).not.toBe(atHead) // more history ⇒ different digest - await brain.close() - - // Reopen-stable: same history, same digests (packed prefix stability). - const reopened = await openBrain(dir) - expect(await reopened.generationDigest(gen)).toBe(atHead) - expect(await reopened.generationDigest(3)).toBe(atMid) - - // New history diverges the head digest. - await reopened.update({ id, metadata: { v: 7 } }) - await reopened.flush() - expect(await reopened.generationDigest(reopened.generation())).not.toBe(atHead) - - // Below the horizon: LOUD, never a silent pin of reclaimed history. - await reopened.compactHistory({ maxGenerations: 2 }) - await expect(reopened.generationDigest(1)).rejects.toBeInstanceOf(GenerationCompactedError) - await reopened.close() - }) - - it('a spent time budget is a consistent no-op; the next pass resumes', async () => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = 2 - const dir = tempDir() - const brain = await openBrain(dir) - const id = await brain.add({ data: 'budget-probe', type: NounType.Document, metadata: { v: 0 } }) - for (let v = 1; v <= 6; v++) await brain.update({ id, metadata: { v } }) - await brain.flush() - - const bounded = await brain.repackHistory({ timeBudgetMs: 0 }) - expect(bounded).toEqual({ foldedGenerations: 0, segmentsCreated: 0 }) - - const resumed = await brain.repackHistory() - expect(resumed.foldedGenerations).toBeGreaterThan(0) - const db = await brain.asOf(3) - expect((await db.get(id))?.metadata?.v).toBeDefined() - await db.release() - await brain.close() - }) -}) diff --git a/tests/unit/brainy/warm.test.ts b/tests/unit/brainy/warm.test.ts deleted file mode 100644 index ce213696..00000000 --- a/tests/unit/brainy/warm.test.ts +++ /dev/null @@ -1,310 +0,0 @@ -/** - * @module tests/unit/brainy/warm - * @description Coverage for `brain.warm()` / `warmOnOpen` / the provider - * `warm?()` contract (the cold-restart readiness fix): first operations after - * a cold restart should run at steady-state cost instead of paying - * demand-load latency on the critical path. - * - * Uses a real, in-process fake plugin provider implementing the actual - * `VectorIndexProvider` contract from `src/plugin.ts` — the real seam a - * native provider (e.g. a disk-native accelerator) plugs into. The real - * built-in JS metadata index and graph adjacency index run against real - * (filesystem or in-memory) storage with pre-existing data, so their - * hydration paths are exercised for real, not mocked. - */ -import { describe, it, expect, afterEach } from 'vitest' -import * as fs from 'node:fs' -import * as os from 'node:os' -import * as path from 'node:path' -import { Brainy } from '../../../src/brainy.js' -import { NounType, VerbType } from '../../../src/types/graphTypes.js' -import type { VectorIndexProvider } from '../../../src/plugin.js' -import type { VectorDocument, Vector } from '../../../src/coreTypes.js' -import { MetadataIndexManager } from '../../../src/utils/metadataIndex.js' -import { GraphAdjacencyIndex } from '../../../src/graph/graphAdjacencyIndex.js' - -const tmpDirs: string[] = [] -function mkTmp(): string { - const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-warm-')) - tmpDirs.push(d) - return d -} -afterEach(() => { - for (const d of tmpDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true }) -}) - -// Brainy's ValidationConfig fixes vectors at exactly 384 dimensions -// (src/utils/paramValidation.ts) — match it so `add()` doesn't reject test data. -const DIM = 384 -const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin(seed + i)) - -/** - * A real (not mocked) VectorIndexProvider implementation, backed by a plain - * Map, that optionally implements `warm()` — the exact seam - * `AddToVectorIndexOperation` / `brain.warm()` call through. - */ -class FakeVectorProvider implements VectorIndexProvider { - readonly name = 'fake-vector-provider' - readonly items = new Map() - warmCalls = 0 - searchCalls: Array<{ k?: number }> = [] - // Only present on the instance when the constructor is told to — mirrors a - // real provider that may or may not implement the optional hook. Assigned - // in the constructor BODY (not as a field initializer): under native ES - // class fields, field initializers run before constructor-body statements - // — including the parameter-property assignment — so referencing a - // parameter property from a field initializer would see it as still - // `undefined`. - warm?: () => Promise - - constructor(hasWarm: boolean) { - if (hasWarm) { - this.warm = async (): Promise => { - this.warmCalls++ - } - } - } - - async addItem(item: VectorDocument): Promise { - this.items.set(item.id, item.vector) - return item.id - } - async removeItem(id: string): Promise { - return this.items.delete(id) - } - async search(_queryVector: Vector, k?: number): Promise> { - this.searchCalls.push({ k }) - return [...this.items.keys()].slice(0, k ?? 10).map((id) => [id, 0]) - } - size(): number { - return this.items.size - } - clear(): void { - this.items.clear() - } - async rebuild(): Promise {} - async flush(): Promise { - return 0 - } - getPersistMode(): 'immediate' | 'deferred' { - return 'deferred' - } -} - -/** Registers `provider` under the `'vector'` plugin key, before `init()`. */ -function useFakeVectorProvider(brain: Brainy, provider: FakeVectorProvider): void { - brain.use({ - name: 'fake-vector-provider', - activate: async (ctx: any) => { - ctx.registerProvider('vector', () => provider) - return true - } - }) -} - -describe('brain.warm()', () => { - it('(a) calls the vector provider\'s warm() when present and reports "warmed"', async () => { - const brain = new Brainy({ - requireSubtype: false, - storage: { type: 'memory' }, - silent: true - }) - const provider = new FakeVectorProvider(true) - useFakeVectorProvider(brain, provider) - await brain.init() - await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) }) - - const report = await brain.warm() - - expect(provider.warmCalls).toBe(1) - expect(provider.searchCalls.length).toBe(0) // warm() ran — no probe fallback - expect(report.vector.outcome).toBe('warmed') - await brain.close() - }) - - it('(b) falls back to a probe search() when warm() is absent and reports "probed", never "warmed"', async () => { - const brain = new Brainy({ - requireSubtype: false, - storage: { type: 'memory' }, - silent: true - }) - const provider = new FakeVectorProvider(false) - useFakeVectorProvider(brain, provider) - await brain.init() - await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) }) - await brain.add({ data: 'b', type: NounType.Thing, vector: V(2) }) - - const report = await brain.warm() - - expect(provider.warmCalls).toBe(0) // no warm() on this provider - expect(provider.searchCalls.length).toBe(1) // the probe ran - expect(provider.searchCalls[0].k).toBe(Math.min(100, provider.size())) // k = min(100, size) - expect(report.vector.outcome).toBe('probed') - expect(report.vector.outcome).not.toBe('warmed') // never conflated - await brain.close() - }) - - it('vector: reports "unavailable" when there is nothing to probe (empty index, unknown dimension)', async () => { - const brain = new Brainy({ - requireSubtype: false, - storage: { type: 'memory' }, - silent: true - }) - // A disk-native provider MAY honestly report 0 resident entries while - // durable data exists on disk (the same posture documented on - // `VectorIndexProvider.isReady` — "an mmap/disk-native index may - // legitimately report 0 resident entries"). warm()'s probe fallback - // reads `size()`, so this is the real trigger for "nothing to probe": - // never a k=0 search, an honest skip. - const provider = new FakeVectorProvider(false) - provider.size = () => 0 - useFakeVectorProvider(brain, provider) - await brain.init() // the VFS root bootstrap write sets `dimensions`, but size() still reports 0 - - const report = await brain.warm() - - expect(provider.searchCalls.length).toBe(0) // nothing probed - expect(report.vector.outcome).toBe('unavailable') - await brain.close() - }) - - it('(c) metadata + graph hydration paths actually execute against filesystem storage with pre-existing data', async () => { - const dir = mkTmp() - - // Build a brain with real data (including a field NOT in the metadata - // index's own common-fields warm subset — 'wave' — so hydrateAll()'s - // FULL hydration is distinguishable from init()'s partial warmCache()), - // and real graph edges, then close it (persisting everything). - const seed = new Brainy({ - requireSubtype: false, - storage: { type: 'filesystem', path: dir }, - silent: true - }) - await seed.init() - const ids: string[] = [] - for (let i = 0; i < 6; i++) { - ids.push( - await seed.add({ - data: `entity ${i}`, - type: NounType.Thing, - metadata: { wave: i % 3 }, - vector: V(i + 1) - }) - ) - } - for (let i = 0; i + 1 < ids.length; i++) { - await seed.relate({ from: ids[i], to: ids[i + 1], type: VerbType.RelatedTo }) - } - await seed.close() - - // Cold-reopen a FRESH instance and spy on the real hydration seams before - // calling warm(), so we assert they actually ran (not just that the - // report claims they did). - const metaHydrateCalls: number[] = [] - const loadedFields: string[] = [] - const graphInitCalls: number[] = [] - - const origHydrateAll = MetadataIndexManager.prototype.hydrateAll - const origLoadSparseIndex = (MetadataIndexManager.prototype as any).loadSparseIndex - const origGraphInit = GraphAdjacencyIndex.prototype.init - - MetadataIndexManager.prototype.hydrateAll = async function (...args: any[]) { - metaHydrateCalls.push(1) - return origHydrateAll.apply(this, args as any) - } - ;(MetadataIndexManager.prototype as any).loadSparseIndex = async function ( - field: string, - ...args: any[] - ) { - loadedFields.push(field) - return origLoadSparseIndex.apply(this, [field, ...args] as any) - } - GraphAdjacencyIndex.prototype.init = async function (...args: any[]) { - graphInitCalls.push(1) - return origGraphInit.apply(this, args as any) - } - - try { - const brain = new Brainy({ - requireSubtype: false, - storage: { type: 'filesystem', path: dir }, - silent: true - }) - await brain.init() - - const report = await brain.warm() - - expect(metaHydrateCalls.length).toBe(1) // hydrateAll() actually ran - expect(loadedFields).toContain('wave') // a NON-common field was loaded — full hydration, not the heuristic subset - expect(report.metadata.outcome).toBe('warmed') - - expect(graphInitCalls.length).toBeGreaterThanOrEqual(1) // graph's full-load seam ran (once at brain init, once via warm()) - expect(report.graph.outcome).toBe('warmed') - - // Correctness survives: the hydrated data still answers queries. - const byWhere = await brain.find({ type: NounType.Thing, where: { wave: 1 } }) - expect(byWhere.length).toBe(2) // waves 1,4 of 0..5 - - await brain.close() - } finally { - MetadataIndexManager.prototype.hydrateAll = origHydrateAll - ;(MetadataIndexManager.prototype as any).loadSparseIndex = origLoadSparseIndex - GraphAdjacencyIndex.prototype.init = origGraphInit - } - }) - - it('(d) warmOnOpen: true runs warm() during init() — observable via the fake provider', async () => { - const brain = new Brainy({ - requireSubtype: false, - storage: { type: 'memory' }, - warmOnOpen: true, - silent: true - }) - const provider = new FakeVectorProvider(true) - useFakeVectorProvider(brain, provider) - - // No explicit brain.warm() call — warmOnOpen must have run it as part of init(). - await brain.init() - - expect(provider.warmCalls).toBe(1) - await brain.close() - }) - - it('warmOnOpen defaults to false — init() does NOT run warm() unless opted in', async () => { - const brain = new Brainy({ - requireSubtype: false, - storage: { type: 'memory' }, - silent: true - }) - const provider = new FakeVectorProvider(true) - useFakeVectorProvider(brain, provider) - - await brain.init() - - expect(provider.warmCalls).toBe(0) - await brain.close() - }) - - it('(e) WarmReport shape: outcome literal + durationMs per surface, plus totalDurationMs', async () => { - const brain = new Brainy({ - requireSubtype: false, - storage: { type: 'memory' }, - silent: true - }) - const provider = new FakeVectorProvider(true) - useFakeVectorProvider(brain, provider) - await brain.init() - await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) }) - - const report = await brain.warm() - - for (const surface of ['vector', 'metadata', 'graph'] as const) { - expect(['warmed', 'probed', 'unavailable']).toContain(report[surface].outcome) - expect(typeof report[surface].durationMs).toBe('number') - expect(report[surface].durationMs).toBeGreaterThanOrEqual(0) - } - expect(typeof report.totalDurationMs).toBe('number') - expect(report.totalDurationMs).toBeGreaterThanOrEqual(0) - await brain.close() - }) -}) diff --git a/tests/unit/db/fact-log.test.ts b/tests/unit/db/fact-log.test.ts index f1c226cc..abce2dc9 100644 --- a/tests/unit/db/fact-log.test.ts +++ b/tests/unit/db/fact-log.test.ts @@ -186,52 +186,4 @@ describe('fact log — round-trip, framing, reconcile, rotation, scan', () => { await log.sync() expect(log.segmentPaths()).toEqual([]) // only a tail exists — nothing sealed }) - - describe('scanFacts liveness contract (Stage-2 D1)', () => { - it('a wedged store fails LOUDLY within the first-batch bound — never a silent hang', async () => { - // Force a sealed segment (tiny rotateBytes) so the scan must READ from - // storage, then wedge that read: the exact production shape (a - // backlogged brain whose segment read never returned). - const mem: any = new MemoryStorage() - await mem.init() - const wedgeable = new FactLog(mem, { rotateBytes: 1 }) - await wedgeable.open(0) - await wedgeable.append(fact(1)) - await wedgeable.append(fact(2)) // second append rotates → seg 1 sealed - await wedgeable.sync() - - const realRead = mem.readRawBytes.bind(mem) - mem.readRawBytes = (p: string) => - p.includes('facts/seg-') ? new Promise(() => {}) : realRead(p) // hangs forever - - const scan = wedgeable.scanFacts({ firstBatchTimeoutMs: 200 }) - const started = Date.now() - await expect(scan.batches().next()).rejects.toThrow(/no first batch within 200ms/) - expect(Date.now() - started).toBeLessThan(5_000) // bound held, not a hang - }) - - it('a healthy scan is unaffected — first batch well inside the bound, all facts delivered', async () => { - for (let g = 1; g <= 5; g++) await log.append(fact(g)) - await log.sync() - const scan = log.scanFacts({ batchSize: 2 }) - const all: CommitFact[] = [] - for await (const b of scan.batches()) all.push(...b.facts) - expect(all.map((f) => f.generation)).toEqual([1, 2, 3, 4, 5]) - expect(scan.summary().factsYielded).toBe(5) - }) - - it('consumer think-time between pulls never counts against the producer', async () => { - for (let g = 1; g <= 4; g++) await log.append(fact(g)) - await log.sync() - // Bound tighter than the consumer's pause: only the FIRST pull is - // raced, so a slow consumer after batch 1 must not trip the deadline. - const gen = log.scanFacts({ batchSize: 2, firstBatchTimeoutMs: 150 }).batches() - const first = await gen.next() - expect(first.done).toBe(false) - await new Promise((r) => setTimeout(r, 400)) // dawdle past the bound - const second = await gen.next() - expect(second.done).toBe(false) - expect((await gen.next()).done).toBe(true) - }) - }) }) diff --git a/tests/unit/db/generation-segments.test.ts b/tests/unit/db/generation-segments.test.ts deleted file mode 100644 index 27ab85cb..00000000 --- a/tests/unit/db/generation-segments.test.ts +++ /dev/null @@ -1,150 +0,0 @@ -/** - * @module tests/unit/db/generation-segments - * @description The generation-segment store (Stage-2 D1+D3 file format). - * Laws: (1) fold → read round-trips deltas and records byte-faithfully via - * sidecar point-reads; (2) the manifest is the ONLY discovery path — reopen - * reads one file, never a listing; (3) a lost/corrupt sidecar rebuilds from - * its segment loudly, a damaged SEGMENT fails loudly (never silent wrong - * data); (4) D3 reclaim drops whole segments only and bumps compactedBelow; - * (5) the packed digest is deterministic across reopen; (6) immutability — - * fold refuses overlap with sealed ranges. - */ -import { describe, it, expect, beforeEach } from 'vitest' -import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js' -import { - GenerationSegmentStore, - SEGMENTS_PREFIX, - type FoldGeneration -} from '../../../src/db/generationSegments.js' - -const UUID = (n: number): string => `00000000-0000-4000-8000-${String(n).padStart(12, '0')}` - -const gen = (g: number, recordCount = 2): FoldGeneration => ({ - generation: g, - timestamp: 1_700_000_000_000 + g, - delta: { generation: g, nouns: [UUID(g)], verbs: [], bytes: 123 + g }, - records: Array.from({ length: recordCount }, (_, i) => ({ - kind: (i % 2 === 0 ? 'noun' : 'verb') as 'noun' | 'verb', - id: UUID(g * 100 + i), - record: { metadata: { noun: 'document', v: g }, vector: { v: [g, i] } } - })) -}) - -describe('db/GenerationSegmentStore — the D1+D3 packed tier', () => { - let storage: MemoryStorage - let store: GenerationSegmentStore - - beforeEach(async () => { - storage = new MemoryStorage() - await storage.init() - store = new GenerationSegmentStore(storage as any) - await store.open() - }) - - it('fold → read round-trips deltas and records via sidecar point-reads', async () => { - const meta = await store.fold([gen(1), gen(2), gen(3)]) - expect(meta).toMatchObject({ firstGeneration: 1, lastGeneration: 3, frames: 3 }) - expect(meta.checksum).toBeGreaterThan(0) - - expect(store.hasGeneration(2)).toBe(true) - expect(store.hasGeneration(4)).toBe(false) - - const d2 = await store.readDelta(2) - expect(d2?.delta).toEqual({ generation: 2, nouns: [UUID(2)], verbs: [], bytes: 125 }) - expect(d2?.timestamp).toBe(1_700_000_000_002) - - const records = await store.readRecords(3) - expect(records).toHaveLength(2) - expect(records![0]).toEqual({ - kind: 'noun', - id: UUID(300), - record: { metadata: { noun: 'document', v: 3 }, vector: { v: [3, 0] } } - }) - // Point read by id, both kinds. - expect(await store.readRecord(3, 'verb', UUID(301))).toEqual({ - metadata: { noun: 'document', v: 3 }, - vector: { v: [3, 1] } - }) - expect(await store.readRecord(3, 'noun', UUID(999))).toBeNull() - }) - - it('reopen discovers everything from the manifest alone — no listing', async () => { - await store.fold([gen(1), gen(2)]) - await store.fold([gen(3), gen(4)]) - - const reopened = new GenerationSegmentStore(storage as any) - await reopened.open() - expect(reopened.segments()).toHaveLength(2) - expect(reopened.hasGeneration(4)).toBe(true) - expect((await reopened.readDelta(1))?.timestamp).toBe(1_700_000_000_001) - }) - - it('a lost sidecar rebuilds from its segment; a damaged segment fails LOUDLY', async () => { - const meta = await store.fold([gen(1), gen(2)]) - const idxPath = `${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.idx` - await storage.deleteRawObject(idxPath) - - const reopened = new GenerationSegmentStore(storage as any) - await reopened.open() - // Rebuild path: still serves correct data. - expect((await reopened.readRecords(2))!).toHaveLength(2) - - // Now damage the SEGMENT itself: flip a payload byte → CRC mismatch, loud. - const segPath = `${SEGMENTS_PREFIX}/${meta.file}` - const bytes = (await storage.readRawBytes(segPath))! - bytes[bytes.length - 3] ^= 0xff - await storage.writeRawBytes(segPath, bytes) - const damaged = new GenerationSegmentStore(storage as any) - await damaged.open() - ;(damaged as any).sidecars.clear() - await storage.deleteRawObject(idxPath) // force the sequential rebuild over damaged bytes - await expect(damaged.readRecords(2)).rejects.toThrow(/CRC mismatch|damaged/) - }) - - it('D3 reclaim drops whole segments only and bumps compactedBelow', async () => { - await store.fold([gen(1), gen(2)]) - await store.fold([gen(3), gen(4)]) - await store.fold([gen(5), gen(6)]) - - // Horizon mid-segment-2 (below 4): only segment 1 is FULLY below → drops. - const r1 = await store.dropSegmentsBelow(4) - expect(r1).toEqual({ dropped: 1, compactedBelow: 3 }) - expect(store.hasGeneration(1)).toBe(false) - expect(store.hasGeneration(3)).toBe(true) // partial segment survives whole - - // Bytes actually gone. - expect(await storage.readRawBytes(`${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.bgs`)).toBeNull() - - // Horizon past everything: the rest drop; compactedBelow is durable. - const r2 = await store.dropSegmentsBelow(7) - expect(r2.dropped).toBe(2) - const reopened = new GenerationSegmentStore(storage as any) - await reopened.open() - expect(reopened.compactedBelow()).toBe(7) - expect(reopened.segments()).toHaveLength(0) - }) - - it('the packed digest is deterministic across reopen and changes with history', async () => { - await store.fold([gen(1), gen(2), gen(3)]) - const atSeal = await store.digestThroughPacked(3) - const midSegment = await store.digestThroughPacked(2) - expect(atSeal).not.toBeNull() - expect(midSegment).not.toBeNull() - expect(midSegment).not.toBe(atSeal) - - const reopened = new GenerationSegmentStore(storage as any) - await reopened.open() - expect(await reopened.digestThroughPacked(3)).toBe(atSeal) - expect(await reopened.digestThroughPacked(2)).toBe(midSegment) - - await reopened.fold([gen(4)]) - expect(await reopened.digestThroughPacked(4)).not.toBe(atSeal) - }) - - it('sealed segments are immutable — fold refuses overlap, requires ascending input', async () => { - await store.fold([gen(1), gen(2)]) - await expect(store.fold([gen(2), gen(3)])).rejects.toThrow(/overlaps the packed tier/) - await expect(store.fold([gen(4), gen(4)])).rejects.toThrow(/strictly ascending/) - await expect(store.fold([])).rejects.toThrow(/at least one generation/) - }) -}) diff --git a/tests/unit/transaction/budget-commit-completed-work.test.ts b/tests/unit/transaction/budget-commit-completed-work.test.ts deleted file mode 100644 index a32489de..00000000 --- a/tests/unit/transaction/budget-commit-completed-work.test.ts +++ /dev/null @@ -1,149 +0,0 @@ -/** - * @module tests/unit/transaction/budget-commit-completed-work - * @description Regression coverage for the "commit-completed-work" transaction - * budget semantics: the budget gates STARTING the next operation; it never - * converts already-completed work into a rollback. Concretely: - * - * - A single-op transaction can never time out post-hoc — its one operation - * either runs (and the transaction commits, however long it took) or it - * never gets to start (which cannot happen: there is no operation before it - * to have overrun the budget). - * - A multi-op transaction whose op `i` overruns the budget stops op `i+1` - * from starting: everything applied so far rolls back atomically and a - * `TransactionTimeoutError` is thrown — the mid-batch zero-loss guarantee is - * unchanged. - * - `transactTimeoutBudget`'s scaling floor (`max(floorMs, opCount * 2000)`) - * is overridable per call, the seam `BrainyConfig.transactionBudgetFloorMs` - * feeds at the brainy.ts read sites. - * - * Uses a real class implementing the `Operation` interface (not an anonymous - * object literal, not a mock of Transaction internals) so the exercised path - * is exactly what a real caller's operation looks like. - */ -import { describe, it, expect } from 'vitest' -import { Transaction, transactTimeoutBudget } from '../../../src/transaction/Transaction.js' -import type { Operation, RollbackAction } from '../../../src/transaction/types.js' -import { TransactionTimeoutError } from '../../../src/transaction/errors.js' - -const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) - -/** - * A real `Operation` implementation that sleeps for `delayMs` before applying - * a write to `log`, and returns a rollback action that removes it again. Used - * to deterministically make one operation "slow" relative to a transaction's - * budget without touching any Transaction internals. - */ -class SlowOperation implements Operation { - readonly name: string - executed = false - rolledBack = false - - constructor( - private readonly log: string[], - label: string, - private readonly delayMs: number - ) { - this.name = label - } - - async execute(): Promise { - this.executed = true - if (this.delayMs > 0) await sleep(this.delayMs) - this.log.push(this.name) - return async () => { - this.rolledBack = true - const idx = this.log.indexOf(this.name) - if (idx >= 0) this.log.splice(idx, 1) - } - } -} - -describe('Transaction budget — commit-completed-work semantics', () => { - it('(a) single-op transaction whose op overruns the budget COMMITS — no rollback, no throw', async () => { - const log: string[] = [] - const op = new SlowOperation(log, 'slow-single-op', 40) - - // Budget (5ms) is far smaller than the op's 40ms — under the OLD - // (post-completion-check) semantics this would have thrown and rolled - // back after the op finished. Under the new semantics there is no - // operation after it to gate, so it commits. - const tx = new Transaction({ timeout: 5 }) - tx.addOperation(op) - - await expect(tx.execute()).resolves.toBeUndefined() - - expect(tx.getState()).toBe('committed') - expect(op.executed).toBe(true) - expect(op.rolledBack).toBe(false) - expect(log).toEqual(['slow-single-op']) - }) - - it('(b) two-op transaction: op0 overruns the budget → op1 never starts, op0 rolls back, throws TransactionTimeoutError', async () => { - const log: string[] = [] - const op0 = new SlowOperation(log, 'op0-overruns', 40) - const op1 = new SlowOperation(log, 'op1-never-starts', 0) - - const tx = new Transaction({ timeout: 5 }) - tx.addOperation(op0) - tx.addOperation(op1) - - const error = await tx.execute().catch((e) => e) - - expect(error).toBeInstanceOf(TransactionTimeoutError) - expect(op0.executed).toBe(true) - expect(op0.rolledBack).toBe(true) - expect(op1.executed).toBe(false) - expect(op1.rolledBack).toBe(false) - expect(log).toEqual([]) // op0's write was undone; op1 never wrote - expect(tx.getState()).toBe('rolled_back') - }) - - it('a single-op transaction never checks the budget before its only operation starts', async () => { - // Budget of 0ms: under the old "check before every op including 0" code - // this could theoretically trip before op 0 even started (if any - // measurable time elapsed between startTime capture and the check). The - // documented contract is stronger: operation 0 ALWAYS gets to start. - const log: string[] = [] - const op = new SlowOperation(log, 'only-op', 5) - - const tx = new Transaction({ timeout: 0 }) - tx.addOperation(op) - - await expect(tx.execute()).resolves.toBeUndefined() - expect(tx.getState()).toBe('committed') - expect(op.executed).toBe(true) - }) - - it('(c) transactTimeoutBudget: an explicit floorMs overrides the 30s default floor', () => { - // Below the per-op scaling term, a raised floor wins. - expect(transactTimeoutBudget(1, undefined, 5_000)).toBe(5_000) // 1 op × 2000 = 2000 < 5000 floor - expect(transactTimeoutBudget(1)).toBe(30_000) // unchanged default when floorMs omitted - - // Above the floor, opCount * 2000 still wins over a smaller floor. - expect(transactTimeoutBudget(10, undefined, 5_000)).toBe(20_000) // 10 × 2000 = 20000 > 5000 floor - - // A full `override` always wins, regardless of floorMs. - expect(transactTimeoutBudget(10, 999, 5_000)).toBe(999) - }) - - it('(c) a configured floor changes real Transaction behavior end-to-end via TransactionManager-style options', async () => { - // Simulates how brainy.ts wires `this.config.transactionBudgetFloorMs` - // into `transactTimeoutBudget()`: opCount 0 isolates the floor term - // (0 × 2000 = 0), so a lowered floor makes a normally-generous budget - // fail-fast for a real 2-op Transaction. - const lowFloorBudget = transactTimeoutBudget(0, undefined, 10) // 10ms floor - expect(lowFloorBudget).toBe(10) - - const log: string[] = [] - const op0 = new SlowOperation(log, 'op0', 30) - const op1 = new SlowOperation(log, 'op1-gated-by-lowered-floor', 0) - - const tx = new Transaction({ timeout: lowFloorBudget }) - tx.addOperation(op0) - tx.addOperation(op1) - - const error = await tx.execute().catch((e) => e) - expect(error).toBeInstanceOf(TransactionTimeoutError) - expect(op1.executed).toBe(false) - }) -}) diff --git a/tests/unit/transaction/vectorIndexOperations-rename.test.ts b/tests/unit/transaction/vectorIndexOperations-rename.test.ts deleted file mode 100644 index bdfe58a3..00000000 --- a/tests/unit/transaction/vectorIndexOperations-rename.test.ts +++ /dev/null @@ -1,176 +0,0 @@ -/** - * @module tests/unit/transaction/vectorIndexOperations-rename - * @description Coverage for the backend-neutral vector-index transaction - * operations (formerly `AddToHNSWOperation` / `RemoveFromHNSWOperation`). - * These op-name strings surface directly in consumer-visible transaction - * journals and timings — a fossil "HNSW" name misdirects an operator running - * a native (non-HNSW) vector provider. Verifies: - * 1. rollback wiring stayed byte-identical under the rename, - * 2. the emitted `name` stamps the active backend — `'hnsw-js'` for - * Brainy's own JS index, a provider's own required `name` when it - * self-identifies, and the tolerant-loud `'unknown-provider'` literal - * (plus one console.warn) when a runtime instance lacks `name` - * altogether (an older native provider compiled against the previous, - * optional `providerId` contract) — never a silently-wrong guess. - */ -import { describe, it, expect, vi, afterEach } from 'vitest' -import { - AddToVectorIndexOperation, - RemoveFromVectorIndexOperation, - BatchAddToVectorIndexOperation -} from '../../../src/transaction/operations/IndexOperations.js' -import type { VectorIndexProvider } from '../../../src/plugin.js' -import type { VectorDocument, Vector } from '../../../src/coreTypes.js' -import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js' - -/** - * A minimal, real (not mocked) VectorIndexProvider implementation backed by - * a plain Map — exercises the exact contract the operations call through, - * without any Brainy internals. - */ -class FakeVectorProvider implements VectorIndexProvider { - readonly items = new Map() - constructor(readonly name: string) {} - - async addItem(item: VectorDocument): Promise { - this.items.set(item.id, item.vector) - return item.id - } - async removeItem(id: string): Promise { - return this.items.delete(id) - } - async getItem(id: string): Promise { - const vector = this.items.get(id) - return vector ? { id, vector } : undefined - } - async search(): Promise> { - return [] - } - size(): number { - return this.items.size - } - clear(): void { - this.items.clear() - } - async rebuild(): Promise {} - async flush(): Promise { - return 0 - } - getPersistMode(): 'immediate' | 'deferred' { - return 'immediate' - } -} - -describe('Vector index transaction operations — backend-neutral rename', () => { - describe('rollback wiring (byte-identical behavior under the rename)', () => { - it('AddToVectorIndexOperation rollback removes a newly-added item', async () => { - const provider = new FakeVectorProvider('fake-provider') - const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3]) - - const rollback = await op.execute() - expect(provider.items.has('id-1')).toBe(true) - - await rollback() - expect(provider.items.has('id-1')).toBe(false) - }) - - it('AddToVectorIndexOperation rollback is a no-op when the item pre-existed (update semantics)', async () => { - const provider = new FakeVectorProvider('fake-provider') - await provider.addItem({ id: 'id-1', vector: [9, 9, 9] }) - - const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3]) - const rollback = await op.execute() - expect(provider.items.get('id-1')).toEqual([1, 2, 3]) - - await rollback() - // Pre-existing item is NOT removed by rollback — it stays (the update - // itself is not undone by this op; that's RemoveFromVectorIndexOperation's job). - expect(provider.items.has('id-1')).toBe(true) - }) - - it('RemoveFromVectorIndexOperation rollback re-adds the item with its original vector', async () => { - const provider = new FakeVectorProvider('fake-provider') - await provider.addItem({ id: 'id-1', vector: [4, 5, 6] }) - - const op = new RemoveFromVectorIndexOperation(provider, 'id-1', [4, 5, 6]) - const rollback = await op.execute() - expect(provider.items.has('id-1')).toBe(false) - - await rollback() - expect(provider.items.get('id-1')).toEqual([4, 5, 6]) - }) - - it('BatchAddToVectorIndexOperation rolls back every item in reverse order on undo', async () => { - const provider = new FakeVectorProvider('fake-provider') - const op = new BatchAddToVectorIndexOperation(provider, [ - { id: 'a', vector: [1] }, - { id: 'b', vector: [2] }, - { id: 'c', vector: [3] } - ]) - - const rollback = await op.execute() - expect([...provider.items.keys()].sort()).toEqual(['a', 'b', 'c']) - - await rollback() - expect(provider.items.size).toBe(0) - }) - }) - - describe('backend stamping in the emitted op name', () => { - afterEach(() => { - vi.restoreAllMocks() - }) - - it('stamps the real JS HNSW index as hnsw-js (self-identifying name)', () => { - const index = new JsHnswVectorIndex() - const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3]) - const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3]) - - expect(addOp.name).toBe('AddToVectorIndex(hnsw-js)') - expect(removeOp.name).toBe('RemoveFromVectorIndex(hnsw-js)') - }) - - it('stamps a self-identifying provider\'s own name, never "HNSW"', () => { - const provider = new FakeVectorProvider('acme-diskann') - const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3]) - const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3]) - const batchOp = new BatchAddToVectorIndexOperation(provider, [{ id: 'a', vector: [1] }]) - - expect(addOp.name).toBe('AddToVectorIndex(acme-diskann)') - expect(removeOp.name).toBe('RemoveFromVectorIndex(acme-diskann)') - expect(batchOp.name).toBe('BatchAddToVectorIndex(acme-diskann)') - expect(addOp.name).not.toContain('HNSW') - expect(removeOp.name).not.toContain('HNSW') - }) - - it('tolerant-loud: stamps "unknown-provider" and warns exactly once when a runtime provider lacks the required `name` (an older native provider compiled against the previous optional `providerId` contract)', () => { - const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}) - // Simulate a native provider compiled before `name` was required — the - // TypeScript contract requires `name`, but `as unknown as` mirrors what - // actually reaches this code at runtime from an un-rebuilt native addon. - const legacyProvider = { - addItem: async (item: VectorDocument) => item.id, - removeItem: async () => true, - search: async () => [], - size: () => 0, - clear: () => {}, - rebuild: async () => {}, - flush: async () => 0, - getPersistMode: () => 'immediate' as const - } as unknown as VectorIndexProvider - - const addOp = new AddToVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3]) - const removeOp = new RemoveFromVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3]) - - expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)') - expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)') - // Never silently falls back to the JS engine's name for a provider it - // knows nothing about — that's the exact fossil-naming bug being fixed. - expect(addOp.name).not.toContain('hnsw-js') - // Loud, not silent — but exactly once per provider instance, not once - // per op stamped against it. - expect(warnSpy).toHaveBeenCalledTimes(1) - expect(warnSpy.mock.calls[0][0]).toContain('name') - }) - }) -})