brainy/tests/integration/rev-and-ifabsent.test.ts

292 lines
11 KiB
TypeScript
Raw Normal View History

feat: per-entity _rev + update({ ifRev }) CAS + add({ ifAbsent }) Optimistic concurrency for multi-writer coordination — the read-then-CAS lock pattern + idempotent-bootstrap singleton inserts. Lands the surface the SDK scheduler asked for in BRAINY-EXPOSE-TRANSACTIONS, scoped to what ships cleanly without painting the 8.0 transact() API into a corner. A. PER-ENTITY _rev FIELD - src/coreTypes.ts — HNSWNounWithMetadata gains optional `_rev?: number`; STANDARD_ENTITY_FIELDS includes it so resolveEntityField routes correctly. - src/types/brainy.types.ts — Entity<T> gains optional `_rev?: number`; Result<T> mirrors it on the convenience flatten layer. - src/brainy.ts add() — initializes `_rev: 1` in storageMetadata. New entities always start at 1. - src/brainy.ts update() — reads currentRev off the persisted metadata (falls back to 1 for pre-7.31.0 entities with no _rev), writes `_rev: currentRev + 1` into the updated metadata. Every successful update() bumps by exactly 1. - src/brainy.ts convertNounToEntity + convertMetadataToEntity — both pull _rev out of the storage metadata and surface it at the top level of Entity. Pre-7.31.0 entities (no _rev in storage) read as `_rev: 1` so consumers see a consistent value. - src/storage/baseStorage.ts — six destructure sites updated to pull _rev out of the metadata bag so it doesn't leak into customMetadata. Noun-side returns surface _rev; verb-side destructures correctly but doesn't expose it on HNSWVerbWithMetadata (verb CAS is future work). - src/brainy.ts createResult() — flattens entity._rev onto the Result for backward compat with the existing convenience-field layer. B. update({ ifRev }) OPTIMISTIC CONCURRENCY - src/types/brainy.types.ts — UpdateParams<T> gains optional `ifRev?: number`. - src/transaction/RevisionConflictError.ts (NEW) — carries `{ id, expected, actual }`. Message names the recipe: refetch with brain.get() and retry with the latest _rev. Subclass of Error. - src/brainy.ts update() — when params.ifRev is provided, compares against currentRev (the rev we just read) and throws RevisionConflictError on mismatch before any storage write. Omitting ifRev keeps the prior unconditional-update behavior; existing consumers see no change. - src/transaction/index.ts — exports RevisionConflictError. - src/index.ts — public export of RevisionConflictError. C. add({ ifAbsent }) BY-ID IDEMPOTENT INSERT - src/types/brainy.types.ts — AddParams<T> gains optional `ifAbsent?: boolean`; AddManyParams<T> mirrors it as a batch-level flag. - src/brainy.ts add() — when params.id AND params.ifAbsent, pre-reads storage.getNounMetadata(id); if present, returns the existing id without writing. No throw, no overwrite. Ignored when id is omitted (a fresh UUID can never collide). - src/brainy.ts addMany() — propagates the batch-level ifAbsent to each item's add() call. Per-item ifAbsent takes precedence so callers can override individual rows. WHAT'S NOT SHIPPED (and why) A public brain.transaction(fn) wrapper was on the table but was cut. The internal TransactionManager exposes raw Operation classes (SaveNounMetadata, etc.) that take StorageAdapter as a constructor argument. A clean high-level facade in 7.31.0 would have meant either: (a) Delegate to brain.add() / update() / relate(). Each of those opens its own internal transaction and commits before the closure returns. Looks atomic, isn't — a footgun. (b) Thread an optional `tx?` through every internal write site in brainy.ts (8 transactionManager.executeTransaction sites). ~2 days of real refactor with regression surface, and the API shape changes again in 8.0 anyway. The SDK scheduler's actual ask (BRAINY-EXPOSE-TRANSACTIONS) is the read-then-CAS lock pattern. _rev + ifRev solves it completely. Multi-write atomicity is the 8.0 brain.transact() use case where the Datomic-style immutable Db makes it atomic by construction — that's the right home. TESTS - New tests/integration/rev-and-ifabsent.test.ts (18 tests): - _rev initialization (1 on add) and surface on get fast/full paths + find - _rev auto-bump on update across multiple writes - update({ ifRev }) pass / fail / message format / omitted / legacy-no-rev - add({ ifAbsent }) writes when absent / no-op when present / id-required - addMany({ ifAbsent }) propagation + per-item override - SDK-scheduler scenario: two concurrent CAS updates, one wins one throws - Unit suite unchanged: 1468/1468. - All integration subtype + verb + strict + find-limits + new rev suites pass: 96/96. DOCS - New docs/guides/optimistic-concurrency.md (public: true) — full reference: the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev interacts with brain.versions, branches/fork, and VFS, what's coming in 8.0. - docs/api/README.md — add() and update() entries get the new params + tips pointing at the new guide. - RELEASES.md v7.31.0 entry. CORTEX COMPATIBILITY Zero changes required. _rev is a metadata column already supported by NativeColumnStore. The auto-bump runs in Brainy JS before any storage call; Cortex never sees the per-entity counter. 8.0 FORWARD-COMPAT _rev, ifRev, RevisionConflictError, and ifAbsent survive the 8.0 Db redesign unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record patterns (job locks, idempotent state machines), generation-based for "did the world move under me." Locked in .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md § C-6. Verification - npx tsc --noEmit: clean - npm test: 1468 / 1468 unit - All integration suites including new rev-and-ifabsent: 96/96 - npm run build: clean - Closed-source product reference audit: clean
2026-06-09 10:04:24 -07:00
/**
* @module tests/integration/rev-and-ifabsent
* @description End-to-end tests for the 7.31.0 optimistic-concurrency surface:
* - `_rev` is initialized to 1 on add() and surfaced on get()/find()/search()
* - update() auto-bumps `_rev`
* - update({ ifRev }) accepts a matching rev and rejects a stale one with
* RevisionConflictError carrying { id, expected, actual }
* - add({ ifAbsent: true }) is a no-op when the id is already present
* - addMany({ ifAbsent: true }) applies the flag to every item
*/
import { describe, it, expect, beforeEach } from 'vitest'
import { Brainy } from '../../src/brainy.js'
import { RevisionConflictError } from '../../src/transaction/RevisionConflictError.js'
import { NounType } from '../../src/types/graphTypes.js'
describe('7.31.0 — _rev CAS + ifAbsent', () => {
let brain: Brainy
beforeEach(async () => {
feat(8.0)!: flip requireSubtype default to true (BRAINY-8.0-SUBTYPE-CONTRACT § C-1) Brainy 8.0 makes subtype required by default on every public write path (`add`, `addMany`, `update`, `relate`, `relateMany`, `updateRelation`, import). Per the locked C-1 contract, every entity and relation gets a non-empty subtype string by the time the storage layer sees it. OPT-OUT REMAINS FULLY SUPPORTED The runtime flag is still consumer-controlled. Three opt-out paths cover migration / legacy fixtures / typed escape: - `new Brainy({ requireSubtype: false })` — last-resort: turn off the contract entirely. Recommended only for migration windows or test fixtures that legitimately can't supply a subtype. - `new Brainy({ requireSubtype: { except: [NounType.Thing, ...] } })` — per-type allowlist: strict everywhere except the listed types. - `brain.requireSubtype(type, options)` — per-type registration with optional vocabulary. Composes with the brain-wide flag. Default is now `true`. Opt-out is explicit and documented; nothing silently degrades. TEST SWEEP Bulk-applied `requireSubtype: false` to every `new Brainy({...})` call site across 120 test files. Three sed patterns covered the shapes: - `new Brainy({` → `new Brainy({ requireSubtype: false,` - `new Brainy<T>({` → `new Brainy<T>({ requireSubtype: false,` - `new Brainy()` → `new Brainy({ requireSubtype: false })` tests/helpers/test-factory.ts → createTestConfig() defaults `requireSubtype: false` so test files using the helper inherit the opt-out without per-site edits. The test sites that DO exercise subtype semantics (the subtype-and-facets suite, the strict-mode-self-test suite, the verb- subtype-and-enforcement suite, etc.) already pass real subtypes — they were the 7.30.x acceptance tests for this contract. Those tests continue to pass unchanged. CHANGES src/brainy.ts - normalizeConfig() — `requireSubtype` default `false` → `true`. Comment refreshed to document the three opt-out paths. tests/* (120 files) - Bulk-edited brain construction sites. No functional test changes; the opt-out preserves the test author's original intent. tests/helpers/test-factory.ts - createTestConfig() base config gains `requireSubtype: false`. NO-OP for consumers who were already passing subtype on every write. For consumers who weren't, the upgrade path is one of the three opt-out forms above. Migration recipe documented in 8.0 release notes (next commit). VERIFICATION - npx tsc --noEmit: clean - npm test: 1408 / 1409 (same pre-existing race-condition outstanding; no other regressions from the flip)
2026-06-09 14:58:25 -07:00
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' as const } })
feat: per-entity _rev + update({ ifRev }) CAS + add({ ifAbsent }) Optimistic concurrency for multi-writer coordination — the read-then-CAS lock pattern + idempotent-bootstrap singleton inserts. Lands the surface the SDK scheduler asked for in BRAINY-EXPOSE-TRANSACTIONS, scoped to what ships cleanly without painting the 8.0 transact() API into a corner. A. PER-ENTITY _rev FIELD - src/coreTypes.ts — HNSWNounWithMetadata gains optional `_rev?: number`; STANDARD_ENTITY_FIELDS includes it so resolveEntityField routes correctly. - src/types/brainy.types.ts — Entity<T> gains optional `_rev?: number`; Result<T> mirrors it on the convenience flatten layer. - src/brainy.ts add() — initializes `_rev: 1` in storageMetadata. New entities always start at 1. - src/brainy.ts update() — reads currentRev off the persisted metadata (falls back to 1 for pre-7.31.0 entities with no _rev), writes `_rev: currentRev + 1` into the updated metadata. Every successful update() bumps by exactly 1. - src/brainy.ts convertNounToEntity + convertMetadataToEntity — both pull _rev out of the storage metadata and surface it at the top level of Entity. Pre-7.31.0 entities (no _rev in storage) read as `_rev: 1` so consumers see a consistent value. - src/storage/baseStorage.ts — six destructure sites updated to pull _rev out of the metadata bag so it doesn't leak into customMetadata. Noun-side returns surface _rev; verb-side destructures correctly but doesn't expose it on HNSWVerbWithMetadata (verb CAS is future work). - src/brainy.ts createResult() — flattens entity._rev onto the Result for backward compat with the existing convenience-field layer. B. update({ ifRev }) OPTIMISTIC CONCURRENCY - src/types/brainy.types.ts — UpdateParams<T> gains optional `ifRev?: number`. - src/transaction/RevisionConflictError.ts (NEW) — carries `{ id, expected, actual }`. Message names the recipe: refetch with brain.get() and retry with the latest _rev. Subclass of Error. - src/brainy.ts update() — when params.ifRev is provided, compares against currentRev (the rev we just read) and throws RevisionConflictError on mismatch before any storage write. Omitting ifRev keeps the prior unconditional-update behavior; existing consumers see no change. - src/transaction/index.ts — exports RevisionConflictError. - src/index.ts — public export of RevisionConflictError. C. add({ ifAbsent }) BY-ID IDEMPOTENT INSERT - src/types/brainy.types.ts — AddParams<T> gains optional `ifAbsent?: boolean`; AddManyParams<T> mirrors it as a batch-level flag. - src/brainy.ts add() — when params.id AND params.ifAbsent, pre-reads storage.getNounMetadata(id); if present, returns the existing id without writing. No throw, no overwrite. Ignored when id is omitted (a fresh UUID can never collide). - src/brainy.ts addMany() — propagates the batch-level ifAbsent to each item's add() call. Per-item ifAbsent takes precedence so callers can override individual rows. WHAT'S NOT SHIPPED (and why) A public brain.transaction(fn) wrapper was on the table but was cut. The internal TransactionManager exposes raw Operation classes (SaveNounMetadata, etc.) that take StorageAdapter as a constructor argument. A clean high-level facade in 7.31.0 would have meant either: (a) Delegate to brain.add() / update() / relate(). Each of those opens its own internal transaction and commits before the closure returns. Looks atomic, isn't — a footgun. (b) Thread an optional `tx?` through every internal write site in brainy.ts (8 transactionManager.executeTransaction sites). ~2 days of real refactor with regression surface, and the API shape changes again in 8.0 anyway. The SDK scheduler's actual ask (BRAINY-EXPOSE-TRANSACTIONS) is the read-then-CAS lock pattern. _rev + ifRev solves it completely. Multi-write atomicity is the 8.0 brain.transact() use case where the Datomic-style immutable Db makes it atomic by construction — that's the right home. TESTS - New tests/integration/rev-and-ifabsent.test.ts (18 tests): - _rev initialization (1 on add) and surface on get fast/full paths + find - _rev auto-bump on update across multiple writes - update({ ifRev }) pass / fail / message format / omitted / legacy-no-rev - add({ ifAbsent }) writes when absent / no-op when present / id-required - addMany({ ifAbsent }) propagation + per-item override - SDK-scheduler scenario: two concurrent CAS updates, one wins one throws - Unit suite unchanged: 1468/1468. - All integration subtype + verb + strict + find-limits + new rev suites pass: 96/96. DOCS - New docs/guides/optimistic-concurrency.md (public: true) — full reference: the lock pattern, read-modify-write retry, idempotent bootstrap, how _rev interacts with brain.versions, branches/fork, and VFS, what's coming in 8.0. - docs/api/README.md — add() and update() entries get the new params + tips pointing at the new guide. - RELEASES.md v7.31.0 entry. CORTEX COMPATIBILITY Zero changes required. _rev is a metadata column already supported by NativeColumnStore. The auto-bump runs in Brainy JS before any storage call; Cortex never sees the per-entity counter. 8.0 FORWARD-COMPAT _rev, ifRev, RevisionConflictError, and ifAbsent survive the 8.0 Db redesign unchanged. 8.0 layers brain.transact(tx, { ifAtGeneration }) for whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record patterns (job locks, idempotent state machines), generation-based for "did the world move under me." Locked in .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md § C-6. Verification - npx tsc --noEmit: clean - npm test: 1468 / 1468 unit - All integration suites including new rev-and-ifabsent: 96/96 - npm run build: clean - Closed-source product reference audit: clean
2026-06-09 10:04:24 -07:00
await brain.init()
})
describe('_rev initialization + surface', () => {
it('initializes _rev to 1 on add()', async () => {
const id = await brain.add({ data: 'hello', type: NounType.Document })
const entity = await brain.get(id)
expect(entity?._rev).toBe(1)
})
it('surfaces _rev on metadata-only get() fast path', async () => {
const id = await brain.add({ data: 'fast-path', type: NounType.Document })
const entity = await brain.get(id) // metadata-only by default
expect(entity?._rev).toBe(1)
})
it('surfaces _rev on includeVectors get() full path', async () => {
const id = await brain.add({ data: 'full-path', type: NounType.Document })
const entity = await brain.get(id, { includeVectors: true })
expect(entity?._rev).toBe(1)
})
it('surfaces _rev on find() results', async () => {
const id = await brain.add({ data: 'findable', type: NounType.Document })
const results = await brain.find({ type: NounType.Document })
const hit = results.find((r) => r.id === id)
expect(hit?._rev).toBe(1)
})
})
describe('_rev auto-bump on update()', () => {
it('bumps _rev to 2 after first update()', async () => {
const id = await brain.add({ data: 'v1', type: NounType.Document })
await brain.update({ id, data: 'v2' })
const entity = await brain.get(id)
expect(entity?._rev).toBe(2)
})
it('bumps _rev monotonically across multiple updates', async () => {
const id = await brain.add({ data: 'v1', type: NounType.Document })
for (let i = 2; i <= 5; i++) {
await brain.update({ id, data: `v${i}` })
const entity = await brain.get(id)
expect(entity?._rev).toBe(i)
}
})
it('bumps _rev on metadata-only update', async () => {
const id = await brain.add({
data: 'doc',
type: NounType.Document,
metadata: { status: 'draft' }
})
await brain.update({ id, metadata: { status: 'published' } })
const entity = await brain.get(id)
expect(entity?._rev).toBe(2)
expect(entity?.metadata?.status).toBe('published')
})
})
describe('update({ ifRev }) optimistic concurrency', () => {
it('passes when ifRev matches the persisted _rev', async () => {
const id = await brain.add({ data: 'v1', type: NounType.Document })
await expect(brain.update({ id, data: 'v2', ifRev: 1 })).resolves.not.toThrow()
const after = await brain.get(id)
expect(after?._rev).toBe(2)
})
it('throws RevisionConflictError when ifRev is stale', async () => {
const id = await brain.add({ data: 'v1', type: NounType.Document })
await brain.update({ id, data: 'v2' }) // bumps to 2
let caught: unknown = null
try {
await brain.update({ id, data: 'v3-from-stale-reader', ifRev: 1 })
} catch (err) {
caught = err
}
expect(caught).toBeInstanceOf(RevisionConflictError)
const e = caught as RevisionConflictError
expect(e.id).toBe(id)
expect(e.expected).toBe(1)
expect(e.actual).toBe(2)
})
it('emits a message that points at the recovery recipe', async () => {
const id = await brain.add({ data: 'v1', type: NounType.Document })
await brain.update({ id, data: 'v2' })
try {
await brain.update({ id, ifRev: 1, data: 'v3' })
throw new Error('should have thrown')
} catch (err) {
const msg = (err as Error).message
expect(msg).toMatch(/persisted _rev is 2/)
expect(msg).toMatch(/brain\.get/)
}
})
it('omits the check entirely when ifRev is not provided', async () => {
const id = await brain.add({ data: 'v1', type: NounType.Document })
await brain.update({ id, data: 'v2' })
await brain.update({ id, data: 'v3' }) // no ifRev — must succeed
const after = await brain.get(id)
expect(after?._rev).toBe(3)
})
it('treats pre-7.31.0 metadata with no _rev as rev 1', async () => {
// Simulate a legacy entity by writing directly through the storage adapter
// with no _rev field, then verifying ifRev: 1 succeeds.
const id = 'aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa'
const storage = (brain as any).storage
await storage.saveNounMetadata(id, {
data: 'pre-7.31',
noun: NounType.Document,
createdAt: Date.now(),
updatedAt: Date.now()
// intentionally no _rev
})
// Also save the vector so get() doesn't return null on the full path.
const vector = await (brain as any).embed('pre-7.31')
await storage.saveNoun({
id,
vector,
connections: new Map(),
level: 0
})
const legacy = await brain.get(id)
expect(legacy?._rev).toBe(1)
await expect(brain.update({ id, data: 'updated', ifRev: 1 })).resolves.not.toThrow()
const after = await brain.get(id)
expect(after?._rev).toBe(2)
})
})
describe('add({ ifAbsent: true })', () => {
it('writes when no entity with the id exists', async () => {
const id = await brain.add({
id: '11111111-1111-4111-8111-111111111111',
data: 'first-write',
type: NounType.Document,
ifAbsent: true
})
expect(id).toBe('11111111-1111-4111-8111-111111111111')
const entity = await brain.get(id)
expect(entity?.data).toBe('first-write')
expect(entity?._rev).toBe(1)
})
it('returns the existing id without writing when the id is taken', async () => {
await brain.add({
id: '22222222-2222-4222-8222-222222222222',
data: 'original',
type: NounType.Document
})
const id = await brain.add({
id: '22222222-2222-4222-8222-222222222222',
data: 'attempted-overwrite',
type: NounType.Document,
ifAbsent: true
})
expect(id).toBe('22222222-2222-4222-8222-222222222222')
const entity = await brain.get(id)
expect(entity?.data).toBe('original') // not overwritten
expect(entity?._rev).toBe(1) // not bumped (no write happened)
})
it('is a no-op (writes anyway) when no id is supplied', async () => {
// ifAbsent without id can't mean anything — a fresh UUID can't collide.
// The contract says "ignored when id is omitted." Verify it writes.
const id = await brain.add({
data: 'no-id-uuid-generated',
type: NounType.Document,
ifAbsent: true
})
expect(id).toMatch(/^[0-9a-f]{8}-/) // UUID v4
const entity = await brain.get(id)
expect(entity?.data).toBe('no-id-uuid-generated')
})
})
describe('addMany({ ifAbsent: true })', () => {
it('applies ifAbsent to every item', async () => {
// Seed one entity that should NOT be overwritten by the batch.
await brain.add({
id: '33333333-3333-4333-8333-333333333333',
data: 'original',
type: NounType.Document
})
const result = await brain.addMany({
ifAbsent: true,
items: [
{ id: '33333333-3333-4333-8333-333333333333', data: 'should-not-overwrite', type: NounType.Document },
{ id: '44444444-4444-4444-8444-444444444444', data: 'new', type: NounType.Document },
{ id: '55555555-5555-4555-8555-555555555555', data: 'new', type: NounType.Document }
]
})
expect(result.successful).toContain('33333333-3333-4333-8333-333333333333')
expect(result.successful).toContain('44444444-4444-4444-8444-444444444444')
expect(result.successful).toContain('55555555-5555-4555-8555-555555555555')
const seeded = await brain.get('33333333-3333-4333-8333-333333333333')
expect(seeded?.data).toBe('original')
const fresh1 = await brain.get('44444444-4444-4444-8444-444444444444')
expect(fresh1?.data).toBe('new')
})
it('per-item ifAbsent overrides the batch flag', async () => {
await brain.add({
id: '77777777-7777-4777-8777-777777777777',
data: 'keep',
type: NounType.Document
})
// Batch flag is false (default), but one item opts in.
const result = await brain.addMany({
items: [
{ id: '77777777-7777-4777-8777-777777777777', data: 'overwrite-attempt', type: NounType.Document, ifAbsent: true },
{ id: '66666666-6666-4666-8666-666666666666', data: 'new', type: NounType.Document }
]
})
expect(result.successful).toContain('77777777-7777-4777-8777-777777777777')
const protectedEntity = await brain.get('77777777-7777-4777-8777-777777777777')
expect(protectedEntity?.data).toBe('keep')
})
})
describe('SDK scheduler use case — read-then-CAS lock pattern', () => {
it('two concurrent CAS updates: one wins, one throws RevisionConflictError', async () => {
const lockId = 'bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
await brain.add({
id: lockId,
data: { owner: null, expiresAt: 0 },
type: NounType.Document,
metadata: { kind: 'job-state' },
ifAbsent: true
})
// Two readers see the same _rev
const a = await brain.get(lockId)
const b = await brain.get(lockId)
expect(a?._rev).toBe(b?._rev)
// A wins the race
await brain.update({
id: lockId,
data: { owner: 'worker-A', expiresAt: Date.now() + 300_000 },
ifRev: a!._rev
})
// B's CAS now fails because the rev has moved
let caught: unknown = null
try {
await brain.update({
id: lockId,
data: { owner: 'worker-B', expiresAt: Date.now() + 300_000 },
ifRev: b!._rev
})
} catch (err) {
caught = err
}
expect(caught).toBeInstanceOf(RevisionConflictError)
// Final state: A owns the lock
const final = await brain.get(lockId)
expect((final?.data as any)?.owner).toBe('worker-A')
})
})
})