soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: atomic ifAbsent/upsert inserts + exact blob reference counts under concurrency

Browse source 
The fast-follow to the ifRev CAS fix: the sweep for the same check-then-act
class found two more instances, both now closed with the same discipline —
the decision runs at the serialization point that guards the apply.

add({ifAbsent}) / add({upsert}): the absence check ran before the commit
mutex, so N concurrent same-id creates could all pass it and all write — the
second silently overwriting the first, violating ifAbsent's "return the
existing id WITHOUT writing" contract and upsert's merge-never-clobber
contract. The insert leg now carries a must-be-absent commit precondition
(the same conditional-commit primitive ifRev uses), verified under the commit
mutex against the authoritative before-image. A losing caller takes its
documented resolution instead of overwriting: ifAbsent returns the existing
id with zero writes; upsert merges into the now-existing entity via update()
(shared param mapping in upsertMergeParams so the planning-time branch and
the conflict retry can never drift), with a bounded retry if a concurrent
delete lands between the conflict and the merge. The planning-time checks
remain as fast-fails. transact()'s batch-level ifAbsent/upsert keep
planning-time semantics (documented, converges to a valid entity).

BlobStorage: write()'s dedup decision (exists → increment refCount, absent →
create at 1) and delete()'s decrement-then-remove were unserialized
read-modify-writes over blob-meta:<hash>. Concurrent writes of identical
content could lose references, so a later delete removed bytes another file
still referenced (data loss), or leaked unreferenced blobs. All
reference-count-bearing mutations now serialize through a per-hash
InMemoryMutex — distinct content never contends; incrementRefCount/
decrementRefCount are documented lock-assumed internals.

Both fixes are complete by construction in-process: storage enforces
single-writer-per-directory, so the process is the whole concurrency domain.

Tests (tests/integration/ifabsent-upsert-blob-concurrency.test.ts): 8-way
ifAbsent storm advances the store by exactly one generation with _rev 1;
8-way upsert storm on an absent id yields one create + seven merges
(_rev === 8); sequential ifAbsent/upsert semantics pinned unchanged; N
identical concurrent blob writes → refCount === N; the blob survives until
the true last reference drops; interleaved write/delete storm never loses a
landed reference.
This commit is contained in:
David Snelling 2026-07-08 15:13:26 -07:00
parent 7146ce3544
commit 867939ed50
3 changed files with 387 additions and 61 deletions
Download patch file Download diff file Expand all files Collapse all files

117
src/brainy.ts
View file

@ -337,6 +337,22 @@ interface PlannedTransact {
createdNouns: Set<string>
}
/**
* Internal control-flow signal: an insert guarded by a must-be-absent
* precondition (`add({ ifAbsent })` / `add({ upsert })`) found the entity
* already present in the authoritative before-image under the commit mutex —
* a concurrent writer created it after the planning-time absence check.
* Never escapes `add()`: the caller converts it into the documented
* resolution (ifAbsent → return the existing id without writing; upsert →
* merge into the now-existing entity via `update()`).
*/
class InsertPreconditionExistsSignal extends Error {
constructor(readonly id: string) {
super(`insert precondition: entity ${id} already exists`)
this.name = 'InsertPreconditionExistsSignal'
}
}
/**
* The main Brainy class - Clean, Beautiful, Powerful
* REAL IMPLEMENTATION - No stubs, no mocks
@ -1639,7 +1655,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// ifAbsent (7.31.0) — by-ID idempotent insert. Only meaningful when a custom id
// is supplied; a freshly generated UUID can never collide. Returns the existing
// (canonical) id without writing if the entity is already present (no throw,
// no overwrite).
// no overwrite). This check is a FAST-FAIL (skips the embedding cost); the
// authoritative absence check runs in the insert's commit precondition
// below, atomic with the apply, so a concurrent same-id create cannot make
// both callers write.
if (params.id && params.ifAbsent) {
const existing = await this.storage.getNounMetadata(id)
if (existing) return id
@ -1650,21 +1669,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// delegate to update() so the supplied fields MERGE into the existing entity
// (metadata merge, re-embed on changed data, _rev bump, createdAt preserved)
// instead of the default destructive overwrite. When the id is absent, fall
// through to the normal insert below.
// through to the guarded insert below (whose commit precondition converts a
// concurrent same-id create into this same merge — never an overwrite).
if (params.id && params.upsert) {
const existing = await this.storage.getNounMetadata(id)
if (existing) {
await this.update({
id,
...(params.data !== undefined && { data: params.data }),
...(params.type !== undefined && { type: params.type }),
...(params.subtype !== undefined && { subtype: params.subtype }),
...(params.visibility !== undefined && { visibility: params.visibility }),
...(params.metadata !== undefined && { metadata: params.metadata }),
...(params.vector !== undefined && { vector: params.vector }),
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight })
})
await this.update(this.upsertMergeParams(id, params))
return id
}
}
@ -1736,7 +1746,22 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Execute atomically with transaction system, generation-stamped as one
// immutable Model-B generation (before-image = the absent/create sentinel).
// All operations succeed or all rollback - prevents partial failures
await this.persistSingleOp({ nouns: [id] }, async (tx) => {
// ifAbsent/upsert insert leg: must-be-absent commit precondition, run
// under the commit mutex against the authoritative before-image — the
// planning-time absence checks above are fast-fails that can interleave
// with a concurrent same-id create. Exactly one concurrent create wins;
// every other caller takes its documented resolution instead of
// silently overwriting the winner.
const requireAbsent = Boolean(params.id && (params.ifAbsent || params.upsert))
const insertPrecommit = requireAbsent
? (before: CommitBeforeImages): void => {
if (before.nouns.get(id)?.metadata) {
throw new InsertPreconditionExistsSignal(id)
}
}
: undefined
const runInsert: TransactionFunction<void> = async (tx) => {
// Operation 1: Save metadata FIRST (TypeAwareStorage caching)
// isNew=true: skip pre-read for rollback (entity doesn't exist yet)
tx.addOperation(
@ -1763,7 +1788,44 @@ export class Brainy<T = any> implements BrainyInterface<T> {
tx.addOperation(
new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing)
)
})
}
// Bounded retry closes the remaining interleavings without ever holding
// a lock across the loop: a lost insert race resolves to skip (ifAbsent)
// or merge (upsert); a merge that then hits a concurrent delete retries
// the insert. Each persistSingleOp call constructs fresh operations, so
// re-running the callback is safe.
const MAX_UPSERT_ATTEMPTS = 10
for (let attempt = 0; ; attempt++) {
try {
await this.persistSingleOp({ nouns: [id] }, runInsert, insertPrecommit)
break
} catch (err) {
if (!(err instanceof InsertPreconditionExistsSignal)) {
throw err
}
// A concurrent writer created the entity between our absence check
// and the commit.
if (params.ifAbsent) {
// Documented contract: return the existing id without writing.
return id
}
// upsert: merge into the now-existing entity. A concurrent delete
// between this conflict and the merge throws EntityNotFoundError —
// loop back and retry the insert.
try {
await this.update(this.upsertMergeParams(id, params))
return id
} catch (mergeErr) {
if (
!(mergeErr instanceof EntityNotFoundError) ||
attempt >= MAX_UPSERT_ATTEMPTS
) {
throw mergeErr
}
}
}
}
// Aggregation hook (outside transaction — derived data, can be reconstructed)
if (this._aggregationIndex) {
@ -1773,6 +1835,31 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return id
}
/**
* @description The `update()` param mapping `add({ upsert })` delegates to
* when the target id already exists: every supplied field merges into the
* existing entity (metadata merge, re-embed on changed data, `_rev` bump,
* `createdAt` preserved); absent fields are left untouched. Shared by the
* planning-time upsert branch and the commit-conflict retry so the two
* paths can never drift.
* @param id - The canonical entity id the upsert resolved to.
* @param params - The original `add()` params carrying the fields to merge.
* @returns The `UpdateParams` for the merging `update()` call.
*/
private upsertMergeParams(id: string, params: AddParams<T>): UpdateParams<T> {
return {
id,
...(params.data !== undefined && { data: params.data }),
...(params.type !== undefined && { type: params.type }),
...(params.subtype !== undefined && { subtype: params.subtype }),
...(params.visibility !== undefined && { visibility: params.visibility }),
...(params.metadata !== undefined && { metadata: params.metadata }),
...(params.vector !== undefined && { vector: params.vector }),
...(params.confidence !== undefined && { confidence: params.confidence }),
...(params.weight !== undefined && { weight: params.weight })
} as UpdateParams<T>
}
/**
* Get an entity by ID
*