R5 follow-up: find({type:[array]}) still partial under interleaved writes (7.28.0/cortex 2.5.0) #2

New issue

Open

opened 2026-05-29 00:54:52 +02:00 by dpsifr · 0 comments

dpsifr commented

2026-05-29 00:54:52 +02:00

(Migrated from github.com)

R5 follow-up: `find({ type: [array] })` still returns partial/empty results after interleaved writes (Brainy 7.28.0 + Cortex 2.5.0)

TL;DR

The R5 fix in 7.28 closed the simple repro (find({ type:[arr], orderBy }) on a freshly-written brain returns the right count consistently). But under a realistic access pattern — interleaved add / update / find calls on a long-lived brain — find({ type: [array] }) still intermittently returns 0 entities while the same query without the type filter returns the correct count.

Affects:

@soulcraft/brainy 7.28.0
@soulcraft/cortex 2.5.0

Observed in production-style code

This is how the Memory service triggers it. The "list all memories" query has three flavours that should agree on count; they don't:

// What we want to use (native, fast):
const combo = await brain.find({
  type: ['concept','event','process','person','norm','proposition','quality','organization','project','state'],
  where: { retracted: { missing: true } },
  excludeVFS: true,
  orderBy: 'createdAt',
  order: 'desc',
  limit: 50,
})

// Type filter only:
const noWhere = await brain.find({
  type: [...same array],
  excludeVFS: true,
  limit: 50,
})

// where:retracted:missing only:
const noType = await brain.find({
  where: { retracted: { missing: true } },
  excludeVFS: true,
  limit: 50,
})

Output across calls in a long-lived brain that has interleaved add + update traffic:

[LIST] combo=0  noWhere=0  noType=0      ← all 3 empty first call
[LIST] combo=0  noWhere=0  noType=4      ← type filter empty, where works
[LIST] combo=6  noWhere=6  noType=8      ← type filter recovers; counts now disagree
[LIST] combo=2  noWhere=2  noType=2
[LIST] combo=7  noWhere=7  noType=9
[LIST] combo=6  noWhere=6  noType=8
[LIST] combo=7  noWhere=7  noType=9
[LIST] combo=8  noWhere=8  noType=10
[LIST] combo=10 noWhere=10 noType=12
[LIST] combo=0  noWhere=11 noType=11     ← type-array filter drops to 0 mid-run; where-alone is correct
[LIST] combo=10 noWhere=11 noType=11     ← recovers on next call
[LIST] combo=11 noWhere=12 noType=13
[LIST] combo=12 noWhere=13 noType=14

Two distinct bugs visible:

First-call empties. Even after brain.add(...) + (implicit) flush, the first find({ type:[array] }) call returns 0 — same shape as R5, just under a different trigger (interleaved updates rather than fresh-write).
Mid-stream combo=0. After many successful queries, find({ type:[array], where:{retracted:{missing:true}} }) returns 0 while find({ where:{retracted:{missing:true}} }) (no type filter) returns the correct 11. So the type:[array] filter is intermittently producing an empty intersection.

noWhere and noType never drop to 0 once the brain has entities — the failure mode is specifically on queries that include type: [array].

Minimal repro

import { describe, it, beforeAll, afterAll, expect } from 'vitest'
import { Brainy } from '@soulcraft/brainy'

const MEM = ['concept','event','process','person','norm','proposition','quality','organization','project','state']

describe('R5 follow-up', () => {
  let brain: Brainy
  beforeAll(async () => {
    brain = await Brainy.create({ storage: { type: 'mmap-filesystem', directory: './brainy-r5fu' } })
  })
  afterAll(async () => { await (brain as any).shutdown?.() })

  it('type:[array] returns consistent counts under interleaved write+read', async () => {
    // seed
    for (let i = 0; i < 5; i++) {
      await brain.add({ type: 'concept', data: `m${i}`, metadata: { confidence: 0.7, tags: ['x'] } })
    }

    const counts: number[] = []
    for (let round = 0; round < 10; round++) {
      // mutate one
      const all = await brain.find({ where: { confidence: { gte: 0 } }, limit: 100 }) // no type filter — control
      if (all[0]) await brain.update({ id: all[0].id, metadata: { recallCount: round, lastRecalled: Date.now() } })
      // read with type:[array]
      const r = await brain.find({ type: MEM, excludeVFS: true, limit: 50 })
      counts.push(r.length)
    }

    // expected: every count >= 5; observed: intermittent 0 / undercount
    expect(counts.every(c => c >= 5)).toBe(true)
  })
})

Workaround in consumer code

Dropping the type filter and post-filtering noun types in JS makes the query stable. That's what we'd like to not have to do — it forces an over-fetch and a hot-path JS loop on every list/recall query.

Asks

Investigate why find({ type: [array] }) still returns partial/empty results on long-lived brains with interleaved writes — first call AND mid-stream.
Confirm: is the type:[] filter using a different index path than single-type or where? The fact that where:retracted:missing alone is stable while type:[array] alone (and combined) is unstable points at the type-array index code.
Once fixed, bump probe in tests/brainy-probes.test.ts (we maintain a Brainy capability probe that gates upgrades — happy to share or upstream it).

Reproducer from the Memory service. Happy to provide a self-contained brainy directory tarball if useful.

# R5 follow-up: `find({ type: [array] })` still returns partial/empty results after interleaved writes (Brainy 7.28.0 + Cortex 2.5.0) ## TL;DR The R5 fix in 7.28 closed the simple repro (`find({ type:[arr], orderBy })` on a freshly-written brain returns the right count consistently). But under a realistic access pattern — interleaved `add` / `update` / `find` calls on a long-lived brain — `find({ type: [array] })` still intermittently returns **0 entities** while the same query without the `type` filter returns the correct count. Affects: - `@soulcraft/brainy` 7.28.0 - `@soulcraft/cortex` 2.5.0 ## Observed in production-style code This is how the Memory service triggers it. The "list all memories" query has three flavours that should agree on count; they don't: ```ts // What we want to use (native, fast): const combo = await brain.find({ type: ['concept','event','process','person','norm','proposition','quality','organization','project','state'], where: { retracted: { missing: true } }, excludeVFS: true, orderBy: 'createdAt', order: 'desc', limit: 50, }) // Type filter only: const noWhere = await brain.find({ type: [...same array], excludeVFS: true, limit: 50, }) // where:retracted:missing only: const noType = await brain.find({ where: { retracted: { missing: true } }, excludeVFS: true, limit: 50, }) ``` Output across calls in a long-lived brain that has interleaved `add` + `update` traffic: ``` [LIST] combo=0 noWhere=0 noType=0 ← all 3 empty first call [LIST] combo=0 noWhere=0 noType=4 ← type filter empty, where works [LIST] combo=6 noWhere=6 noType=8 ← type filter recovers; counts now disagree [LIST] combo=2 noWhere=2 noType=2 [LIST] combo=7 noWhere=7 noType=9 [LIST] combo=6 noWhere=6 noType=8 [LIST] combo=7 noWhere=7 noType=9 [LIST] combo=8 noWhere=8 noType=10 [LIST] combo=10 noWhere=10 noType=12 [LIST] combo=0 noWhere=11 noType=11 ← type-array filter drops to 0 mid-run; where-alone is correct [LIST] combo=10 noWhere=11 noType=11 ← recovers on next call [LIST] combo=11 noWhere=12 noType=13 [LIST] combo=12 noWhere=13 noType=14 ``` Two distinct bugs visible: 1. **First-call empties.** Even after `brain.add(...)` + (implicit) flush, the first `find({ type:[array] })` call returns 0 — same shape as R5, just under a different trigger (interleaved updates rather than fresh-write). 2. **Mid-stream `combo=0`.** After many successful queries, `find({ type:[array], where:{retracted:{missing:true}} })` returns 0 while `find({ where:{retracted:{missing:true}} })` (no type filter) returns the correct 11. So the `type:[array]` filter is intermittently producing an empty intersection. `noWhere` and `noType` *never* drop to 0 once the brain has entities — the failure mode is specifically on queries that include `type: [array]`. ## Minimal repro ```ts import { describe, it, beforeAll, afterAll, expect } from 'vitest' import { Brainy } from '@soulcraft/brainy' const MEM = ['concept','event','process','person','norm','proposition','quality','organization','project','state'] describe('R5 follow-up', () => { let brain: Brainy beforeAll(async () => { brain = await Brainy.create({ storage: { type: 'mmap-filesystem', directory: './brainy-r5fu' } }) }) afterAll(async () => { await (brain as any).shutdown?.() }) it('type:[array] returns consistent counts under interleaved write+read', async () => { // seed for (let i = 0; i < 5; i++) { await brain.add({ type: 'concept', data: `m${i}`, metadata: { confidence: 0.7, tags: ['x'] } }) } const counts: number[] = [] for (let round = 0; round < 10; round++) { // mutate one const all = await brain.find({ where: { confidence: { gte: 0 } }, limit: 100 }) // no type filter — control if (all[0]) await brain.update({ id: all[0].id, metadata: { recallCount: round, lastRecalled: Date.now() } }) // read with type:[array] const r = await brain.find({ type: MEM, excludeVFS: true, limit: 50 }) counts.push(r.length) } // expected: every count >= 5; observed: intermittent 0 / undercount expect(counts.every(c => c >= 5)).toBe(true) }) }) ``` ## Workaround in consumer code Dropping the `type` filter and post-filtering noun types in JS makes the query stable. That's what we'd like to **not** have to do — it forces an over-fetch and a hot-path JS loop on every list/recall query. ## Asks 1. Investigate why `find({ type: [array] })` still returns partial/empty results on long-lived brains with interleaved writes — first call AND mid-stream. 2. Confirm: is the `type:[]` filter using a different index path than single-`type` or `where`? The fact that `where:retracted:missing` alone is stable while `type:[array]` alone (and combined) is unstable points at the type-array index code. 3. Once fixed, bump probe in `tests/brainy-probes.test.ts` (we maintain a Brainy capability probe that gates upgrades — happy to share or upstream it). Reproducer from the Memory service. Happy to provide a self-contained brainy directory tarball if useful.