fix: getNouns().totalCount reports true total, not page size; quiet benign mmap-vector log

getNounsWithPagination returned collectedNouns.length as totalCount, but the
type-first shard scan early-terminates at offset+limit — so
getNouns({ pagination: { limit: 1 } }).totalCount was 1 for any non-empty brain.
The index-rebuild gate calls exactly that, so cold starts logged
"Small dataset (1 items) - rebuilding all indexes" and rebuilt from scratch
regardless of corpus size (a production deployment saw this for an ~8,800-entity
brain). Now reports the authoritative O(1) noun counter (maintained on add/delete,
rehydrated from counts.json on init) as the unfiltered total and derives hasMore
from it. Filtered scans unchanged. Layout-independent (branch/COW included).

Also downgrade the "mmap-vector backend not wired" console.log to prodLog.debug:
it is benign in the native-vector-index model (the native provider owns its own
vector storage and has no setVectorBackend hook), but it fired on every init and
was repeatedly mistaken for the cold-start cause.

Regression: tests/unit/storage/getNouns-totalCount.test.ts. Full unit suite green (1505).
This commit is contained in:
David Snelling 2026-06-17 14:01:33 -07:00
parent adec0ba3c3
commit edff637bfa
4 changed files with 156 additions and 8 deletions

View file

@ -9473,12 +9473,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
setVectorBackend?: (backend: MmapVectorBackend) => void
}
if (typeof indexWithBackend.setVectorBackend !== 'function') {
if (!this.config.silent) {
console.log(
'[brainy] mmap-vector backend not wired (vector index manages its own ' +
'vector storage; no setVectorBackend hook) — per-entity reads in use'
)
}
// Expected in the native-vector-index model: a native provider (e.g.
// @soulcraft/cortex) replaces the JS HNSW index and owns its own vector
// storage + persisted snapshot, so there is no setVectorBackend hook to
// wire here. This is benign, not a fault, and does NOT by itself imply
// per-entity reads — keep it at debug level so it never reads as a problem
// in normal operation. (The old console.log fired on every init and was
// repeatedly mistaken for the cold-start cause; see BRAINY-MMAP-VECTOR-HOOK.)
prodLog.debug(
'[brainy] mmap-vector backend not wired (native vector index manages ' +
'its own vector storage; no setVectorBackend hook)'
)
return
}

View file

@ -1515,11 +1515,28 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Apply pagination
const paginatedNouns = collectedNouns.slice(offset, offset + limit)
const hasMore = collectedNouns.length > targetCount
// totalCount must be the TRUE dataset total, not the size of this page.
// The shard scan above early-terminates at `targetCount = offset + limit`
// for memory efficiency, so `collectedNouns.length` only ever reaches the
// page size — returning it as `totalCount` made every non-empty brain look
// like it held exactly `limit` items. In particular
// `getNouns({ pagination: { limit: 1 } })` reported `totalCount: 1`, which
// tripped the index-rebuild gate into logging "Small dataset (1 items)" and
// rebuilding from scratch regardless of the real corpus size. For the
// unfiltered case the authoritative total is the O(1) counter maintained on
// every add/delete and rehydrated from `counts.json` on init; `Math.max`
// guards against a stale counter ever under-reporting below what we
// actually collected. A filtered scan has no cheap exact total, so it keeps
// the collected length (a lower bound — unchanged behaviour).
const totalCount = filter
? collectedNouns.length
: Math.max(this.totalNounCount, collectedNouns.length)
const hasMore = offset + paginatedNouns.length < totalCount
return {
items: paginatedNouns,
totalCount: collectedNouns.length,
totalCount,
hasMore,
nextCursor: hasMore && paginatedNouns.length > 0
? paginatedNouns[paginatedNouns.length - 1].id