fix: getNouns().totalCount reports true total, not page size; quiet benign mmap-vector log
getNounsWithPagination returned collectedNouns.length as totalCount, but the
type-first shard scan early-terminates at offset+limit — so
getNouns({ pagination: { limit: 1 } }).totalCount was 1 for any non-empty brain.
The index-rebuild gate calls exactly that, so cold starts logged
"Small dataset (1 items) - rebuilding all indexes" and rebuilt from scratch
regardless of corpus size (a production deployment saw this for an ~8,800-entity
brain). Now reports the authoritative O(1) noun counter (maintained on add/delete,
rehydrated from counts.json on init) as the unfiltered total and derives hasMore
from it. Filtered scans unchanged. Layout-independent (branch/COW included).
Also downgrade the "mmap-vector backend not wired" console.log to prodLog.debug:
it is benign in the native-vector-index model (the native provider owns its own
vector storage and has no setVectorBackend hook), but it fired on every init and
was repeatedly mistaken for the cold-start cause.
Regression: tests/unit/storage/getNouns-totalCount.test.ts. Full unit suite green (1505).
This commit is contained in:
parent
adec0ba3c3
commit
edff637bfa
4 changed files with 156 additions and 8 deletions
|
|
@ -9473,12 +9473,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
setVectorBackend?: (backend: MmapVectorBackend) => void
|
||||
}
|
||||
if (typeof indexWithBackend.setVectorBackend !== 'function') {
|
||||
if (!this.config.silent) {
|
||||
console.log(
|
||||
'[brainy] mmap-vector backend not wired (vector index manages its own ' +
|
||||
'vector storage; no setVectorBackend hook) — per-entity reads in use'
|
||||
)
|
||||
}
|
||||
// Expected in the native-vector-index model: a native provider (e.g.
|
||||
// @soulcraft/cortex) replaces the JS HNSW index and owns its own vector
|
||||
// storage + persisted snapshot, so there is no setVectorBackend hook to
|
||||
// wire here. This is benign, not a fault, and does NOT by itself imply
|
||||
// per-entity reads — keep it at debug level so it never reads as a problem
|
||||
// in normal operation. (The old console.log fired on every init and was
|
||||
// repeatedly mistaken for the cold-start cause; see BRAINY-MMAP-VECTOR-HOOK.)
|
||||
prodLog.debug(
|
||||
'[brainy] mmap-vector backend not wired (native vector index manages ' +
|
||||
'its own vector storage; no setVectorBackend hook)'
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1515,11 +1515,28 @@ export abstract class BaseStorage extends BaseStorageAdapter {
|
|||
|
||||
// Apply pagination
|
||||
const paginatedNouns = collectedNouns.slice(offset, offset + limit)
|
||||
const hasMore = collectedNouns.length > targetCount
|
||||
|
||||
// totalCount must be the TRUE dataset total, not the size of this page.
|
||||
// The shard scan above early-terminates at `targetCount = offset + limit`
|
||||
// for memory efficiency, so `collectedNouns.length` only ever reaches the
|
||||
// page size — returning it as `totalCount` made every non-empty brain look
|
||||
// like it held exactly `limit` items. In particular
|
||||
// `getNouns({ pagination: { limit: 1 } })` reported `totalCount: 1`, which
|
||||
// tripped the index-rebuild gate into logging "Small dataset (1 items)" and
|
||||
// rebuilding from scratch regardless of the real corpus size. For the
|
||||
// unfiltered case the authoritative total is the O(1) counter maintained on
|
||||
// every add/delete and rehydrated from `counts.json` on init; `Math.max`
|
||||
// guards against a stale counter ever under-reporting below what we
|
||||
// actually collected. A filtered scan has no cheap exact total, so it keeps
|
||||
// the collected length (a lower bound — unchanged behaviour).
|
||||
const totalCount = filter
|
||||
? collectedNouns.length
|
||||
: Math.max(this.totalNounCount, collectedNouns.length)
|
||||
const hasMore = offset + paginatedNouns.length < totalCount
|
||||
|
||||
return {
|
||||
items: paginatedNouns,
|
||||
totalCount: collectedNouns.length,
|
||||
totalCount,
|
||||
hasMore,
|
||||
nextCursor: hasMore && paginatedNouns.length > 0
|
||||
? paginatedNouns[paginatedNouns.length - 1].id
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue