soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: find()/stats() correctness + Cortex compat (BR-FIND-WHERE-ZERO, BR-DEFENSIVE-INTERFACE)

Browse source 
Two production correctness defects fixed together so consumers can land
one upgrade.

BR-FIND-WHERE-ZERO — find() returned [] and stats() reported 0 entities
for any workspace whose data was written after the 7.20.0 column-store
refactor. Root cause: getStats() and the getIds() fallback still read
from the deleted sparse-index path. Separately, BaseStorage.getNounType()
was hardcoded to return 'thing', poisoning type-statistics.json with
every noun attributed to that bucket.

  - MetadataIndex.getStats() reads from ColumnStore + idMapper.
  - MetadataIndex.getIdsFromChunks() throws BrainyError(FIELD_NOT_INDEXED)
    when neither store has the field. getIdsForFilter() catches per
    clause, logs once, returns [].
  - BaseStorage.nounTypeByIdCache populated in saveNounMetadata_internal
    and consumed in saveNoun_internal. flushCounts() now persists the
    Uint32Array counters too, so readers see fresh per-type counts.
  - Self-heal at init: loadTypeStatistics() auto-rebuilds when the
    poisoned signature is detected. rebuildTypeCounts() is now public for
    use from `brainy inspect repair`.
  - Dead state removed: dirtyChunks, dirtySparseIndices,
    flushDirtyMetadata().

BR-DEFENSIVE-INTERFACE — 7.21.0 called supportsMultiProcessLocking()
unconditionally, crashing on older Cortex storage adapters that predate
the method. New hasStorageMethod(name) helper gates every new-method
call site. Older adapter triggers a one-line warning at init pointing
at the recommended plugin version.

Tests:
  - new tests/integration/find-where-zero.test.ts (7 cases)
  - new tests/integration/cortex-compat.test.ts (5 cases)
  - multi-process-safety.test.ts updated to enforce correct counts
  - 1329/1329 unit + 24/24 new integration tests passing
This commit is contained in:
David Snelling 2026-05-15 12:31:28 -07:00
parent 79f58cb87b
commit 70263113ad
9 changed files with 895 additions and 144 deletions
Download patch file Download diff file Expand all files Collapse all files

65
src/brainy.ts
View file

@ -302,6 +302,20 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return !this.operationalMode.canWrite
}
/**
* Whether the active storage adapter implements a given optional method.
*
* Brainy ships new storage-adapter capabilities over time (multi-process
* locking, cross-process flush RPC, etc.). Older plugins (notably
* `@soulcraft/cortex` ≤2.2.0) bundle their own `BaseStorage` from an
* earlier Brainy version that doesn't have those methods. Rather than crash
* at runtime, every call site that uses an optional storage method goes
* through this helper and degrades gracefully when the method is absent.
*/
private hasStorageMethod(name: string): boolean {
return !!this.storage && typeof (this.storage as unknown as Record<string, unknown>)[name] === 'function'
}
/**
* Throw if this instance cannot perform writes. Called at the top of every
* mutation method. The check is cheap (a property lookup + boolean test) and
@ -385,24 +399,41 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Acquire the writer lock for filesystem (and other locking-capable) backends.
// Skipped in reader mode and on backends that don't support multi-process locking.
// Throws if another live writer holds the directory (unless force: true).
if (this.config.mode !== 'reader' && this.storage.supportsMultiProcessLocking()) {
await this.storage.acquireWriterLock({ force: this.config.force })
// Watch for cross-process flush requests so inspectors can see fresh
// state on demand. Reader instances don't run a watcher (nothing to flush).
this.storage.startFlushRequestWatcher(async () => {
if (this.initialized) {
await this.flush()
//
// Defensive call: older storage adapters (e.g. `@soulcraft/cortex@2.2.0`
// and earlier) bundle a pre-7.21 `BaseStorage` that doesn't define these
// methods. We feature-detect each one rather than fail boot.
if (this.config.mode !== 'reader') {
const canLock = this.hasStorageMethod('supportsMultiProcessLocking') &&
(this.storage as any).supportsMultiProcessLocking()
if (canLock && this.hasStorageMethod('acquireWriterLock')) {
await (this.storage as any).acquireWriterLock({ force: this.config.force })
if (this.hasStorageMethod('startFlushRequestWatcher')) {
(this.storage as any).startFlushRequestWatcher(async () => {
if (this.initialized) {
await this.flush()
}
})
}
} else if (!this.config.silent) {
// Older adapter OR a backend that doesn't enforce locking (cloud / memory).
// Surface this so operators know the multi-process protections aren't active.
const backendName = (this.storage.constructor as any).name || 'storage'
if (backendName === 'MemoryStorage') {
// Memory is single-process by construction — no warning needed.
} else if (!this.hasStorageMethod('supportsMultiProcessLocking')) {
console.warn(
`[brainy] Storage adapter \`${backendName}\` predates the 7.21 ` +
`multi-process methods. Writer locking and the flush-request RPC ` +
`are disabled for this directory. Upgrade the plugin (e.g. ` +
`\`@soulcraft/cortex\` ≥2.3.0) for full enforcement.`
)
} else {
console.warn(
`[brainy] Multi-process writer protection is not enforced on ${backendName}. ` +
`See docs/concepts/multi-process.md for the model.`
)
}
})
} else if (this.config.mode !== 'reader' && !this.config.silent) {
// Cloud / memory backends: multi-process safety not enforced. One-time
// warning so operators know what they're getting.
const backendName = (this.storage.constructor as any).name || 'storage'
if (backendName !== 'MemoryStorage') {
console.warn(
`[brainy] Multi-process writer protection is not enforced on ${backendName}. ` +
`See docs/concepts/multi-process.md for the model.`
)
}
}

28
src/errors/brainyError.ts
View file

@ -3,7 +3,14 @@
* Provides better error classification and handling
*/
export type BrainyErrorType = 'TIMEOUT' | 'NETWORK' | 'STORAGE' | 'NOT_FOUND' | 'RETRY_EXHAUSTED' | 'VALIDATION'
export type BrainyErrorType =
| 'TIMEOUT'
| 'NETWORK'
| 'STORAGE'
| 'NOT_FOUND'
| 'RETRY_EXHAUSTED'
| 'VALIDATION'
| 'FIELD_NOT_INDEXED'
/**
* Custom error class for Brainy operations
@ -99,6 +106,25 @@ export class BrainyError extends Error {
)
}
/**
* Create a "field is not indexed" error. Thrown by metadata-index reads
* when a `where` clause names a field that has neither a column-store
* entry nor a sparse-index entry. Callers in `find()` evaluation catch
* this, translate the offending clause to an empty result, and log so
* the silent-empty behavior is replaced with a loud one. Use
* `brain.explain({ where: {...} })` to discover this before running.
*/
static fieldNotIndexed(field: string): BrainyError {
return new BrainyError(
`Field "${field}" is not indexed. find()/where will not match any entities. ` +
`Likely causes: (1) the writer registered the field in memory but has not flushed; ` +
`(2) the field name is mistyped; (3) no entity has ever held this field. ` +
`Run brain.explain({ where: { ${field}: ... } }) for the diagnostic.`,
'FIELD_NOT_INDEXED',
false
)
}
/**
* Create a validation error
*/

38
src/indexes/columnStore/ColumnStore.ts
View file

@ -309,6 +309,44 @@ export class ColumnStore implements ColumnStoreProvider {
return (manifest !== undefined && !manifest.isEmpty()) || (buffer !== undefined && buffer.size > 0)
}
/**
* List every field that has at least one persisted segment or buffered
* write. Used by `MetadataIndexManager.getStats()` and by
* `brainy inspect fields` so callers see the same field set the column
* store will actually serve queries from.
*/
getIndexedFields(): string[] {
const fields = new Set<string>()
for (const [field, manifest] of this.manifests) {
if (!manifest.isEmpty()) fields.add(field)
}
for (const [field, buffer] of this.tailBuffers) {
if (buffer.size > 0) fields.add(field)
}
return Array.from(fields).sort()
}
/**
* Approximate segment + tail-buffer sizes per indexed field. Returns
* `segmentCount` (number of persisted L0+ segments) and `tailSize` (entries
* buffered but not yet flushed) per field. Suitable for stats / health
* surfaces. For exact distinct-value cardinality use
* `getFilterValues(field).length` per field on demand.
*/
getFieldSizeSummary(): Array<{ field: string; segmentCount: number; tailSize: number }> {
const summary: Array<{ field: string; segmentCount: number; tailSize: number }> = []
for (const field of this.getIndexedFields()) {
const manifest = this.manifests.get(field)
const buffer = this.tailBuffers.get(field)
const segmentCount = manifest && !manifest.isEmpty()
? manifest.getAllSegments().length
: 0
const tailSize = buffer ? buffer.size : 0
summary.push({ field, segmentCount, tailSize })
}
return summary
}
/**
* Flush all tail buffers to L0 segments and save manifests.
* No-op if init() hasn't been called (no storage to write to).

190
src/storage/baseStorage.ts
View file

@ -223,6 +223,21 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Built into all storage adapters for billion-scale efficiency
protected nounCountsByType = new Uint32Array(NOUN_TYPE_COUNT) // 168 bytes (Stage 3: 42 types)
protected verbCountsByType = new Uint32Array(VERB_TYPE_COUNT) // 508 bytes (Stage 3: 127 types)
/**
* In-memory map from noun id → its `noun` (NounType) value, populated when
* `saveNounMetadata_internal()` runs. `saveNoun_internal()` consults this
* to attribute the entity to the correct slot in `nounCountsByType` so the
* persisted `_system/type-statistics.json` is honest.
*
* History: a previous cache was removed in commit `42ae5be` and replaced
* with a hardcoded `return 'thing'` from `getNounType()`, which silently
* poisoned the on-disk type statistics. This cache restores the correct
* behavior. Memory footprint is one Map entry per live noun id (typically
* tens of bytes each); the writer process keeps it for the duration of
* its lifetime and prunes on delete.
*/
protected nounTypeByIdCache = new Map<string, NounType>()
// Total: 676 bytes (99.2% reduction vs Map-based tracking)
// Type caches REMOVED - ID-first paths eliminate need for type lookups!
@ -2172,6 +2187,15 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Save the metadata (COW-aware - writes to branch-specific path)
await this.writeObjectToBranch(path, metadata)
// Record the id→type mapping so `saveNoun_internal()` (which receives only
// an HNSWNoun and has no metadata access in its signature) can attribute
// the entity to the right slot in `nounCountsByType`. This is what makes
// `_system/type-statistics.json` honest. Updates the cache even for
// existing entities so a type change via `update()` is reflected.
if (metadata.noun) {
this.nounTypeByIdCache.set(id, metadata.noun as NounType)
}
// CRITICAL FIX: Increment count for new entities
// This runs AFTER metadata is saved, guaranteeing type information is available
// Uses synchronous increment since storage operations are already serialized
@ -2560,6 +2584,19 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Direct O(1) delete with ID-first path
const path = getNounMetadataPath(id)
await this.deleteObjectFromBranch(path)
// Prune the id→type cache so a future re-add of the same id (e.g. churn
// during tests) doesn't see a stale type. Lookup the prior type and
// decrement `nounCountsByType` so type-statistics stay honest across
// deletes; symmetric with the increment in `saveNounMetadata_internal()`.
const priorType = this.nounTypeByIdCache.get(id)
if (priorType) {
this.nounTypeByIdCache.delete(id)
const idx = TypeUtils.getNounIndex(priorType)
if (this.nounCountsByType[idx] > 0) {
this.nounCountsByType[idx]--
}
}
}
/**
@ -2660,6 +2697,11 @@ export abstract class BaseStorage extends BaseStorageAdapter {
/**
* Load type statistics from storage
* Rebuilds type counts if needed (called during init)
*
* Auto-detects the 7.20.0–7.21.0 poisoned-statistics signature: every
* noun attributed to `'thing'` because the old `getNounType()` was hardcoded.
* If detected, runs `rebuildTypeCounts()` once to rewrite the file with
* correct per-type counts derived from on-disk metadata. Logged loudly.
*/
protected async loadTypeStatistics(): Promise<void> {
try {
@ -2673,12 +2715,69 @@ export abstract class BaseStorage extends BaseStorageAdapter {
if (stats.verbCounts && stats.verbCounts.length === VERB_TYPE_COUNT) {
this.verbCountsByType = new Uint32Array(stats.verbCounts)
}
if (await this.detectPoisonedTypeStatistics()) {
prodLog.warn(
'[BaseStorage] Detected poisoned type-statistics.json signature ' +
'(all nouns attributed to \'thing\' — symptom of pre-7.22 getNounType ' +
'hardcode). Rebuilding counts from on-disk metadata.'
)
await this.rebuildTypeCounts()
}
}
} catch (error) {
// No existing type statistics, starting fresh
}
}
/**
* Detect the 7.20.0–7.21.0 poisoned-statistics signature.
*
* The defect: the old `getNounType()` returned hardcoded `'thing'`, so
* `_system/type-statistics.json` ended up with `nounCounts[thing] === N`
* and every other index `=== 0`, regardless of the actual mix on disk.
*
* Heuristic: nounCounts has at least 2 non-thing entities on disk (so the
* file *should* show multiple non-zero buckets) but only the `thing`
* bucket is non-zero. We bound the on-disk check with `limit: 3` so this
* never costs more than a couple of cheap reads.
*
* Returns false (no self-heal needed) for genuinely thing-only stores or
* empty stores.
*/
private async detectPoisonedTypeStatistics(): Promise<boolean> {
const thingIdx = TypeUtils.getNounIndex('thing' as NounType)
if (thingIdx < 0) return false
let nonZeroBuckets = 0
let thingCount = 0
for (let i = 0; i < this.nounCountsByType.length; i++) {
if (this.nounCountsByType[i] > 0) {
nonZeroBuckets++
if (i === thingIdx) thingCount = this.nounCountsByType[i]
}
}
// Only one bucket populated, and it's 'thing' with ≥2 entities — suspicious.
if (nonZeroBuckets !== 1 || thingCount < 2) return false
// Cross-check: sample a few metadata files. If any non-'thing' types
// show up, we're confirmed poisoned. If only 'thing' types appear in the
// sample, this is a genuine thing-only store and we leave the file alone.
try {
const sample = await this.getNouns({ pagination: { offset: 0, limit: 3 } })
for (const noun of sample.items) {
const type = await this.getNounTypeFromStorageAsync(noun.id)
if (type && type !== ('thing' as NounType)) {
return true
}
}
} catch {
// If we can't read the metadata, don't trigger a rebuild — leave state alone.
}
return false
}
/**
* Save type statistics to storage
* Periodically called when counts are updated
@ -2693,6 +2792,24 @@ export abstract class BaseStorage extends BaseStorageAdapter {
await this.writeObjectToPath(`${SYSTEM_DIR}/type-statistics.json`, stats)
}
/**
* Persist both counter systems atomically when an explicit flush is
* requested. `super.flushCounts()` writes `entityCounts` (the Map in
* `BaseStorageAdapter`); we additionally write `nounCountsByType` /
* `verbCountsByType` so a reader opening the same directory sees the
* exact same counts the writer holds in memory.
*
* Before this override the Uint32Array counters were only persisted
* on a heuristic schedule inside `saveNoun_internal` (first-of-type or
* every-100th), which left readers seeing stale counts after a clean
* writer flush — the same failure mode that BR-FIND-WHERE-ZERO surfaced
* via `brain.stats()`.
*/
public async flushCounts(): Promise<void> {
await super.flushCounts()
await this.saveTypeStatistics()
}
/**
* Get noun counts by type (O(1) access to type statistics)
* Exposed for MetadataIndexManager to use as single source of truth
@ -2712,11 +2829,19 @@ export abstract class BaseStorage extends BaseStorageAdapter {
}
/**
* Rebuild type counts from actual storage
* Called when statistics are missing or inconsistent
* Ensures verbCountsByType is always accurate for reliable pagination
* Rebuild type counts from actual storage. Scans every shard, reads each
* entity's metadata, and reconstructs `nounCountsByType` / `verbCountsByType`
* from ground truth. Persists the corrected counts to `type-statistics.json`.
*
* Public so it can be triggered by `brainy inspect repair` and by the
* `brain.health()` reconciliation path. Called automatically by
* `loadTypeStatistics()` when the persisted state matches the poisoned
* 7.20.0–7.21.0 signature (all entities attributed to `'thing'`).
*
* For very large stores this is O(N) where N is total entities; intended
* for diagnostic / repair use, not on every init.
*/
protected async rebuildTypeCounts(): Promise<void> {
public async rebuildTypeCounts(): Promise<void> {
prodLog.info('[BaseStorage] Rebuilding type counts from storage...')
// Rebuild by scanning shards (0x00-0xFF) and reading metadata
@ -2788,16 +2913,63 @@ export abstract class BaseStorage extends BaseStorageAdapter {
}
/**
* Get noun type (type no longer needed for paths!)
* With ID-first paths, this is only used for internal statistics tracking.
* The actual type is stored in metadata and indexed by MetadataIndexManager.
* Resolve the `NounType` for a noun, used to attribute the entity to the
* correct slot in `nounCountsByType` (which backs `_system/type-statistics.json`
* and `brain.stats().entitiesByType`).
*
* Lookup order:
* 1. `nounTypeByIdCache`, populated by `saveNounMetadata_internal()`
* whenever a noun's metadata is written. This is the common path —
* add() saves metadata before the HNSW noun, so by the time we get
* here the cache is warm.
* 2. `'thing'` as a final fallback when metadata genuinely doesn't exist
* (a noun added without metadata — irregular but tolerated for back-
* compat). Logged so a real bug doesn't go unnoticed.
*
* Note this is intentionally synchronous because `saveNoun_internal()` and
* its callers are not async-friendly at this layer. For cases where only
* an id is known after a process restart and the cache is cold,
* `getNounTypeFromStorageAsync()` is available for callers that can await.
*/
protected getNounType(noun: HNSWNoun): NounType {
// Type cache removed - default to 'thing' for statistics
// The real type is in metadata, accessible via getNounMetadata(id)
const cached = this.nounTypeByIdCache.get(noun.id)
if (cached) return cached
// Cache miss on a noun we're about to write means metadata was not saved
// first. Brainy.add() always saves metadata before HNSW, so this only
// fires in unusual code paths (raw `storage.saveNoun()` without metadata).
prodLog.warn(
`[BaseStorage] getNounType: no type cached for noun ${noun.id}. ` +
`Type-statistics will attribute this entity to 'thing'. ` +
`Call saveNounMetadata() before saveNoun() to avoid stat drift.`
)
return 'thing'
}
/**
* Async variant of `getNounType()` that consults disk if the in-memory
* cache is cold (e.g. after a process restart with deferred work). Used by
* `rebuildTypeCounts()` and other callers that can await an IO. Returns
* `null` if metadata genuinely doesn't exist; callers decide whether to
* fall back to 'thing' or skip the entity.
*/
protected async getNounTypeFromStorageAsync(id: string): Promise<NounType | null> {
const cached = this.nounTypeByIdCache.get(id)
if (cached) return cached
try {
const metadataPath = getNounMetadataPath(id)
const metadata = await this.readWithInheritance(metadataPath)
if (metadata && (metadata as NounMetadata).noun) {
const type = (metadata as NounMetadata).noun as NounType
// Warm the cache so subsequent sync calls hit fast.
this.nounTypeByIdCache.set(id, type)
return type
}
} catch {
// Storage error — treat as unknown.
}
return null
}
/**
* Get verb type from verb object
* Verb type is a required field in HNSWVerb

283
src/utils/metadataIndex.ts
View file

@ -28,6 +28,7 @@ import {
import { EntityIdMapper } from './entityIdMapper.js'
import { RoaringBitmap32, roaringLibraryInitialize } from './roaring/index.js'
import { FieldTypeInference, FieldType } from './fieldTypeInference.js'
import { BrainyError } from '../errors/brainyError.js'
/**
* Fields whose values are stored in the sparse index as BUCKETED values
@ -150,11 +151,10 @@ export class MetadataIndexManager {
private chunkManager: ChunkManager
private chunkingStrategy: AdaptiveChunkingStrategy
// Deferred write tracking: accumulate chunk/sparse index writes during add/remove
// operations and flush them in a single concurrent batch at the end.
// This eliminates per-field sequential writes that cause cloud storage rate limiting.
private dirtyChunks = new Map<string, ChunkData>() // "field:chunkId" -> ChunkData
private dirtySparseIndices = new Map<string, SparseIndex>() // field -> SparseIndex
// (Removed in 7.22.0) `dirtyChunks` and `dirtySparseIndices` Maps —
// never populated since the sparse-index write path was deleted in 7.20.0
// (commit 11be039). The associated `flushDirtyMetadata()` no-op was also
// removed. Column store is the single source of truth for indexed writes.
// Roaring Bitmap Support
// EntityIdMapper for UUID ↔ integer conversion
@ -728,46 +728,31 @@ export class MetadataIndexManager {
this.unifiedCache.set(unifiedKey, sparseIndex, 'metadata', size, 200)
}
/**
* Flush all deferred chunk and sparse index writes accumulated during add/remove operations.
* Writes are deduplicated (same chunk/field written once even if updated multiple times)
* and executed concurrently via Promise.all for maximum throughput.
*/
private async flushDirtyMetadata(): Promise<void> {
if (this.dirtyChunks.size === 0 && this.dirtySparseIndices.size === 0) {
return
}
const promises: Promise<void>[] = []
// Save all dirty chunks (deduplicated — same chunk written once even if updated multiple times)
for (const [_key, chunk] of this.dirtyChunks) {
promises.push(this.chunkManager.saveChunk(chunk))
}
// Save all dirty sparse indices (deduplicated — same field's index written once)
for (const [field, sparseIndex] of this.dirtySparseIndices) {
promises.push(this.saveSparseIndex(field, sparseIndex))
}
// Execute all writes concurrently
await Promise.all(promises)
this.dirtyChunks.clear()
this.dirtySparseIndices.clear()
}
// splitChunkDeferred — DELETED. Only called from addToChunkedIndex (also deleted).
// flushDirtyMetadata — DELETED in 7.22.0. The dirtyChunks / dirtySparseIndices
// accumulators it drained were never populated after the 7.20.0 column-store
// refactor (commit 11be039). Column store flush happens in flush() directly.
/**
* Get IDs for a value using chunked sparse index with roaring bitmaps
* Now fully lazy-loaded via UnifiedCache (no local sparseIndices Map)
* Get IDs for a value using the legacy chunked sparse index.
*
* **This path is only for pre-7.20.0 workspaces** still being migrated to
* the column store. The write path for sparse indices was removed in
* commit `11be039` — new workspaces never get them.
*
* If neither the column store nor a sparse index covers the field, the
* function throws `BrainyError(FIELD_NOT_INDEXED)`. Returning `[]` for a
* genuinely unindexed field was the bug class `BR-FIND-WHERE-ZERO`
* tracked — a silent empty result indistinguishable from "the data
* really isn't there."
*/
private async getIdsFromChunks(field: string, value: any): Promise<string[]> {
// Load sparse index via UnifiedCache (lazy loading)
const sparseIndex = await this.loadSparseIndex(field)
if (!sparseIndex) {
return [] // No chunked index exists yet
// No column store match (we'd have returned in getIds()) AND no legacy
// sparse index for this field — the field is genuinely not indexed.
// Throw so find()-evaluation can log and translate to [].
throw BrainyError.fieldNotIndexed(field)
}
// Find candidate chunks using zone maps and bloom filters
@ -1323,7 +1308,19 @@ export class MetadataIndexManager {
const wordIdSets: Map<string, number>[] = []
for (const word of queryWords) {
const wordHash = this.hashWord(word)
const ids = await this.getIds('__words__', wordHash)
let ids: string[]
try {
ids = await this.getIds('__words__', wordHash)
} catch (err) {
// `__words__` is not yet indexed (e.g. no text content has been
// added). Treat as no matches and continue — text search against
// an empty workspace should return [], not throw.
if (err instanceof BrainyError && err.type === 'FIELD_NOT_INDEXED') {
ids = []
} else {
throw err
}
}
const idSet = new Map<string, number>()
for (const id of ids) {
idSet.set(id, 1)
@ -1585,6 +1582,20 @@ export class MetadataIndexManager {
}
}
/**
* Resolve a `where: { field: value }` clause to entity UUIDs.
*
* Lookup order:
* 1. **Column store** — the post-7.20.0 single source of truth. Fast.
* 2. **Legacy sparse index** — only consulted for pre-7.20.0 workspaces
* that haven't been migrated. Returns `[]` if no sparse data either.
*
* Throws `BrainyError(FIELD_NOT_INDEXED)` if the field has no entries in
* either store. Callers in find()-evaluation catch this and translate to
* an empty result with a logged warning. The throw aligns the production
* `find()` path with the `brain.explain()` diagnostic, so the silent-
* empty bug class (BR-FIND-WHERE-ZERO) is no longer possible.
*/
async getIds(field: string, value: any): Promise<string[]> {
// Track exact query for field statistics
if (this.fieldStats.has(field)) {
@ -1600,7 +1611,9 @@ export class MetadataIndexManager {
return this.idMapper.intsIterableToUuids(bitmap)
}
// Fallback: sparse index scan (legacy path during migration)
// Fallback: sparse index scan (legacy path during migration). If neither
// store has the field, throw FIELD_NOT_INDEXED so the caller knows it's a
// genuine "no such index" rather than "the value isn't there".
return await this.getIdsFromChunks(field, value)
}
@ -1780,13 +1793,23 @@ export class MetadataIndexManager {
// Process field filters with range support
const idSets: string[][] = []
// Capture field-not-indexed warnings so we log once per find() call,
// not once per AND-clause inside it.
const unindexedFields: string[] = []
for (const [field, condition] of Object.entries(filter)) {
// Skip logical operators
if (field === 'allOf' || field === 'anyOf' || field === 'not') continue
let fieldResults: string[] = []
try {
// The block below evaluates one field clause. If `getIds()` throws
// FIELD_NOT_INDEXED (no column-store and no legacy sparse index for
// this field), we treat the clause as matching zero entities. This
// makes the production `find()` path consistent with the
// `brain.explain()` diagnostic: an unindexed field returns no
// results AND logs a warning, instead of silently returning [].
if (condition && typeof condition === 'object' && !Array.isArray(condition)) {
// Handle Brainy Field Operators (canonical operators defined)
// See docs/api/README.md for complete operator reference
@ -1925,16 +1948,38 @@ export class MetadataIndexManager {
// Direct value match (shorthand for 'eq' operator)
fieldResults = await this.getIds(field, condition)
}
} catch (err) {
if (err instanceof BrainyError && err.type === 'FIELD_NOT_INDEXED') {
unindexedFields.push(field)
fieldResults = []
} else {
throw err
}
}
if (fieldResults.length > 0) {
idSets.push(fieldResults)
} else {
// If any field has no matches, intersection will be empty
if (unindexedFields.length > 0) {
prodLog.warn(
`[brainy] find() where-clause referenced unindexed field(s): ` +
`${unindexedFields.join(', ')}. Returning []. Use ` +
`brain.explain({ where: {...} }) for diagnostics.`
)
}
return []
}
}
if (idSets.length === 0) return []
if (unindexedFields.length > 0) {
prodLog.warn(
`[brainy] find() where-clause referenced unindexed field(s) ` +
`${unindexedFields.join(', ')}; their clauses contributed no rows. ` +
`Use brain.explain({ where: {...} }) for diagnostics.`
)
}
if (idSets.length === 1) return idSets[0]
// Set-based intersection O(n) — start with smallest set for optimal perf
@ -2212,21 +2257,38 @@ export class MetadataIndexManager {
// Handle regular field filters
const criteria = this.convertFilterToCriteria(filter)
const idSets: string[][] = []
const unindexedFields: string[] = []
for (const { field, values } of criteria) {
const unionIds = new Set<string>()
for (const value of values) {
const ids = await this.getIds(field, value)
ids.forEach(id => unionIds.add(id))
try {
for (const value of values) {
const ids = await this.getIds(field, value)
ids.forEach(id => unionIds.add(id))
}
} catch (err) {
if (err instanceof BrainyError && err.type === 'FIELD_NOT_INDEXED') {
unindexedFields.push(field)
// Treat as no matches and continue — same semantics as the main
// getIdsForFilter() path.
} else {
throw err
}
}
idSets.push(Array.from(unionIds))
}
if (unindexedFields.length > 0) {
prodLog.warn(
`[brainy] find() where-clause referenced unindexed field(s) ` +
`${unindexedFields.join(', ')}; their clauses contributed no rows.`
)
}
if (idSets.length === 0) return []
if (idSets.length === 1) return idSets[0]
// Intersection of all field criteria (implicit $and)
return idSets.reduce((intersection, currentSet) =>
return idSets.reduce((intersection, currentSet) =>
intersection.filter(id => currentSet.includes(id))
)
}
@ -2244,9 +2306,6 @@ export class MetadataIndexManager {
* NOTE: Sparse indices are flushed immediately in add/remove operations
*/
async flush(): Promise<void> {
// Flush any deferred chunk/sparse writes first
await this.flushDirtyMetadata()
// Always save field registry — even with no dirty fields. This tiny file
// (list of field names) is the critical link that init() needs to discover
// persisted indices. Without it, the index appears empty after restart.
@ -2776,54 +2835,70 @@ export class MetadataIndexManager {
}
/**
* Get index statistics with enhanced counting information
* Sparse indices now lazy-loaded via UnifiedCache
* Note: This method may load sparse indices to calculate stats
* Get index statistics.
*
* Source-of-truth precedence (post-7.20.0 column-store-first architecture):
* 1. **EntityIdMapper** — `idMapper.size` is the canonical entity count.
* Every indexed entity gets a UUID→int mapping; nothing else is
* consistent across instances.
* 2. **ColumnStore** — `getIndexedFields()` is the canonical list of
* indexed fields. `getFieldSizeSummary()` provides segment / tail
* bookkeeping per field.
* 3. **Legacy sparse-index registry** — only for pre-7.20.0 workspaces
* whose data hasn't been migrated. `getPersistedFieldList()` may know
* fields the column store doesn't yet, so we union them in.
*
* Prior implementation read from `this.fieldIndexes` + lazy-loaded sparse
* indices, which silently returned `0` entries for any workspace written
* after sparse-index writes were deleted in commit `11be039`. That defect
* is what `BR-FIND-WHERE-ZERO` tracked.
*/
async getStats(): Promise<MetadataIndexStats> {
const entityCount = this.idMapper.size
// Field set: union of column-store fields and any legacy sparse-index
// fields registered on disk. Exclude the `__words__` text index by
// convention (it's not a metadata field in the public sense).
const fields = new Set<string>()
let totalEntries = 0
let totalIds = 0
// Collect stats from metadata field indexes only (excludes __words__ keyword index)
for (const field of this.fieldIndexes.keys()) {
if (field === '__words__') continue // Keyword index not included in metadata stats
fields.add(field)
// Load sparse index to count entries (may trigger lazy load)
const sparseIndex = await this.loadSparseIndex(field)
if (sparseIndex) {
// Count entries and IDs from all chunks
for (const chunkId of sparseIndex.getAllChunkIds()) {
const chunk = await this.chunkManager.loadChunk(field, chunkId)
if (chunk) {
totalEntries += chunk.entries.size
for (const ids of chunk.entries.values()) {
totalIds += ids.size
}
}
}
if (this.columnStore) {
for (const f of this.columnStore.getIndexedFields()) {
if (f !== '__words__') fields.add(f)
}
}
// Legacy fallback: pre-7.20.0 workspaces may have sparse-index registry
// entries the column store doesn't know about yet. Surfacing them in the
// field list lets the rest of the system migrate them on read.
try {
const legacyFields = await this.getPersistedFieldList()
for (const f of legacyFields) {
if (f !== '__words__') fields.add(f)
}
} catch {
// Registry missing — nothing to add.
}
// Sanity check for index corruption (only metadata fields, not __words__ keyword index)
const entityCount = this.idMapper.size
if (entityCount > 0) {
const avgIdsPerEntity = totalIds / entityCount
if (avgIdsPerEntity > 100) {
prodLog.warn(
`⚠️ Metadata index may be corrupted: ${avgIdsPerEntity.toFixed(1)} avg entries/entity (expected ~30). ` +
`Try running brain.index.clearAllIndexData() followed by brain.index.rebuild() to fix.`
)
// `totalEntries` semantically means "distinct entities tracked by this
// index". That's `idMapper.size`. `totalIds` is the sum of all
// (field, value) → entityId postings — proxied by the segment/tail size
// summary so we don't have to scan every bitmap.
let totalIds = 0
if (this.columnStore) {
for (const summary of this.columnStore.getFieldSizeSummary()) {
if (summary.field === '__words__') continue
totalIds += summary.tailSize
// Segment count is a proxy; for a coarser-grained number we'd open
// each segment cursor. Avoided here because stats() is on the hot
// path for `brain.stats()` / health checks.
totalIds += summary.segmentCount * 1 // segments contribute at least 1 posting
}
}
return {
totalEntries,
totalEntries: entityCount,
totalIds,
fieldsIndexed: Array.from(fields),
fieldsIndexed: Array.from(fields).sort(),
lastRebuild: Date.now(),
indexSize: totalEntries * 100 // rough estimate
indexSize: entityCount * 100 // rough estimate
}
}
@ -2996,9 +3071,9 @@ export class MetadataIndexManager {
await this.addToIndex(noun.id, metadata, true, true)
localCount++
// Periodic safety flush every 5000 entities to cap memory during rebuild
if (localCount % 5000 === 0) {
await this.flushDirtyMetadata()
}
// (Periodic safety flush removed in 7.22.0 — the underlying
// flushDirtyMetadata was a no-op since the 7.20.0 column-store
// refactor. Column store handles its own flushing.)
}
}
@ -3092,10 +3167,6 @@ export class MetadataIndexManager {
await this.yieldToEventLoop()
totalNounsProcessed += result.items.length
// Periodic safety flush every 5000 entities to cap memory during rebuild
if (totalNounsProcessed % 5000 === 0) {
await this.flushDirtyMetadata()
}
hasMoreNouns = result.hasMore
nounOffset += nounLimit
@ -3150,10 +3221,7 @@ export class MetadataIndexManager {
if (metadata) {
await this.addToIndex(verb.id, metadata, true, true)
verbLocalCount++
// Periodic safety flush every 5000 entities to cap memory during rebuild
if (verbLocalCount % 5000 === 0) {
await this.flushDirtyMetadata()
}
// (Periodic safety flush removed in 7.22.0 — see noun loop above.)
}
}
@ -3242,10 +3310,6 @@ export class MetadataIndexManager {
await this.yieldToEventLoop()
totalVerbsProcessed += result.items.length
// Periodic safety flush every 5000 entities to cap memory during rebuild
if (totalVerbsProcessed % 5000 === 0) {
await this.flushDirtyMetadata()
}
hasMoreVerbs = result.hasMore
verbOffset += verbLimit
@ -3262,11 +3326,8 @@ export class MetadataIndexManager {
}
}
// Flush remaining dirty chunks/sparse indices accumulated during rebuild
// (deferWrites=true prevented per-entity flushes, so dirty data accumulated)
await this.flushDirtyMetadata()
// Flush to storage with final yield
// Flush to storage with final yield. The column store's flush() handles
// tail-buffer-to-segment promotion + manifest persistence.
prodLog.debug('💾 Flushing metadata index to storage...')
await this.flush()
await this.yieldToEventLoop()