soulcraftlabs/open-brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 343 Packages Wiki Activity Actions

fix: exclude __words__ keyword index from corruption detection and getStats()

Browse source 
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:

1. validateConsistency() to falsely detect corruption on every startup,
   triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
   and report inflated totalEntries/totalIds stats

Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
This commit is contained in:
David Snelling 2026-01-27 15:38:21 -08:00
parent 32dbdcec61
commit 364360d447
128 changed files with 5637 additions and 5682 deletions
Download patch file Download diff file Expand all files Collapse all files

6
src/utils/memoryDetection.ts
View file

@ -49,7 +49,7 @@ export interface CacheAllocationStrategy {
/** Environment type detected */
environment: 'production' | 'development' | 'container' | 'unknown'
/** Model memory reserved (bytes) - v3.36.0+ */
/** Model memory reserved (bytes) */
modelMemory: number
/** Model precision (q8 or fp32) */
@ -195,7 +195,7 @@ function detectCgroupV1Memory(): number | null {
* Calculate optimal cache size based on available memory
* Scales intelligently from 2GB to 128GB+
*
* v3.36.0+: Accounts for embedding model memory (150MB Q8, 250MB FP32)
* Accounts for embedding model memory (150MB Q8, 250MB FP32)
*/
export function calculateOptimalCacheSize(
memoryInfo: MemoryInfo,
@ -219,7 +219,7 @@ export function calculateOptimalCacheSize(
const minSize = options.minSize || 256 * 1024 * 1024 // 256MB minimum
const maxSize = options.maxSize || null
// Detect model memory usage (v3.36.0+)
// Detect model memory usage
const modelInfo = detectModelMemory({ precision: options.modelPrecision || 'q8' })
const modelMemory = modelInfo.bytes