fix(8.0): drive query-cap off MemAvailable + floor auto-detected caps
The auto-detected query-limit cap was computed from os.freemem() (MemFree), which excludes reclaimable page cache. On hosts that memory-map large index files the cache holding those pages dominates RAM, so MemFree collapses to a sliver and the cap cratered to its floor on perfectly healthy machines, rejecting legitimate find() calls. - getAvailableMemory() now prefers /proc/meminfo MemAvailable (counts reclaimable cache), falling back to os.freemem() off-Linux, then a 2 GB constant where no OS module is available. - Auto-detected caps (container + free-memory tiers) are floored at MIN_AUTO_QUERY_LIMIT (10_000); a transient low reading can never throttle queries to a near-useless ceiling. Consumer-supplied maxQueryLimit / reservedQueryMemory bypass the floor — an explicit caller knows their box. Also scrubs two stale comments referencing the removed cloud storage adapters and the retired mmap-vector backend: 8.0's native vector provider persists its own .dkann file via getBinaryBlobPath, so the old rootDirectory hook does not apply on this line. Tests: memoryLimits 26/26 (4 new floor regressions), unit 1398/1398, find-limits + db-mvcc + api-parameter-validation 37/37.
This commit is contained in:
parent
c605b34f98
commit
b26d3d42b3
3 changed files with 117 additions and 17 deletions
|
|
@ -115,17 +115,66 @@ describe('Memory Limits - Container Detection & Smart Calculation', () => {
|
|||
})
|
||||
|
||||
it('should handle small containers gracefully', () => {
|
||||
process.env.CLOUD_RUN_MEMORY = '512Mi' // Use 512MB instead of 128MB for realistic test
|
||||
process.env.CLOUD_RUN_MEMORY = '512Mi'
|
||||
|
||||
const config = ValidationConfig.getInstance()
|
||||
|
||||
// 512MB * 0.25 = 128MB for queries
|
||||
// 128MB / 100MB = 1.28 floor to 1 * 1000 = 1000 limit
|
||||
expect(config.maxLimit).toBeGreaterThan(0)
|
||||
// 512 MB × 0.25 = 128 MB for queries; at 25 KB / result that raw figure
|
||||
// is ~5_242 → floor to 5 × 1000 = 5_000, which the MIN_AUTO_QUERY_LIMIT
|
||||
// floor lifts to 10_000. A small container must never throttle queries to
|
||||
// a near-useless ceiling (BRAINY-QUERYCAP-MISREAD).
|
||||
expect(config.maxLimit).toBe(10000)
|
||||
expect(config.limitBasis).toBe('containerMemory')
|
||||
})
|
||||
})
|
||||
|
||||
describe('Auto-cap floor (BRAINY-QUERYCAP-MISREAD regression)', () => {
|
||||
// A transiently low memory reading once collapsed the auto-detected query
|
||||
// cap to a near-useless ceiling on healthy hosts (the cap was driven off
|
||||
// os.freemem()/MemFree, which excludes reclaimable page cache). The fix:
|
||||
// drive detection off /proc/meminfo MemAvailable AND floor every
|
||||
// *auto-detected* tier at MIN_AUTO_QUERY_LIMIT (10_000). Consumer-supplied
|
||||
// limits bypass the floor — an explicit caller knows their box best.
|
||||
|
||||
it('floors a tiny container to 10_000, never below', () => {
|
||||
// 50 MB × 0.25 = 12.5 MB for queries → raw cap rounds to 0; the floor
|
||||
// must lift it to 10_000 rather than let it throttle to nothing.
|
||||
process.env.MEMORY_LIMIT = String(50 * 1024 * 1024)
|
||||
|
||||
const config = ValidationConfig.getInstance()
|
||||
|
||||
expect(config.limitBasis).toBe('containerMemory')
|
||||
expect(config.maxLimit).toBe(10000)
|
||||
})
|
||||
|
||||
it('floors the free-memory tier at 10_000', () => {
|
||||
// No container env → free-memory tier (now MemAvailable-based). Whatever
|
||||
// the host reports, the auto cap is guaranteed never to fall below 10_000.
|
||||
const config = ValidationConfig.getInstance()
|
||||
|
||||
expect(config.limitBasis).toBe('freeMemory')
|
||||
expect(config.maxLimit).toBeGreaterThanOrEqual(10000)
|
||||
})
|
||||
|
||||
it('does NOT floor an explicit maxQueryLimit below 10_000', () => {
|
||||
// A consumer asking for 500 means 500 — the floor is for auto-detection.
|
||||
const config = ValidationConfig.getInstance({ maxQueryLimit: 500 })
|
||||
|
||||
expect(config.limitBasis).toBe('override')
|
||||
expect(config.maxLimit).toBe(500)
|
||||
})
|
||||
|
||||
it('does NOT floor an explicit reservedQueryMemory below 10_000', () => {
|
||||
// 100 MB / 25 KB = ~4_000 → kept as-is (no floor on the explicit tier).
|
||||
const config = ValidationConfig.getInstance({
|
||||
reservedQueryMemory: 100 * 1024 * 1024
|
||||
})
|
||||
|
||||
expect(config.limitBasis).toBe('reservedMemory')
|
||||
expect(config.maxLimit).toBe(4000)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Configuration Overrides', () => {
|
||||
it('should respect maxQueryLimit override', () => {
|
||||
const config = ValidationConfig.getInstance({ maxQueryLimit: 50000 })
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue