fix(8.0): drive query-cap off MemAvailable + floor auto-detected caps

The auto-detected query-limit cap was computed from os.freemem() (MemFree),
which excludes reclaimable page cache. On hosts that memory-map large index
files the cache holding those pages dominates RAM, so MemFree collapses to a
sliver and the cap cratered to its floor on perfectly healthy machines,
rejecting legitimate find() calls.

- getAvailableMemory() now prefers /proc/meminfo MemAvailable (counts
  reclaimable cache), falling back to os.freemem() off-Linux, then a 2 GB
  constant where no OS module is available.
- Auto-detected caps (container + free-memory tiers) are floored at
  MIN_AUTO_QUERY_LIMIT (10_000); a transient low reading can never throttle
  queries to a near-useless ceiling. Consumer-supplied maxQueryLimit /
  reservedQueryMemory bypass the floor — an explicit caller knows their box.

Also scrubs two stale comments referencing the removed cloud storage
adapters and the retired mmap-vector backend: 8.0's native vector provider
persists its own .dkann file via getBinaryBlobPath, so the old rootDirectory
hook does not apply on this line.

Tests: memoryLimits 26/26 (4 new floor regressions), unit 1398/1398,
find-limits + db-mvcc + api-parameter-validation 37/37.
This commit is contained in:
David Snelling 2026-06-16 09:01:45 -07:00
parent c605b34f98
commit b26d3d42b3
3 changed files with 117 additions and 17 deletions

View file

@ -115,17 +115,66 @@ describe('Memory Limits - Container Detection & Smart Calculation', () => {
})
it('should handle small containers gracefully', () => {
process.env.CLOUD_RUN_MEMORY = '512Mi' // Use 512MB instead of 128MB for realistic test
process.env.CLOUD_RUN_MEMORY = '512Mi'
const config = ValidationConfig.getInstance()
// 512MB * 0.25 = 128MB for queries
// 128MB / 100MB = 1.28 floor to 1 * 1000 = 1000 limit
expect(config.maxLimit).toBeGreaterThan(0)
// 512 MB × 0.25 = 128 MB for queries; at 25 KB / result that raw figure
// is ~5_242 → floor to 5 × 1000 = 5_000, which the MIN_AUTO_QUERY_LIMIT
// floor lifts to 10_000. A small container must never throttle queries to
// a near-useless ceiling (BRAINY-QUERYCAP-MISREAD).
expect(config.maxLimit).toBe(10000)
expect(config.limitBasis).toBe('containerMemory')
})
})
describe('Auto-cap floor (BRAINY-QUERYCAP-MISREAD regression)', () => {
// A transiently low memory reading once collapsed the auto-detected query
// cap to a near-useless ceiling on healthy hosts (the cap was driven off
// os.freemem()/MemFree, which excludes reclaimable page cache). The fix:
// drive detection off /proc/meminfo MemAvailable AND floor every
// *auto-detected* tier at MIN_AUTO_QUERY_LIMIT (10_000). Consumer-supplied
// limits bypass the floor — an explicit caller knows their box best.
it('floors a tiny container to 10_000, never below', () => {
// 50 MB × 0.25 = 12.5 MB for queries → raw cap rounds to 0; the floor
// must lift it to 10_000 rather than let it throttle to nothing.
process.env.MEMORY_LIMIT = String(50 * 1024 * 1024)
const config = ValidationConfig.getInstance()
expect(config.limitBasis).toBe('containerMemory')
expect(config.maxLimit).toBe(10000)
})
it('floors the free-memory tier at 10_000', () => {
// No container env → free-memory tier (now MemAvailable-based). Whatever
// the host reports, the auto cap is guaranteed never to fall below 10_000.
const config = ValidationConfig.getInstance()
expect(config.limitBasis).toBe('freeMemory')
expect(config.maxLimit).toBeGreaterThanOrEqual(10000)
})
it('does NOT floor an explicit maxQueryLimit below 10_000', () => {
// A consumer asking for 500 means 500 — the floor is for auto-detection.
const config = ValidationConfig.getInstance({ maxQueryLimit: 500 })
expect(config.limitBasis).toBe('override')
expect(config.maxLimit).toBe(500)
})
it('does NOT floor an explicit reservedQueryMemory below 10_000', () => {
// 100 MB / 25 KB = ~4_000 → kept as-is (no floor on the explicit tier).
const config = ValidationConfig.getInstance({
reservedQueryMemory: 100 * 1024 * 1024
})
expect(config.limitBasis).toBe('reservedMemory')
expect(config.maxLimit).toBe(4000)
})
})
describe('Configuration Overrides', () => {
it('should respect maxQueryLimit override', () => {
const config = ValidationConfig.getInstance({ maxQueryLimit: 50000 })