test(budgets): iron-honest wall-clock budgets — 3x the worst honest-iron measurement
Seven micro-budget tests were calibrated on one fast desktop and failed on
other honest iron with zero functional failures (bisect-proven pre-existing;
David-waived for 10.1/10.2 with this recalibration filed as the cure). Every
budget is now at least 3x the worst measurement observed across three
machines, each with a comment naming its calibration basis; the find-unified
micro-comparison of two sub-millisecond timings becomes a ratio assertion
(absolute equality of microsecond pairs can never be stable). The
inference-bound trim-history correctness test gets a timeout covering its
slowest observed run (174s) — its assertions are exact and untouched.
These remain order-of-magnitude guards; real perf enforcement lives in the
dedicated perf lanes with iron-specific budgets, per the gate-speed standard.
Known non-test artifact, documented not hidden: on slow-inference machines a
minutes-long awaited-embed loop can trip vitest's worker-RPC 60s tolerance
('Timeout calling onTaskUpdate') — all tests pass, vitest exits 1 on the
unhandled orchestration error. The CI lanes on faster iron exit clean; if a
lane ever trips it, the test moves to deterministic embeddings (its
assertions are size-bookkeeping, not embedding quality).
This commit is contained in:
parent
292e7c0406
commit
314e0e6c29
7 changed files with 46 additions and 17 deletions
|
|
@ -456,7 +456,9 @@ describe('Brainy Batch Operations', () => {
|
|||
// Verify batch operation completed successfully
|
||||
// Note: Performance can vary based on system load and embedding generation
|
||||
expect(batchIds).toHaveLength(itemCount)
|
||||
expect(batchTime).toBeLessThan(5000) // Reasonable timeout for 50 items
|
||||
// order-of-magnitude guard: worst honest-iron measurement 11.9s (CPU-only
|
||||
// inference, 32-core box), 3x headroom for 50-item batch
|
||||
expect(batchTime).toBeLessThan(40000)
|
||||
|
||||
console.log(`Individual: ${individualTime}ms, Batch: ${batchTime}ms`)
|
||||
if (batchTime < individualTime) {
|
||||
|
|
@ -510,7 +512,9 @@ describe('Brainy Batch Operations', () => {
|
|||
|
||||
const totalTime = Date.now() - startTime
|
||||
|
||||
expect(totalTime).toBeLessThan(3000) // v5.4.0: Type-first storage takes longer
|
||||
// order-of-magnitude guard: worst honest-iron measurement 6652ms
|
||||
// (mixed batch under CPU-only inference), 3x headroom
|
||||
expect(totalTime).toBeLessThan(20000)
|
||||
|
||||
// Verify final state
|
||||
const remaining = await brain.get(initialIds[0])
|
||||
|
|
@ -556,7 +560,12 @@ describe('Brainy Batch Operations', () => {
|
|||
// Might throw if there's a limit
|
||||
expect(error).toBeDefined()
|
||||
}
|
||||
}, 60000)
|
||||
// order-of-magnitude guard: this test batches 20x the item count of the
|
||||
// sibling "perform better" test above (worst measured 11.9s for 50
|
||||
// items on CPU-only honest iron); the prior 60s timeout was itself
|
||||
// observed being hit, so this is 3x that floor rather than a scaled
|
||||
// extrapolation, to leave real headroom for run-to-run variance
|
||||
}, 180000)
|
||||
|
||||
it('should provide meaningful error messages', async () => {
|
||||
try {
|
||||
|
|
|
|||
Reference in a new issue