David Snelling
52782898a3
Progressive intervals adjust dynamically based on current entity count (not total), making them work for both known and unknown totals. **Key Features:** - 0-999 entities: Flush every 100 (frequent early updates for UX) - 1K-9.9K: Flush every 1000 (balanced performance) - 10K+: Flush every 5000 (minimal overhead ~0.3%) **Benefits:** - Works with known totals (file imports) - Works with unknown totals (streaming APIs, database cursors) - Adapts automatically as import grows - Zero configuration required **Implementation:** - Replaced adaptive intervals (requires total count) with progressive - Added interval transition logging for observability - Enhanced documentation to highlight engineering sophistication - Final flush with statistics reporting **Documentation:** - Added "Engineering Insight" section showcasing advanced approach - Updated all interval references from "adaptive" to "progressive" - Added comprehensive examples in streaming-imports.md Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| api | ||
| augmentations/intelligentImport | ||
| benchmarks | ||
| comprehensive | ||
| configs | ||
| fixtures/import | ||
| helpers | ||
| integration | ||
| performance | ||
| scripts | ||
| unit | ||
| vfs | ||
| brainy-3.test.ts | ||
| critical-error-handling.test.ts | ||
| critical-neural-validation.test.ts | ||
| critical-performance-benchmark.test.ts | ||
| distributed-demo.test.ts | ||
| model-loading.test.ts | ||
| opfs-storage.test.ts | ||
| package-size-breakdown.test.ts | ||
| package-size-limit.test.ts | ||
| setup-integration.ts | ||
| setup-unit.ts | ||
| setup.ts | ||
| streaming-pipeline.test.ts | ||
| type-utils.test.ts | ||
| typeAwareHNSWIndex.test.ts | ||
| typeAwareQueryPlanner.test.ts | ||