CRITICAL FEATURE: Automatic detection and adaptation for storage service throttling
(GCS, AWS S3, Cloudflare R2, etc.) that can prevent service initialization.
Automatic Throttling Detection:
- Detect 429 (Too Many Requests) and 503 (Service Unavailable) responses
- Recognize throttling keywords in error messages
- Identify connection resets and timeouts from rate limiting
Smart Retry Logic:
- Exponential backoff starting at 1s up to 30s maximum
- Automatic recovery detection when throttling clears
- Smart delays that adapt to current throttling status
- Enhanced error logging with throttling context
Adaptive Behavior:
- Higher error rates trigger longer delays
- Recent throttling history influences delay timing
- Prevents cascading throttling through intelligent spacing
- Works universally across all S3-compatible storage services
Expected Impact:
- Services will automatically adapt to GCS/S3 rate limiting
- Initialization will complete even under throttling conditions
- Clear logging shows throttling status and recovery
- No manual intervention needed for throttling scenarios
Addresses potential GCS S3 API throttling that could explain
persistent initialization issues despite socket exhaustion fixes.
David Snelling
cfbadec330