feat: add native Google Cloud Storage adapter with ADC support

Implement native @google-cloud/storage adapter for better performance
and easier authentication in Cloud Run/GCE environments.

Features:
- Application Default Credentials (ADC) for zero-config auth
- Service account authentication (keyFilename, credentials)
- HMAC fallback for backward compatibility
- Full UUID-based sharding preservation
- Write buffers for high-volume mode
- Multi-level caching and adaptive backpressure
- Complete feature parity with S3-compatible adapter

Benefits over S3-compatible GCS:
- No HMAC key management required
- Native SDK performance optimizations
- Automatic authentication in Cloud Run/GCE
- Simpler configuration

Configuration:
- type: 'gcs-native'
- gcsNativeStorage: { bucketName, keyFilename?, credentials? }
- Zero data migration required (same path structure)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-10-08 14:08:43 -07:00
parent 8a9cf1bd51
commit e2aa8e3253
6 changed files with 2824 additions and 35 deletions

View file

@ -157,6 +157,7 @@
},
"dependencies": {
"@aws-sdk/client-s3": "^3.540.0",
"@google-cloud/storage": "^7.14.0",
"@huggingface/transformers": "^3.7.2",
"boxen": "^8.0.1",
"chalk": "^5.3.0",