feat: add native Google Cloud Storage adapter with ADC support
Implement native @google-cloud/storage adapter for better performance
and easier authentication in Cloud Run/GCE environments.
Features:
- Application Default Credentials (ADC) for zero-config auth
- Service account authentication (keyFilename, credentials)
- HMAC fallback for backward compatibility
- Full UUID-based sharding preservation
- Write buffers for high-volume mode
- Multi-level caching and adaptive backpressure
- Complete feature parity with S3-compatible adapter
Benefits over S3-compatible GCS:
- No HMAC key management required
- Native SDK performance optimizations
- Automatic authentication in Cloud Run/GCE
- Simpler configuration
Configuration:
- type: 'gcs-native'
- gcsNativeStorage: { bucketName, keyFilename?, credentials? }
- Zero data migration required (same path structure)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
8a9cf1bd51
commit
e2aa8e3253
6 changed files with 2824 additions and 35 deletions
|
|
@ -157,6 +157,7 @@
|
|||
},
|
||||
"dependencies": {
|
||||
"@aws-sdk/client-s3": "^3.540.0",
|
||||
"@google-cloud/storage": "^7.14.0",
|
||||
"@huggingface/transformers": "^3.7.2",
|
||||
"boxen": "^8.0.1",
|
||||
"chalk": "^5.3.0",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue