feat: add native Google Cloud Storage adapter with ADC support

Implement native @google-cloud/storage adapter for better performance
and easier authentication in Cloud Run/GCE environments.

Features:
- Application Default Credentials (ADC) for zero-config auth
- Service account authentication (keyFilename, credentials)
- HMAC fallback for backward compatibility
- Full UUID-based sharding preservation
- Write buffers for high-volume mode
- Multi-level caching and adaptive backpressure
- Complete feature parity with S3-compatible adapter

Benefits over S3-compatible GCS:
- No HMAC key management required
- Native SDK performance optimizations
- Automatic authentication in Cloud Run/GCE
- Simpler configuration

Configuration:
- type: 'gcs-native'
- gcsNativeStorage: { bucketName, keyFilename?, credentials? }
- Zero data migration required (same path structure)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
David Snelling 2025-10-08 14:08:43 -07:00
parent 8a9cf1bd51
commit e2aa8e3253
6 changed files with 2824 additions and 35 deletions

View file

@ -548,6 +548,185 @@ const valid = await security.verify(data, signature, secret)
---
## Storage Configuration
Brainy supports multiple storage backends for different deployment scenarios.
### Storage Types
#### Memory Storage (Default)
Fast in-memory storage, ideal for testing and development.
```typescript
const brain = new Brainy({
storage: { type: 'memory' }
})
```
#### File System Storage
Persistent local storage using the filesystem.
```typescript
const brain = new Brainy({
storage: {
type: 'filesystem',
rootDirectory: './brainy-data'
}
})
```
#### Amazon S3 Storage
Scalable cloud storage with S3.
```typescript
const brain = new Brainy({
storage: {
type: 's3',
s3Storage: {
bucketName: 'my-bucket',
region: 'us-east-1',
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
}
}
})
```
#### Cloudflare R2 Storage
Scalable cloud storage with Cloudflare R2 (S3-compatible).
```typescript
const brain = new Brainy({
storage: {
type: 'r2',
r2Storage: {
bucketName: 'my-bucket',
accountId: process.env.CF_ACCOUNT_ID,
accessKeyId: process.env.R2_ACCESS_KEY_ID,
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY
}
}
})
```
#### Google Cloud Storage (S3-Compatible)
GCS using HMAC keys for S3-compatible access.
```typescript
const brain = new Brainy({
storage: {
type: 'gcs',
gcsStorage: {
bucketName: 'my-bucket',
region: 'us-central1',
accessKeyId: process.env.GCS_ACCESS_KEY_ID,
secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY,
endpoint: 'https://storage.googleapis.com'
}
}
})
```
#### Google Cloud Storage (Native SDK) 🆕
**Recommended for GCS deployments.** Uses native `@google-cloud/storage` SDK with automatic authentication.
**Key Benefits:**
- ✅ Application Default Credentials (ADC) - Zero config in Cloud Run/GCE
- ✅ Better performance with native GCS optimizations
- ✅ No HMAC key management required
- ✅ Automatic service account integration
**With Application Default Credentials (Cloud Run/GCE):**
```typescript
const brain = new Brainy({
storage: {
type: 'gcs-native',
gcsNativeStorage: {
bucketName: 'my-bucket'
// No credentials needed - ADC automatic!
}
}
})
```
**With Service Account Key File:**
```typescript
const brain = new Brainy({
storage: {
type: 'gcs-native',
gcsNativeStorage: {
bucketName: 'my-bucket',
keyFilename: '/path/to/service-account.json'
}
}
})
```
**With Service Account Credentials Object:**
```typescript
const brain = new Brainy({
storage: {
type: 'gcs-native',
gcsNativeStorage: {
bucketName: 'my-bucket',
credentials: {
client_email: 'service@project.iam.gserviceaccount.com',
private_key: process.env.GCS_PRIVATE_KEY
}
}
}
})
```
### Storage Features
All storage adapters support:
- ✅ **UUID-based sharding** - 256 buckets (00-ff) for scalability
- ✅ **Pagination** - Efficient cursor-based pagination across shards
- ✅ **Statistics** - O(1) count operations
- ✅ **Caching** - Multi-level cache for performance
- ✅ **Backpressure** - Automatic flow control
- ✅ **Throttling detection** - Adaptive retry on rate limits
### Migration from HMAC to Native GCS
If you're currently using `type: 'gcs'` with HMAC keys, migrating to `type: 'gcs-native'` is straightforward:
**Before (HMAC):**
```typescript
const brain = new Brainy({
storage: {
type: 'gcs',
gcsStorage: {
bucketName: 'my-bucket',
accessKeyId: process.env.GCS_ACCESS_KEY_ID,
secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY
}
}
})
```
**After (Native with ADC):**
```typescript
const brain = new Brainy({
storage: {
type: 'gcs-native',
gcsNativeStorage: {
bucketName: 'my-bucket'
// ADC handles authentication automatically
}
}
})
```
**Data Migration:**
No data migration required! Both adapters use the same path structure:
- `entities/nouns/vectors/{shard}/{id}.json`
- `entities/nouns/metadata/{shard}/{id}.json`
- `entities/verbs/vectors/{shard}/{id}.json`
---
## Configuration API
Access via: `const config = await brain.config()`

579
package-lock.json generated
View file

@ -10,6 +10,7 @@
"license": "MIT",
"dependencies": {
"@aws-sdk/client-s3": "^3.540.0",
"@google-cloud/storage": "^7.14.0",
"@huggingface/transformers": "^3.7.2",
"boxen": "^8.0.1",
"chalk": "^5.3.0",
@ -1689,6 +1690,111 @@
"node": "^18.18.0 || ^20.9.0 || >=21.1.0"
}
},
"node_modules/@google-cloud/paginator": {
"version": "5.0.2",
"resolved": "https://registry.npmjs.org/@google-cloud/paginator/-/paginator-5.0.2.tgz",
"integrity": "sha512-DJS3s0OVH4zFDB1PzjxAsHqJT6sKVbRwwML0ZBP9PbU7Yebtu/7SWMRzvO2J3nUi9pRNITCfu4LJeooM2w4pjg==",
"license": "Apache-2.0",
"dependencies": {
"arrify": "^2.0.0",
"extend": "^3.0.2"
},
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/@google-cloud/paginator/node_modules/arrify": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/arrify/-/arrify-2.0.1.tgz",
"integrity": "sha512-3duEwti880xqi4eAMN8AyR4a0ByT90zoYdLlevfrvU43vb0YZwZVfxOgxWrLXXXpyugL0hNZc9G6BiB5B3nUug==",
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/@google-cloud/projectify": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@google-cloud/projectify/-/projectify-4.0.0.tgz",
"integrity": "sha512-MmaX6HeSvyPbWGwFq7mXdo0uQZLGBYCwziiLIGq5JVX+/bdI3SAq6bP98trV5eTWfLuvsMcIC1YJOF2vfteLFA==",
"license": "Apache-2.0",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/@google-cloud/promisify": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@google-cloud/promisify/-/promisify-4.0.0.tgz",
"integrity": "sha512-Orxzlfb9c67A15cq2JQEyVc7wEsmFBmHjZWZYQMUyJ1qivXyMwdyNOs9odi79hze+2zqdTtu1E19IM/FtqZ10g==",
"license": "Apache-2.0",
"engines": {
"node": ">=14"
}
},
"node_modules/@google-cloud/storage": {
"version": "7.17.2",
"resolved": "https://registry.npmjs.org/@google-cloud/storage/-/storage-7.17.2.tgz",
"integrity": "sha512-6xN0KNO8L/LIA5zu3CJwHkJiB6n65eykBLOb0E+RooiHYgX8CSao6lvQiKT9TBk2gL5g33LL3fmhDodZnt56rw==",
"license": "Apache-2.0",
"dependencies": {
"@google-cloud/paginator": "^5.0.0",
"@google-cloud/projectify": "^4.0.0",
"@google-cloud/promisify": "<4.1.0",
"abort-controller": "^3.0.0",
"async-retry": "^1.3.3",
"duplexify": "^4.1.3",
"fast-xml-parser": "^4.4.1",
"gaxios": "^6.0.2",
"google-auth-library": "^9.6.3",
"html-entities": "^2.5.2",
"mime": "^3.0.0",
"p-limit": "^3.0.1",
"retry-request": "^7.0.0",
"teeny-request": "^9.0.0",
"uuid": "^8.0.0"
},
"engines": {
"node": ">=14"
}
},
"node_modules/@google-cloud/storage/node_modules/fast-xml-parser": {
"version": "4.5.3",
"resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-4.5.3.tgz",
"integrity": "sha512-RKihhV+SHsIUGXObeVy9AXiBbFwkVk7Syp8XgwN5U3JV416+Gwp/GO9i0JYKmikykgz/UHRrrV4ROuZEo/T0ig==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/NaturalIntelligence"
}
],
"license": "MIT",
"dependencies": {
"strnum": "^1.1.1"
},
"bin": {
"fxparser": "src/cli/cli.js"
}
},
"node_modules/@google-cloud/storage/node_modules/strnum": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/strnum/-/strnum-1.1.2.tgz",
"integrity": "sha512-vrN+B7DBIoTTZjnPNewwhx6cBA/H+IS7rfW68n7XxC1y7uoiGQBxaKzqucGUgavX15dJgiGztLJ8vxuEzwqBdA==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/NaturalIntelligence"
}
],
"license": "MIT"
},
"node_modules/@google-cloud/storage/node_modules/uuid": {
"version": "8.3.2",
"resolved": "https://registry.npmjs.org/uuid/-/uuid-8.3.2.tgz",
"integrity": "sha512-+NYs2QeMWy+GWFOEm9xnn6HCDp0l7QBD7ml8zLUmJ+93Q5NF0NocErnwkTkXVFNiX3/fpC6afS8Dhb/gz7R7eg==",
"license": "MIT",
"bin": {
"uuid": "dist/bin/uuid"
}
},
"node_modules/@grpc/grpc-js": {
"version": "1.13.4",
"resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.13.4.tgz",
@ -4303,6 +4409,21 @@
"testcontainers": "^11.5.1"
}
},
"node_modules/@tootallnate/once": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/@tootallnate/once/-/once-2.0.0.tgz",
"integrity": "sha512-XCuKFP5PS55gnMVu3dty8KPatLqUoy/ZYzDzAGCQ8JNFCkLXzmI7vNHCR+XpbZaMWQK/vQubr7PkYq8g470J/A==",
"license": "MIT",
"engines": {
"node": ">= 10"
}
},
"node_modules/@types/caseless": {
"version": "0.12.5",
"resolved": "https://registry.npmjs.org/@types/caseless/-/caseless-0.12.5.tgz",
"integrity": "sha512-hWtVTC2q7hc7xZ/RLbxapMvDMgUnDvKvMOpKal4DrMyfGBUfB1oKaZlIRr6mJL+If3bAP6sV/QneGzF6tJjZDg==",
"license": "MIT"
},
"node_modules/@types/chai": {
"version": "5.2.2",
"resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.2.tgz",
@ -4396,6 +4517,18 @@
"license": "MIT",
"optional": true
},
"node_modules/@types/request": {
"version": "2.48.13",
"resolved": "https://registry.npmjs.org/@types/request/-/request-2.48.13.tgz",
"integrity": "sha512-FGJ6udDNUCjd19pp0Q3iTiDkwhYup7J8hpMW9c4k53NrccQFFWKRho6hvtPPEhnXWKvukfwAlB6DbDz4yhH5Gg==",
"license": "MIT",
"dependencies": {
"@types/caseless": "*",
"@types/node": "*",
"@types/tough-cookie": "*",
"form-data": "^2.5.5"
}
},
"node_modules/@types/resolve": {
"version": "1.20.2",
"resolved": "https://registry.npmjs.org/@types/resolve/-/resolve-1.20.2.tgz",
@ -4440,6 +4573,12 @@
"dev": true,
"license": "MIT"
},
"node_modules/@types/tough-cookie": {
"version": "4.0.5",
"resolved": "https://registry.npmjs.org/@types/tough-cookie/-/tough-cookie-4.0.5.tgz",
"integrity": "sha512-/Ad8+nIOV7Rl++6f1BdKxFSMgmoqEoYbHRpPcx3JEfv8VRsQe9Z4mCXeJBzxs7mbHY/XOZZuXlRNfhpVPbs6ZA==",
"license": "MIT"
},
"node_modules/@types/trusted-types": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/@types/trusted-types/-/trusted-types-2.0.7.tgz",
@ -4871,7 +5010,6 @@
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz",
"integrity": "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg==",
"dev": true,
"license": "MIT",
"dependencies": {
"event-target-shim": "^5.0.0"
@ -4920,6 +5058,15 @@
"node": ">=0.8"
}
},
"node_modules/agent-base": {
"version": "7.1.4",
"resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz",
"integrity": "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==",
"license": "MIT",
"engines": {
"node": ">= 14"
}
},
"node_modules/ajv": {
"version": "6.12.6",
"resolved": "https://registry.npmjs.org/ajv/-/ajv-6.12.6.tgz",
@ -5192,6 +5339,30 @@
"dev": true,
"license": "MIT"
},
"node_modules/async-retry": {
"version": "1.3.3",
"resolved": "https://registry.npmjs.org/async-retry/-/async-retry-1.3.3.tgz",
"integrity": "sha512-wfr/jstw9xNi/0teMHrRW7dsz3Lt5ARhYNZ2ewpadnhaIp5mbALhOAP+EAdsC7t4Z6wqsDVv9+W6gm1Dk9mEyw==",
"license": "MIT",
"dependencies": {
"retry": "0.13.1"
}
},
"node_modules/async-retry/node_modules/retry": {
"version": "0.13.1",
"resolved": "https://registry.npmjs.org/retry/-/retry-0.13.1.tgz",
"integrity": "sha512-XQBQ3I8W1Cge0Seh+6gjj03LbmRFWuoszgK9ooCpwYIrhhoO80pfq4cUkU5DkknwfOfFteRwlZ56PYOGYyFWdg==",
"license": "MIT",
"engines": {
"node": ">= 4"
}
},
"node_modules/asynckit": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz",
"integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==",
"license": "MIT"
},
"node_modules/available-typed-arrays": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/available-typed-arrays/-/available-typed-arrays-1.0.7.tgz",
@ -5330,7 +5501,6 @@
"version": "1.5.1",
"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
"integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",
"dev": true,
"funding": [
{
"type": "github",
@ -5357,6 +5527,15 @@
"tweetnacl": "^0.14.3"
}
},
"node_modules/bignumber.js": {
"version": "9.3.1",
"resolved": "https://registry.npmjs.org/bignumber.js/-/bignumber.js-9.3.1.tgz",
"integrity": "sha512-Ko0uX15oIUS7wJ3Rb30Fs6SkVbLmPBAKdlm7q9+ak9bbIeFf0MwuBsQV6z7+X768/cHsfg+WlysDWJcmthjsjQ==",
"license": "MIT",
"engines": {
"node": "*"
}
},
"node_modules/bl": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",
@ -5556,6 +5735,12 @@
"node": ">=8.0.0"
}
},
"node_modules/buffer-equal-constant-time": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz",
"integrity": "sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA==",
"license": "BSD-3-Clause"
},
"node_modules/buffer-from": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz",
@ -5616,7 +5801,6 @@
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
"integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
@ -6027,6 +6211,18 @@
"simple-swizzle": "^0.2.2"
}
},
"node_modules/combined-stream": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz",
"integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==",
"license": "MIT",
"dependencies": {
"delayed-stream": "~1.0.0"
},
"engines": {
"node": ">= 0.8"
}
},
"node_modules/commander": {
"version": "11.1.0",
"resolved": "https://registry.npmjs.org/commander/-/commander-11.1.0.tgz",
@ -6524,7 +6720,6 @@
"version": "4.4.1",
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.1.tgz",
"integrity": "sha512-KcKCqiftBJcZr++7ykoDIEwSa3XWowTfNPo92BYxjXiyYEVrUQh2aLyhxBCwww+heortUFxEJYcRzosstTEBYQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"ms": "^2.1.3"
@ -6647,6 +6842,15 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/delayed-stream": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz",
"integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==",
"license": "MIT",
"engines": {
"node": ">=0.4.0"
}
},
"node_modules/detect-indent": {
"version": "6.1.0",
"resolved": "https://registry.npmjs.org/detect-indent/-/detect-indent-6.1.0.tgz",
@ -6913,7 +7117,6 @@
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
"integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==",
"dev": true,
"license": "MIT",
"dependencies": {
"call-bind-apply-helpers": "^1.0.1",
@ -6924,6 +7127,18 @@
"node": ">= 0.4"
}
},
"node_modules/duplexify": {
"version": "4.1.3",
"resolved": "https://registry.npmjs.org/duplexify/-/duplexify-4.1.3.tgz",
"integrity": "sha512-M3BmBhwJRZsSx38lZyhE53Csddgzl5R7xGJNk7CVddZD6CcmwMCH8J+7AprIrQKH7TonKxaCjcv27Qmf+sQ+oA==",
"license": "MIT",
"dependencies": {
"end-of-stream": "^1.4.1",
"inherits": "^2.0.3",
"readable-stream": "^3.1.1",
"stream-shift": "^1.0.2"
}
},
"node_modules/eastasianwidth": {
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/eastasianwidth/-/eastasianwidth-0.2.0.tgz",
@ -6931,6 +7146,15 @@
"dev": true,
"license": "MIT"
},
"node_modules/ecdsa-sig-formatter": {
"version": "1.0.11",
"resolved": "https://registry.npmjs.org/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz",
"integrity": "sha512-nagl3RYrbNv6kQkeJIpt6NJZy8twLB/2vtz6yN9Z4vRKHN4/QZJIEbqohALSgwKdnksuY3k5Addp5lg8sVoVcQ==",
"license": "Apache-2.0",
"dependencies": {
"safe-buffer": "^5.0.1"
}
},
"node_modules/emoji-regex": {
"version": "9.2.2",
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-9.2.2.tgz",
@ -6942,7 +7166,6 @@
"version": "1.4.5",
"resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",
"integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",
"dev": true,
"license": "MIT",
"dependencies": {
"once": "^1.4.0"
@ -6994,7 +7217,6 @@
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz",
"integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==",
"dev": true,
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0"
@ -7003,6 +7225,21 @@
"node": ">= 0.4"
}
},
"node_modules/es-set-tostringtag": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz",
"integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
"get-intrinsic": "^1.2.6",
"has-tostringtag": "^1.0.2",
"hasown": "^2.0.2"
},
"engines": {
"node": ">= 0.4"
}
},
"node_modules/es6-error": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/es6-error/-/es6-error-4.1.1.tgz",
@ -7346,7 +7583,6 @@
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz",
"integrity": "sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
@ -7379,6 +7615,12 @@
"node": ">=12.0.0"
}
},
"node_modules/extend": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz",
"integrity": "sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g==",
"license": "MIT"
},
"node_modules/fast-deep-equal": {
"version": "3.1.3",
"resolved": "https://registry.npmjs.org/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz",
@ -7647,6 +7889,23 @@
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/form-data": {
"version": "2.5.5",
"resolved": "https://registry.npmjs.org/form-data/-/form-data-2.5.5.tgz",
"integrity": "sha512-jqdObeR2rxZZbPSGL+3VckHMYtu+f9//KXBsVny6JSX/pa38Fy+bGjuG8eW/H6USNQWhLi8Num++cU2yOCNz4A==",
"license": "MIT",
"dependencies": {
"asynckit": "^0.4.0",
"combined-stream": "^1.0.8",
"es-set-tostringtag": "^2.1.0",
"hasown": "^2.0.2",
"mime-types": "^2.1.35",
"safe-buffer": "^5.2.1"
},
"engines": {
"node": ">= 0.12"
}
},
"node_modules/frac": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/frac/-/frac-1.1.2.tgz",
@ -7682,12 +7941,41 @@
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
"integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
"dev": true,
"license": "MIT",
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/gaxios": {
"version": "6.7.1",
"resolved": "https://registry.npmjs.org/gaxios/-/gaxios-6.7.1.tgz",
"integrity": "sha512-LDODD4TMYx7XXdpwxAVRAIAuB0bzv0s+ywFonY46k126qzQHT9ygyoa9tncmOiQmmDrik65UYsEkv3lbfqQ3yQ==",
"license": "Apache-2.0",
"dependencies": {
"extend": "^3.0.2",
"https-proxy-agent": "^7.0.1",
"is-stream": "^2.0.0",
"node-fetch": "^2.6.9",
"uuid": "^9.0.1"
},
"engines": {
"node": ">=14"
}
},
"node_modules/gcp-metadata": {
"version": "6.1.1",
"resolved": "https://registry.npmjs.org/gcp-metadata/-/gcp-metadata-6.1.1.tgz",
"integrity": "sha512-a4tiq7E0/5fTjxPAaH4jpjkSv/uCaU2p5KC6HVGrvl0cDjA8iBZv4vv1gyzlmK0ZUKqwpOyQMKzZQe3lTit77A==",
"license": "Apache-2.0",
"dependencies": {
"gaxios": "^6.1.1",
"google-logging-utils": "^0.0.2",
"json-bigint": "^1.0.0"
},
"engines": {
"node": ">=14"
}
},
"node_modules/get-caller-file": {
"version": "2.0.5",
"resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz",
@ -7714,7 +8002,6 @@
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz",
"integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"call-bind-apply-helpers": "^1.0.2",
@ -7815,7 +8102,6 @@
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz",
"integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==",
"dev": true,
"license": "MIT",
"dependencies": {
"dunder-proto": "^1.0.1",
@ -7991,6 +8277,32 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/google-auth-library": {
"version": "9.15.1",
"resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-9.15.1.tgz",
"integrity": "sha512-Jb6Z0+nvECVz+2lzSMt9u98UsoakXxA2HGHMCxh+so3n90XgYWkq5dur19JAJV7ONiJY22yBTyJB1TSkvPq9Ng==",
"license": "Apache-2.0",
"dependencies": {
"base64-js": "^1.3.0",
"ecdsa-sig-formatter": "^1.0.11",
"gaxios": "^6.1.1",
"gcp-metadata": "^6.1.0",
"gtoken": "^7.0.0",
"jws": "^4.0.0"
},
"engines": {
"node": ">=14"
}
},
"node_modules/google-logging-utils": {
"version": "0.0.2",
"resolved": "https://registry.npmjs.org/google-logging-utils/-/google-logging-utils-0.0.2.tgz",
"integrity": "sha512-NEgUnEcBiP5HrPzufUkBzJOD/Sxsco3rLNo1F1TNf7ieU8ryUzBhqba8r756CjLX7rn3fHl6iLEwPYuqpoKgQQ==",
"license": "Apache-2.0",
"engines": {
"node": ">=14"
}
},
"node_modules/gopd": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
@ -8017,6 +8329,19 @@
"dev": true,
"license": "MIT"
},
"node_modules/gtoken": {
"version": "7.1.0",
"resolved": "https://registry.npmjs.org/gtoken/-/gtoken-7.1.0.tgz",
"integrity": "sha512-pCcEwRi+TKpMlxAQObHDQ56KawURgyAf6jtIY046fJ5tIv3zDe/LEIubckAO8fj6JnAxLdmWkUfNyulQ2iKdEw==",
"license": "MIT",
"dependencies": {
"gaxios": "^6.0.0",
"jws": "^4.0.0"
},
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/guid-typescript": {
"version": "1.0.9",
"resolved": "https://registry.npmjs.org/guid-typescript/-/guid-typescript-1.0.9.tgz",
@ -8081,7 +8406,6 @@
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz",
"integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 0.4"
@ -8094,7 +8418,6 @@
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz",
"integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==",
"dev": true,
"license": "MIT",
"dependencies": {
"has-symbols": "^1.0.3"
@ -8110,7 +8433,6 @@
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
"integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"function-bind": "^1.1.2"
@ -8152,6 +8474,22 @@
"dev": true,
"license": "ISC"
},
"node_modules/html-entities": {
"version": "2.6.0",
"resolved": "https://registry.npmjs.org/html-entities/-/html-entities-2.6.0.tgz",
"integrity": "sha512-kig+rMn/QOVRvr7c86gQ8lWXq+Hkv6CbAH1hLu+RG338StTpE8Z0b44SDVaqVu7HGKf27frdmUYEs9hTUX/cLQ==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/mdevils"
},
{
"type": "patreon",
"url": "https://patreon.com/mdevils"
}
],
"license": "MIT"
},
"node_modules/html-escaper": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/html-escaper/-/html-escaper-2.0.2.tgz",
@ -8174,6 +8512,45 @@
"node": ">=8.0.0"
}
},
"node_modules/http-proxy-agent": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-5.0.0.tgz",
"integrity": "sha512-n2hY8YdoRE1i7r6M0w9DIw5GgZN0G25P8zLCRQ8rjXtTU3vsNFBI/vWK/UIeE6g5MUUz6avwAPXmL6Fy9D/90w==",
"license": "MIT",
"dependencies": {
"@tootallnate/once": "2",
"agent-base": "6",
"debug": "4"
},
"engines": {
"node": ">= 6"
}
},
"node_modules/http-proxy-agent/node_modules/agent-base": {
"version": "6.0.2",
"resolved": "https://registry.npmjs.org/agent-base/-/agent-base-6.0.2.tgz",
"integrity": "sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==",
"license": "MIT",
"dependencies": {
"debug": "4"
},
"engines": {
"node": ">= 6.0.0"
}
},
"node_modules/https-proxy-agent": {
"version": "7.0.6",
"resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-7.0.6.tgz",
"integrity": "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==",
"license": "MIT",
"dependencies": {
"agent-base": "^7.1.2",
"debug": "4"
},
"engines": {
"node": ">= 14"
}
},
"node_modules/iconv-lite": {
"version": "0.6.3",
"resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz",
@ -8260,7 +8637,6 @@
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
"dev": true,
"license": "ISC"
},
"node_modules/ini": {
@ -8498,7 +8874,6 @@
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/is-stream/-/is-stream-2.0.1.tgz",
"integrity": "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
@ -8653,6 +9028,15 @@
"js-yaml": "bin/js-yaml.js"
}
},
"node_modules/json-bigint": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/json-bigint/-/json-bigint-1.0.0.tgz",
"integrity": "sha512-SiPv/8VpZuWbvLSMtTDU8hEfrZWg/mH/nV/b4o0CYbSxu1UIQPLdwKOCIyLQX+VIPO5vrLX3i8qtqFyhdPSUSQ==",
"license": "MIT",
"dependencies": {
"bignumber.js": "^9.0.0"
}
},
"node_modules/json-buffer": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/json-buffer/-/json-buffer-3.0.1.tgz",
@ -8742,6 +9126,27 @@
"html2canvas": "^1.0.0-rc.5"
}
},
"node_modules/jwa": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/jwa/-/jwa-2.0.1.tgz",
"integrity": "sha512-hRF04fqJIP8Abbkq5NKGN0Bbr3JxlQ+qhZufXVr0DvujKy93ZCbXZMHDL4EOtodSbCWxOqR8MS1tXA5hwqCXDg==",
"license": "MIT",
"dependencies": {
"buffer-equal-constant-time": "^1.0.1",
"ecdsa-sig-formatter": "1.0.11",
"safe-buffer": "^5.0.1"
}
},
"node_modules/jws": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/jws/-/jws-4.0.0.tgz",
"integrity": "sha512-KDncfTmOZoOMTFG4mBlG0qUIOlc03fmzH+ru6RgYVZhPkyiy/92Owlt/8UEN+a4TXR1FQetfIpJE8ApdvdVxTg==",
"license": "MIT",
"dependencies": {
"jwa": "^2.0.0",
"safe-buffer": "^5.0.1"
}
},
"node_modules/keyv": {
"version": "4.5.4",
"resolved": "https://registry.npmjs.org/keyv/-/keyv-4.5.4.tgz",
@ -9026,7 +9431,6 @@
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
"integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 0.4"
@ -9267,11 +9671,22 @@
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/mime": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/mime/-/mime-3.0.0.tgz",
"integrity": "sha512-jSCU7/VB1loIWBZe14aEYHU/+1UMEHoaO7qxCOVJOw9GgH72VAWppxNcjU+x9a2k3GSIBXNKxXQFqRvvZ7vr3A==",
"license": "MIT",
"bin": {
"mime": "cli.js"
},
"engines": {
"node": ">=10.0.0"
}
},
"node_modules/mime-db": {
"version": "1.52.0",
"resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
"integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 0.6"
@ -9281,7 +9696,6 @@
"version": "2.1.35",
"resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz",
"integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==",
"dev": true,
"license": "MIT",
"dependencies": {
"mime-db": "1.52.0"
@ -9468,7 +9882,6 @@
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
"dev": true,
"license": "MIT"
},
"node_modules/mute-stream": {
@ -9521,6 +9934,26 @@
"dev": true,
"license": "MIT"
},
"node_modules/node-fetch": {
"version": "2.7.0",
"resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz",
"integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==",
"license": "MIT",
"dependencies": {
"whatwg-url": "^5.0.0"
},
"engines": {
"node": "4.x || >=6.0.0"
},
"peerDependencies": {
"encoding": "^0.1.0"
},
"peerDependenciesMeta": {
"encoding": {
"optional": true
}
}
},
"node_modules/normalize-package-data": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/normalize-package-data/-/normalize-package-data-3.0.3.tgz",
@ -9560,7 +9993,6 @@
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
"integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
"dev": true,
"license": "ISC",
"dependencies": {
"wrappy": "1"
@ -9693,7 +10125,6 @@
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/p-limit/-/p-limit-3.1.0.tgz",
"integrity": "sha512-TYOanM3wGwNGsZN2cVTYPArw454xnXj5qmWF1bEoAc4+cU/ol7GVh7odevjp1FNHduHc3KZMcFduxU5Xc6uJRQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"yocto-queue": "^0.1.0"
@ -10307,7 +10738,6 @@
"version": "3.6.2",
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
"integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
"dev": true,
"license": "MIT",
"dependencies": {
"inherits": "^2.0.3",
@ -10441,6 +10871,20 @@
"node": ">= 4"
}
},
"node_modules/retry-request": {
"version": "7.0.2",
"resolved": "https://registry.npmjs.org/retry-request/-/retry-request-7.0.2.tgz",
"integrity": "sha512-dUOvLMJ0/JJYEn8NrpOaGNE7X3vpI5XlZS/u0ANjqtcZVKnIxP7IgCFwrKTxENw29emmwug53awKtaMm4i9g5w==",
"license": "MIT",
"dependencies": {
"@types/request": "^2.48.8",
"extend": "^3.0.2",
"teeny-request": "^9.0.0"
},
"engines": {
"node": ">=14"
}
},
"node_modules/reusify": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/reusify/-/reusify-1.1.0.tgz",
@ -10566,7 +11010,6 @@
"version": "5.2.1",
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
"integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",
"dev": true,
"funding": [
{
"type": "github",
@ -11097,6 +11540,15 @@
"dev": true,
"license": "BSD-3-Clause"
},
"node_modules/stream-events": {
"version": "1.0.5",
"resolved": "https://registry.npmjs.org/stream-events/-/stream-events-1.0.5.tgz",
"integrity": "sha512-E1GUzBSgvct8Jsb3v2X15pjzN1tYebtbLaMg+eBOUOAxgbLoSbT2NS91ckc5lJD1KfLjId+jXJRgo0qnV5Nerg==",
"license": "MIT",
"dependencies": {
"stubs": "^3.0.0"
}
},
"node_modules/stream-json": {
"version": "1.9.1",
"resolved": "https://registry.npmjs.org/stream-json/-/stream-json-1.9.1.tgz",
@ -11107,6 +11559,12 @@
"stream-chain": "^2.2.5"
}
},
"node_modules/stream-shift": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/stream-shift/-/stream-shift-1.0.3.tgz",
"integrity": "sha512-76ORR0DO1o1hlKwTbi/DM3EXWGf3ZJYO8cXX5RJwnul2DEg2oyoZyjLNoQM8WsvZiFKCRfC1O0J7iCvie3RZmQ==",
"license": "MIT"
},
"node_modules/streamx": {
"version": "2.22.1",
"resolved": "https://registry.npmjs.org/streamx/-/streamx-2.22.1.tgz",
@ -11135,7 +11593,6 @@
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
"integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
"dev": true,
"license": "MIT",
"dependencies": {
"safe-buffer": "~5.2.0"
@ -11314,6 +11771,12 @@
],
"license": "MIT"
},
"node_modules/stubs": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/stubs/-/stubs-3.0.0.tgz",
"integrity": "sha512-PdHt7hHUJKxvTCgbKX9C1V/ftOcjJQgz8BZwNfV5c4B6dcGqlpelTbJ999jBGZ2jYiPAwcX5dP6oBwVlBlUbxw==",
"license": "MIT"
},
"node_modules/supports-color": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz",
@ -11395,6 +11858,47 @@
"streamx": "^2.15.0"
}
},
"node_modules/teeny-request": {
"version": "9.0.0",
"resolved": "https://registry.npmjs.org/teeny-request/-/teeny-request-9.0.0.tgz",
"integrity": "sha512-resvxdc6Mgb7YEThw6G6bExlXKkv6+YbuzGg9xuXxSgxJF7Ozs+o8Y9+2R3sArdWdW8nOokoQb1yrpFB0pQK2g==",
"license": "Apache-2.0",
"dependencies": {
"http-proxy-agent": "^5.0.0",
"https-proxy-agent": "^5.0.0",
"node-fetch": "^2.6.9",
"stream-events": "^1.0.5",
"uuid": "^9.0.0"
},
"engines": {
"node": ">=14"
}
},
"node_modules/teeny-request/node_modules/agent-base": {
"version": "6.0.2",
"resolved": "https://registry.npmjs.org/agent-base/-/agent-base-6.0.2.tgz",
"integrity": "sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==",
"license": "MIT",
"dependencies": {
"debug": "4"
},
"engines": {
"node": ">= 6.0.0"
}
},
"node_modules/teeny-request/node_modules/https-proxy-agent": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz",
"integrity": "sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA==",
"license": "MIT",
"dependencies": {
"agent-base": "6",
"debug": "4"
},
"engines": {
"node": ">= 6"
}
},
"node_modules/terser": {
"version": "5.43.1",
"resolved": "https://registry.npmjs.org/terser/-/terser-5.43.1.tgz",
@ -11592,6 +12096,12 @@
"node": ">=8.0"
}
},
"node_modules/tr46": {
"version": "0.0.3",
"resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz",
"integrity": "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==",
"license": "MIT"
},
"node_modules/trim-newlines": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/trim-newlines/-/trim-newlines-3.0.1.tgz",
@ -11754,7 +12264,6 @@
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
"integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",
"dev": true,
"license": "MIT"
},
"node_modules/utrie": {
@ -11976,6 +12485,22 @@
"@zxing/text-encoding": "0.9.0"
}
},
"node_modules/webidl-conversions": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz",
"integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==",
"license": "BSD-2-Clause"
},
"node_modules/whatwg-url": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz",
"integrity": "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==",
"license": "MIT",
"dependencies": {
"tr46": "~0.0.3",
"webidl-conversions": "^3.0.0"
}
},
"node_modules/which": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz",
@ -12207,7 +12732,6 @@
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
"dev": true,
"license": "ISC"
},
"node_modules/ws": {
@ -12396,7 +12920,6 @@
"version": "0.1.0",
"resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz",
"integrity": "sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=10"

View file

@ -157,6 +157,7 @@
},
"dependencies": {
"@aws-sdk/client-s3": "^3.540.0",
"@google-cloud/storage": "^7.14.0",
"@huggingface/transformers": "^3.7.2",
"boxen": "^8.0.1",
"chalk": "^5.3.0",

File diff suppressed because it is too large Load diff

View file

@ -10,6 +10,7 @@ import {
S3CompatibleStorage,
R2Storage
} from './adapters/s3CompatibleStorage.js'
import { GcsStorage } from './adapters/gcsStorage.js'
// FileSystemStorage is dynamically imported to avoid issues in browser environments
import { isBrowser } from '../utils/environment.js'
import { OperationConfig } from '../utils/operationUtils.js'
@ -26,9 +27,10 @@ export interface StorageOptions {
* - 'filesystem': Use file system storage (Node.js only)
* - 's3': Use Amazon S3 storage
* - 'r2': Use Cloudflare R2 storage
* - 'gcs': Use Google Cloud Storage
* - 'gcs': Use Google Cloud Storage (S3-compatible with HMAC keys)
* - 'gcs-native': Use Google Cloud Storage (native SDK with ADC)
*/
type?: 'auto' | 'memory' | 'opfs' | 'filesystem' | 's3' | 'r2' | 'gcs'
type?: 'auto' | 'memory' | 'opfs' | 'filesystem' | 's3' | 'r2' | 'gcs' | 'gcs-native'
/**
* Force the use of memory storage even if other storage types are available
@ -106,7 +108,7 @@ export interface StorageOptions {
}
/**
* Configuration for Google Cloud Storage
* Configuration for Google Cloud Storage (S3-compatible with HMAC keys)
*/
gcsStorage?: {
/**
@ -135,6 +137,36 @@ export interface StorageOptions {
endpoint?: string
}
/**
* Configuration for Google Cloud Storage (native SDK with ADC)
*/
gcsNativeStorage?: {
/**
* GCS bucket name
*/
bucketName: string
/**
* Service account key file path (optional, uses ADC if not provided)
*/
keyFilename?: string
/**
* Service account credentials object (optional, uses ADC if not provided)
*/
credentials?: object
/**
* HMAC access key ID (backward compatibility, not recommended)
*/
accessKeyId?: string
/**
* HMAC secret access key (backward compatibility, not recommended)
*/
secretAccessKey?: string
}
/**
* Configuration for custom S3-compatible storage
*/
@ -357,7 +389,7 @@ export async function createStorage(
case 'gcs':
if (options.gcsStorage) {
console.log('Using Google Cloud Storage')
console.log('Using Google Cloud Storage (S3-compatible)')
return new S3CompatibleStorage({
bucketName: options.gcsStorage.bucketName,
region: options.gcsStorage.region,
@ -375,6 +407,24 @@ export async function createStorage(
return new MemoryStorage()
}
case 'gcs-native':
if (options.gcsNativeStorage) {
console.log('Using Google Cloud Storage (native SDK)')
return new GcsStorage({
bucketName: options.gcsNativeStorage.bucketName,
keyFilename: options.gcsNativeStorage.keyFilename,
credentials: options.gcsNativeStorage.credentials,
accessKeyId: options.gcsNativeStorage.accessKeyId,
secretAccessKey: options.gcsNativeStorage.secretAccessKey,
cacheConfig: options.cacheConfig
})
} else {
console.warn(
'GCS native storage configuration is missing, falling back to memory storage'
)
return new MemoryStorage()
}
default:
console.warn(
`Unknown storage type: ${options.type}, falling back to memory storage`
@ -426,9 +476,22 @@ export async function createStorage(
})
}
// If GCS storage is specified, use it
// If GCS native storage is specified, use it (prioritize native over S3-compatible)
if (options.gcsNativeStorage) {
console.log('Using Google Cloud Storage (native SDK)')
return new GcsStorage({
bucketName: options.gcsNativeStorage.bucketName,
keyFilename: options.gcsNativeStorage.keyFilename,
credentials: options.gcsNativeStorage.credentials,
accessKeyId: options.gcsNativeStorage.accessKeyId,
secretAccessKey: options.gcsNativeStorage.secretAccessKey,
cacheConfig: options.cacheConfig
})
}
// If GCS storage is specified, use it (S3-compatible)
if (options.gcsStorage) {
console.log('Using Google Cloud Storage')
console.log('Using Google Cloud Storage (S3-compatible)')
return new S3CompatibleStorage({
bucketName: options.gcsStorage.bucketName,
region: options.gcsStorage.region,
@ -498,7 +561,8 @@ export {
MemoryStorage,
OPFSStorage,
S3CompatibleStorage,
R2Storage
R2Storage,
GcsStorage
}
// Export FileSystemStorage conditionally

View file

@ -0,0 +1,489 @@
/**
* GCS Native Storage Adapter Integration Tests
*
* This test verifies that the GCS native adapter:
* - Properly authenticates with ADC and service accounts
* - Implements UUID-based sharding correctly
* - Handles pagination across shards
* - Persists data correctly
* - Manages statistics and counts
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { GcsStorage } from '../../src/storage/adapters/gcsStorage.js'
import { randomUUID } from 'node:crypto'
describe('GCS Native Storage Adapter', () => {
// Mock GCS client for testing
let mockGcsObjects: Map<string, any> = new Map()
let storage: GcsStorage | null = null
// Helper to create mock GCS client
function createMockGcsClient() {
const mockBucket = {
exists: async () => [true],
file: (name: string) => ({
save: async (data: string, options: any) => {
mockGcsObjects.set(name, JSON.parse(data))
return {}
},
download: async () => {
const data = mockGcsObjects.get(name)
if (!data) {
const error: any = new Error('Not found')
error.code = 404
throw error
}
return [Buffer.from(JSON.stringify(data))]
},
delete: async () => {
mockGcsObjects.delete(name)
return [{}]
}
}),
getFiles: async (options: any) => {
const prefix = options.prefix || ''
const maxResults = options.maxResults || 1000
const pageToken = options.pageToken
console.log(`[Mock GCS] getFiles: Prefix="${prefix}", maxResults=${maxResults}`)
// Filter objects by prefix
const allKeys = Array.from(mockGcsObjects.keys())
const matchingKeys = allKeys.filter(key => key.startsWith(prefix)).sort()
console.log(`[Mock GCS] Total keys=${allKeys.length}, Matching=${matchingKeys.length}`)
if (matchingKeys.length > 0) {
console.log(`[Mock GCS] First match: ${matchingKeys[0]}`)
}
// Apply pagination
let startIndex = 0
if (pageToken) {
startIndex = parseInt(pageToken)
}
const endIndex = Math.min(startIndex + maxResults, matchingKeys.length)
const pageKeys = matchingKeys.slice(startIndex, endIndex)
const files = pageKeys.map(key => ({
name: key
}))
const response = {
nextPageToken: endIndex < matchingKeys.length ? String(endIndex) : undefined
}
return [files, {}, response]
},
getMetadata: async () => [{
location: 'us-central1',
storageClass: 'STANDARD',
timeCreated: new Date().toISOString()
}]
}
return {
bucket: (name: string) => mockBucket
}
}
beforeEach(() => {
mockGcsObjects.clear()
})
afterEach(async () => {
if (storage) {
try {
await storage.clear()
} catch (error) {
// Ignore cleanup errors
}
storage = null
}
})
it('should initialize with Application Default Credentials (ADC)', async () => {
// Create storage without any credentials (ADC)
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
// Verify initialization
expect((storage as any).bucketName).toBe('test-bucket')
expect((storage as any).keyFilename).toBeUndefined()
expect((storage as any).credentials).toBeUndefined()
})
it('should initialize with service account key file', async () => {
// Create storage with key file
storage = new GcsStorage({
bucketName: 'test-bucket',
keyFilename: '/path/to/service-account.json'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
// Verify initialization
expect((storage as any).keyFilename).toBe('/path/to/service-account.json')
})
it('should write and read data with UUID-based sharding', async () => {
console.log('\n📝 Test: Write and read with UUID sharding...')
// Create storage
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
// Generate UUIDs for testing
const testData = [
{ id: randomUUID(), vector: [0.1, 0.2, 0.3], metadata: { type: 'user', name: 'Alice' } },
{ id: randomUUID(), vector: [0.4, 0.5, 0.6], metadata: { type: 'user', name: 'Bob' } },
{ id: randomUUID(), vector: [0.7, 0.8, 0.9], metadata: { type: 'user', name: 'Charlie' } }
]
// Save nouns with metadata
for (const item of testData) {
const noun = {
id: item.id,
vector: item.vector,
connections: new Map(),
layer: 0
}
await storage.saveNoun(noun)
await (storage as any).saveNounMetadata_internal(item.id, item.metadata)
}
console.log(`✅ Wrote ${testData.length} entities`)
console.log(`📊 Objects in mock storage: ${mockGcsObjects.size}`)
// Verify data was written to UUID-sharded paths
const shardedKeys = Array.from(mockGcsObjects.keys()).filter(k =>
k.match(/entities\/nouns\/vectors\/[0-9a-f]{2}\//)
)
console.log(`🔑 UUID-sharded keys: ${shardedKeys.length}`)
expect(shardedKeys.length).toBe(testData.length)
// Log shard distribution
const shards = new Set(shardedKeys.map(k => k.match(/entities\/nouns\/vectors\/([0-9a-f]{2})\//)?.[1]))
console.log(`📁 Data distributed across ${shards.size} shards: ${Array.from(shards).join(', ')}`)
// Verify each entity can be retrieved
for (const item of testData) {
const noun = await storage.getNoun(item.id)
expect(noun).toBeTruthy()
expect(noun!.id).toBe(item.id)
expect(noun!.vector).toEqual(item.vector)
const metadata = await storage.getNounMetadata(item.id)
expect(metadata).toBeTruthy()
expect(metadata.name).toBe(item.metadata.name)
}
console.log('✅ All entities retrieved successfully')
})
it('should handle pagination across UUID shards', async () => {
console.log('\n🔄 Test: Pagination across shards...')
// Create storage
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
// Write 10 entities with proper UUIDs
console.log('📝 Writing 10 entities...')
for (let i = 0; i < 10; i++) {
const id = randomUUID()
const noun = {
id,
vector: [0.1 * i, 0.2 * i, 0.3 * i],
connections: new Map(),
layer: 0
}
await storage.saveNoun(noun)
}
// Read with pagination (limit: 3)
console.log('\n🔄 Reading with pagination (limit: 3)...')
let allEntities: any[] = []
let cursor: string | undefined
let page = 0
do {
const result = await storage.getNounsWithPagination({
limit: 3,
cursor
})
page++
console.log(`📄 Page ${page}: ${result.items.length} entities, hasMore: ${result.hasMore}`)
allEntities.push(...result.items)
cursor = result.nextCursor
// Safety check to prevent infinite loops
expect(page).toBeLessThan(20)
} while (cursor)
console.log(`✅ Loaded ${allEntities.length} total entities across ${page} pages`)
// Verify all entities were loaded
expect(allEntities.length).toBe(10)
})
it('should return correct totalCount on first call', async () => {
console.log('\n📊 Test: Correct totalCount...')
// Create storage
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
// Write 5 entities
for (let i = 0; i < 5; i++) {
await storage.saveNoun({
id: randomUUID(),
vector: [0.1, 0.2, 0.3],
connections: new Map(),
layer: 0
})
}
// Get first page
const result = await storage.getNounsWithPagination({ limit: 2 })
console.log(`📊 First page: ${result.items.length} items`)
expect(result.items.length).toBeLessThanOrEqual(2)
})
it('should handle verb operations with UUID sharding', async () => {
console.log('\n🔗 Test: Verb operations with UUID sharding...')
// Create storage
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
// Create verb
const verbId = randomUUID()
const verb = {
id: verbId,
vector: [0.1, 0.2, 0.3],
connections: new Map()
}
await storage.saveVerb(verb)
// Verify verb was saved with UUID sharding
const shardedKeys = Array.from(mockGcsObjects.keys()).filter(k =>
k.match(/entities\/verbs\/vectors\/[0-9a-f]{2}\//)
)
expect(shardedKeys.length).toBeGreaterThan(0)
// Retrieve verb
const retrieved = await storage.getVerb(verbId)
expect(retrieved).toBeTruthy()
expect(retrieved!.id).toBe(verbId)
console.log('✅ Verb operations successful')
})
it('should handle metadata sharding correctly', async () => {
console.log('\n🗂 Test: Metadata sharding...')
// Create storage
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
// Create noun with metadata
const nounId = randomUUID()
const noun = {
id: nounId,
vector: [0.1, 0.2, 0.3],
connections: new Map(),
layer: 0
}
const metadata = {
type: 'user',
name: 'Test User',
email: 'test@example.com'
}
await storage.saveNoun(noun)
await (storage as any).saveNounMetadata_internal(nounId, metadata)
// Verify metadata was saved with UUID sharding
const metadataKeys = Array.from(mockGcsObjects.keys()).filter(k =>
k.match(/entities\/nouns\/metadata\/[0-9a-f]{2}\//)
)
expect(metadataKeys.length).toBeGreaterThan(0)
// Retrieve metadata
const retrieved = await storage.getNounMetadata(nounId)
expect(retrieved).toBeTruthy()
expect(retrieved.name).toBe('Test User')
console.log('✅ Metadata sharding successful')
})
it('should clear all data correctly', async () => {
console.log('\n🧹 Test: Clear all data...')
// Create storage
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
// Write some data
for (let i = 0; i < 3; i++) {
await storage.saveNoun({
id: randomUUID(),
vector: [0.1, 0.2, 0.3],
connections: new Map(),
layer: 0
})
}
console.log(`📊 Objects before clear: ${mockGcsObjects.size}`)
// Clear all data
await storage.clear()
console.log(`📊 Objects after clear: ${mockGcsObjects.size}`)
expect(mockGcsObjects.size).toBe(0)
console.log('✅ Clear successful')
})
it('should handle throttling errors correctly', async () => {
console.log('\n🚦 Test: Throttling detection...')
// Create storage
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Test throttling error detection
const throttlingError = { code: 429, message: 'Too Many Requests' }
expect((storage as any).isThrottlingError(throttlingError)).toBe(true)
const quotaError = { code: 403, message: 'Quota exceeded' }
expect((storage as any).isThrottlingError(quotaError)).toBe(true)
const normalError = { code: 500, message: 'Internal Server Error' }
expect((storage as any).isThrottlingError(normalError)).toBe(false)
console.log('✅ Throttling detection working')
})
it('should manage statistics correctly', async () => {
console.log('\n📊 Test: Statistics management...')
// Create storage
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
// Create statistics
const stats = {
nounCount: { 'test-service': 5 },
verbCount: { 'test-service': 3 },
metadataCount: {},
hnswIndexSize: 100,
lastUpdated: new Date().toISOString()
}
// Save statistics
await (storage as any).saveStatisticsData(stats)
// Verify statistics key was created
const statsKeys = Array.from(mockGcsObjects.keys()).filter(k =>
k.includes('_system/statistics.json')
)
expect(statsKeys.length).toBe(1)
// Retrieve statistics
const retrieved = await (storage as any).getStatisticsData()
expect(retrieved).toBeTruthy()
expect(retrieved.nounCount['test-service']).toBe(5)
console.log('✅ Statistics management successful')
})
it('should get storage status', async () => {
console.log('\n📊 Test: Storage status...')
// Create storage
storage = new GcsStorage({
bucketName: 'test-bucket'
})
// Mock the GCS client
;(storage as any).storage = createMockGcsClient()
;(storage as any).bucket = (storage as any).storage.bucket('test-bucket')
;(storage as any).isInitialized = true
const status = await storage.getStorageStatus()
expect(status.type).toBe('gcs-native')
expect(status.details).toBeTruthy()
expect(status.details!.bucket).toBe('test-bucket')
console.log('✅ Storage status retrieved:', status)
})
})
console.log('\n✅ GCS Native Storage Tests Complete')