**refactor(cli): remove unused Brainy CLI implementation**
- Deleted the entire `src/cli.ts` file, effectively removing the Brainy CLI component. - The change streamlines the codebase by eliminating unused and redundant code related to the CLI. - This refactor supports maintaining a cleaner and more focused project architecture.
This commit is contained in:
parent
e2b0dc04ce
commit
10a76a7b69
10 changed files with 120 additions and 92 deletions
32
README.md
32
README.md
|
|
@ -6,7 +6,7 @@
|
|||
[](https://nodejs.org/)
|
||||
[](https://www.typescriptlang.org/)
|
||||
[](CONTRIBUTING.md)
|
||||
[](https://www.npmjs.com/package/@soulcraft/brainy)
|
||||
[](https://www.npmjs.com/package/@soulcraft/brainy)
|
||||
|
||||
[//]: # ([](https://github.com/sodal-project/cartographer))
|
||||
|
||||
|
|
@ -115,7 +115,9 @@ import { BrainyData } from '@soulcraft/brainy/min'
|
|||
import '@soulcraft/brainy/cli'
|
||||
```
|
||||
|
||||
> **Note**: The CLI functionality (4MB) is not included in the standard import, reducing the bundle size for browser and Node.js applications. The CLI is only built and loaded when explicitly imported or when using the command-line interface.
|
||||
> **Note**: The CLI functionality (4MB) is not included in the standard import, reducing the bundle size for browser and
|
||||
> Node.js applications. The CLI is only built and loaded when explicitly imported or when using the command-line
|
||||
> interface.
|
||||
|
||||
### Browser Usage
|
||||
|
||||
|
|
@ -424,7 +426,8 @@ Connections between nouns (edges in the graph):
|
|||
|
||||
## Command Line Interface
|
||||
|
||||
Brainy includes a powerful CLI for managing your data. The CLI is available as a separate package `@soulcraft/brainy-cli` to reduce the bundle size of the main package.
|
||||
Brainy includes a powerful CLI for managing your data. The CLI is available as a separate package
|
||||
`@soulcraft/brainy-cli` to reduce the bundle size of the main package.
|
||||
|
||||
### Installing and Using the CLI
|
||||
|
||||
|
|
@ -452,7 +455,8 @@ brainy visualize --root <id> --depth 3
|
|||
|
||||
### Publishing the CLI Package
|
||||
|
||||
If you need to publish the CLI package to npm, please refer to the [CLI Publishing Guide](docs/publishing-cli.md) for detailed instructions.
|
||||
If you need to publish the CLI package to npm, please refer to the [CLI Publishing Guide](docs/publishing-cli.md) for
|
||||
detailed instructions.
|
||||
|
||||
### Using the CLI in Your Code
|
||||
|
||||
|
|
@ -466,7 +470,8 @@ import '@soulcraft/brainy/cli'
|
|||
// ...
|
||||
```
|
||||
|
||||
This will only build and load the CLI when you explicitly import it, keeping your bundle size small when you don't need the CLI.
|
||||
This will only build and load the CLI when you explicitly import it, keeping your bundle size small when you don't need
|
||||
the CLI.
|
||||
|
||||
### Development Usage
|
||||
|
||||
|
|
@ -1302,17 +1307,24 @@ comprehensive [Scaling Strategy](scalingStrategy.md) document.
|
|||
|
||||
Brainy takes advantage of several optimizations available in Node.js 24:
|
||||
|
||||
1. **Improved Worker Threads Performance**: The multithreading system has been completely rewritten to leverage Node.js 24's enhanced Worker Threads API, resulting in better performance for compute-intensive operations like embedding generation and vector similarity calculations.
|
||||
1. **Improved Worker Threads Performance**: The multithreading system has been completely rewritten to leverage Node.js
|
||||
24's enhanced Worker Threads API, resulting in better performance for compute-intensive operations like embedding
|
||||
generation and vector similarity calculations.
|
||||
|
||||
2. **Worker Pool Management**: A sophisticated worker pool system reuses worker threads to minimize the overhead of creating and destroying threads, leading to more efficient resource utilization.
|
||||
2. **Worker Pool Management**: A sophisticated worker pool system reuses worker threads to minimize the overhead of
|
||||
creating and destroying threads, leading to more efficient resource utilization.
|
||||
|
||||
3. **Dynamic Module Imports**: Uses the new `node:` protocol prefix for importing core modules, which provides better performance and more reliable module resolution.
|
||||
3. **Dynamic Module Imports**: Uses the new `node:` protocol prefix for importing core modules, which provides better
|
||||
performance and more reliable module resolution.
|
||||
|
||||
4. **ES Modules Optimizations**: Takes advantage of Node.js 24's improved ESM implementation for faster module loading and execution.
|
||||
4. **ES Modules Optimizations**: Takes advantage of Node.js 24's improved ESM implementation for faster module loading
|
||||
and execution.
|
||||
|
||||
5. **Enhanced Error Handling**: Implements more robust error handling patterns available in Node.js 24 for better stability and debugging.
|
||||
5. **Enhanced Error Handling**: Implements more robust error handling patterns available in Node.js 24 for better
|
||||
stability and debugging.
|
||||
|
||||
These optimizations are particularly beneficial for:
|
||||
|
||||
- Large-scale vector operations
|
||||
- Batch processing of embeddings
|
||||
- Real-time data processing pipelines
|
||||
|
|
|
|||
|
|
@ -19,16 +19,16 @@ const __dirname = dirname(__filename)
|
|||
// Find the main package
|
||||
const mainPackagePath = join(__dirname, 'node_modules', '@soulcraft', 'brainy')
|
||||
|
||||
// Path to the actual CLI script in the main package
|
||||
const cliPath = join(mainPackagePath, 'dist', 'cli.js')
|
||||
// Path to the actual CLI script in this package
|
||||
const cliPath = join(__dirname, 'dist', 'cli.js')
|
||||
|
||||
// Check if the CLI script exists
|
||||
if (!fs.existsSync(cliPath)) {
|
||||
console.error(`Error: CLI script not found at ${cliPath}`)
|
||||
console.error('This is likely because the main package is not installed correctly.')
|
||||
console.error('This is likely because the CLI was not built during package installation.')
|
||||
console.error('Please reinstall the package with:')
|
||||
console.error('npm uninstall -g @soulcraft/brainy-cli')
|
||||
console.error('npm install -g @soulcraft/brainy-cli --legacy-peer-deps')
|
||||
console.error('npm install -g @soulcraft/brainy-cli')
|
||||
process.exit(1)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "@soulcraft/brainy-cli",
|
||||
"version": "0.9.16",
|
||||
"version": "0.9.17",
|
||||
"description": "Command-line interface for the Brainy vector graph database",
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
|
@ -8,9 +8,13 @@
|
|||
},
|
||||
"files": [
|
||||
"cli-wrapper.js",
|
||||
"README.md"
|
||||
"README.md",
|
||||
"dist/cli.js",
|
||||
"dist/cli.js.map"
|
||||
],
|
||||
"scripts": {
|
||||
"build": "rollup -c rollup.config.js",
|
||||
"prepare": "npm run build",
|
||||
"postinstall": "node cli-wrapper.js --version"
|
||||
},
|
||||
"keywords": [
|
||||
|
|
@ -35,7 +39,20 @@
|
|||
"url": "https://github.com/soulcraft-research/brainy.git"
|
||||
},
|
||||
"dependencies": {
|
||||
"@soulcraft/brainy": "0.9.16"
|
||||
"@soulcraft/brainy": "0.9.17",
|
||||
"commander": "^14.0.0",
|
||||
"omelette": "^0.4.17"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@rollup/plugin-commonjs": "^25.0.7",
|
||||
"@rollup/plugin-json": "^6.1.0",
|
||||
"@rollup/plugin-node-resolve": "^15.2.3",
|
||||
"@rollup/plugin-typescript": "^11.1.6",
|
||||
"@types/node": "^20.4.5",
|
||||
"@types/omelette": "^0.4.5",
|
||||
"rollup": "^4.12.0",
|
||||
"rollup-plugin-terser": "^7.0.2",
|
||||
"typescript": "^5.1.6"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=24.0.0"
|
||||
|
|
|
|||
42
cli-package/rollup.config.js
Normal file
42
cli-package/rollup.config.js
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
import typescript from '@rollup/plugin-typescript'
|
||||
import resolve from '@rollup/plugin-node-resolve'
|
||||
import commonjs from '@rollup/plugin-commonjs'
|
||||
import json from '@rollup/plugin-json'
|
||||
import { terser } from 'rollup-plugin-terser'
|
||||
|
||||
// CLI configuration
|
||||
export default {
|
||||
input: 'src/cli.ts',
|
||||
output: {
|
||||
dir: 'dist',
|
||||
entryFileNames: 'cli.js',
|
||||
format: 'es',
|
||||
sourcemap: true,
|
||||
inlineDynamicImports: true
|
||||
},
|
||||
plugins: [
|
||||
resolve({
|
||||
browser: false,
|
||||
preferBuiltins: true
|
||||
}),
|
||||
commonjs({
|
||||
transformMixedEsModules: true
|
||||
}),
|
||||
json(),
|
||||
typescript({
|
||||
tsconfig: './tsconfig.json',
|
||||
declaration: false,
|
||||
declarationMap: false
|
||||
})
|
||||
],
|
||||
external: [
|
||||
// External dependencies that should not be bundled
|
||||
'@soulcraft/brainy',
|
||||
'commander',
|
||||
'omelette',
|
||||
'fs',
|
||||
'path',
|
||||
'url',
|
||||
'child_process'
|
||||
]
|
||||
}
|
||||
|
|
@ -5,21 +5,22 @@
|
|||
* A command-line interface for interacting with the Brainy vector database
|
||||
*/
|
||||
|
||||
import { BrainyData, NounType, VerbType, FileSystemStorage } from './index.js'
|
||||
import { BrainyData, NounType, VerbType, FileSystemStorage, sequentialPipeline, augmentationPipeline, ExecutionMode, AugmentationType } from '@soulcraft/brainy'
|
||||
import { fileURLToPath } from 'url'
|
||||
import { dirname, join } from 'path'
|
||||
import fs from 'fs'
|
||||
import { Command } from 'commander'
|
||||
import omelette from 'omelette'
|
||||
import { VERSION } from './utils/version.js'
|
||||
import { sequentialPipeline } from './sequentialPipeline.js'
|
||||
import { augmentationPipeline, ExecutionMode } from './augmentationPipeline.js'
|
||||
import { AugmentationType } from './types/augmentations.js'
|
||||
|
||||
// Get the directory of the current module
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = dirname(__filename)
|
||||
|
||||
// Get version from package.json
|
||||
const packageJsonPath = join(__dirname, '..', 'package.json')
|
||||
const packageJson = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8'))
|
||||
const VERSION = packageJson.version
|
||||
|
||||
// Helper function to parse JSON safely
|
||||
function parseJSON(str: string): any {
|
||||
try {
|
||||
|
|
@ -71,14 +72,14 @@ const program = new Command()
|
|||
|
||||
// Configure the program
|
||||
program
|
||||
.name('@soulcraft/brainy')
|
||||
.name('@soulcraft/brainy-cli')
|
||||
.description(
|
||||
'A vector database using HNSW indexing with Origin Private File System storage'
|
||||
)
|
||||
.version(VERSION, '-V, --version', 'Output the current version')
|
||||
|
||||
// Create data directory if it doesn't exist
|
||||
const dataDir = join(__dirname, '..', 'data')
|
||||
const dataDir = join(process.cwd(), 'data')
|
||||
if (!fs.existsSync(dataDir)) {
|
||||
fs.mkdirSync(dataDir, { recursive: true })
|
||||
}
|
||||
|
|
@ -1282,5 +1283,4 @@ program
|
|||
})
|
||||
|
||||
// Parse command line arguments
|
||||
|
||||
program.parse()
|
||||
15
cli-package/tsconfig.json
Normal file
15
cli-package/tsconfig.json
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2020",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "node",
|
||||
"esModuleInterop": true,
|
||||
"strict": true,
|
||||
"outDir": "dist",
|
||||
"declaration": true,
|
||||
"sourceMap": true,
|
||||
"skipLibCheck": true
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
10
package.json
10
package.json
|
|
@ -22,9 +22,6 @@
|
|||
"./types/augmentations": {
|
||||
"import": "./dist/types/augmentations.js",
|
||||
"types": "./dist/types/augmentations.d.ts"
|
||||
},
|
||||
"./cli": {
|
||||
"import": "./dist/cli.js"
|
||||
}
|
||||
},
|
||||
"engines": {
|
||||
|
|
@ -34,7 +31,6 @@
|
|||
"prebuild": "node scripts/generate-version.js",
|
||||
"build": "BUILD_TYPE=unified rollup -c rollup.config.js",
|
||||
"build:browser": "BUILD_TYPE=browser rollup -c rollup.config.js",
|
||||
"build:cli": "BUILD_TYPE=cli rollup -c rollup.config.js",
|
||||
"start": "node dist/unified.js",
|
||||
"demo": "npm run build && npm run build:browser && npx http-server -o /index.html",
|
||||
"version": "node scripts/generate-version.js",
|
||||
|
|
@ -47,8 +43,7 @@
|
|||
"check-format": "prettier --check \"src/**/*.{ts,js}\"",
|
||||
"check-style": "node scripts/check-code-style.js",
|
||||
"prepare": "npm run build",
|
||||
"deploy": "npm run build && npm publish",
|
||||
"deploy:both": "node scripts/publish-cli.js",
|
||||
"deploy": "node scripts/publish-cli.js",
|
||||
"deploy:cloud:aws": "cd cloud-wrapper && npm run build && npm run deploy:aws",
|
||||
"deploy:cloud:gcp": "cd cloud-wrapper && npm run build && npm run deploy:gcp",
|
||||
"deploy:cloud:cloudflare": "cd cloud-wrapper && npm run build && npm run deploy:cloudflare",
|
||||
|
|
@ -94,7 +89,6 @@
|
|||
"@rollup/plugin-replace": "^5.0.5",
|
||||
"@rollup/plugin-typescript": "^11.1.6",
|
||||
"@types/node": "^20.4.5",
|
||||
"@types/omelette": "^0.4.5",
|
||||
"@types/uuid": "^10.0.0",
|
||||
"@typescript-eslint/eslint-plugin": "^6.0.0",
|
||||
"@typescript-eslint/parser": "^6.0.0",
|
||||
|
|
@ -113,8 +107,6 @@
|
|||
"@tensorflow/tfjs-converter": "^4.22.0",
|
||||
"@tensorflow/tfjs-core": "^4.22.0",
|
||||
"buffer": "^6.0.3",
|
||||
"commander": "^14.0.0",
|
||||
"omelette": "^0.4.17",
|
||||
"uuid": "^9.0.0"
|
||||
},
|
||||
"prettier": {
|
||||
|
|
|
|||
|
|
@ -112,14 +112,6 @@ const config = {
|
|||
var global = typeof window !== "undefined" ? window : this;
|
||||
// Buffer polyfill is now included via the buffer-polyfill plugin
|
||||
`
|
||||
},
|
||||
cli: {
|
||||
input: 'src/cli.ts',
|
||||
outputPrefix: 'cli',
|
||||
tsconfig: './tsconfig.unified.json',
|
||||
declaration: false,
|
||||
declarationMap: false,
|
||||
intro: ''
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -187,47 +179,5 @@ const mainConfig = {
|
|||
]
|
||||
}
|
||||
|
||||
// CLI configuration
|
||||
const cliConfig = {
|
||||
input: 'src/cli.ts',
|
||||
output: {
|
||||
dir: 'dist',
|
||||
entryFileNames: 'cli.js',
|
||||
format: 'es',
|
||||
sourcemap: true,
|
||||
inlineDynamicImports: true
|
||||
},
|
||||
plugins: [
|
||||
replace({
|
||||
preventAssignment: true,
|
||||
'process.env.NODE_ENV': JSON.stringify('production')
|
||||
}),
|
||||
fixThisReferences(),
|
||||
resolve({
|
||||
browser: false,
|
||||
preferBuiltins: true
|
||||
}),
|
||||
commonjs({
|
||||
transformMixedEsModules: true
|
||||
}),
|
||||
json(),
|
||||
typescript({
|
||||
tsconfig: './tsconfig.unified.json',
|
||||
declaration: false,
|
||||
declarationMap: false
|
||||
})
|
||||
],
|
||||
external: [
|
||||
// Add any dependencies you want to exclude from the bundle
|
||||
'@aws-sdk/client-s3',
|
||||
'@smithy/util-stream',
|
||||
'@smithy/node-http-handler',
|
||||
'@aws-crypto/crc32c',
|
||||
'node:stream/web',
|
||||
'node:worker_threads'
|
||||
]
|
||||
}
|
||||
|
||||
// Export configurations based on build type
|
||||
// Only include CLI config when specifically building CLI
|
||||
export default buildType === 'cli' ? cliConfig : mainConfig
|
||||
// Export the main configuration
|
||||
export default mainConfig
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@
|
|||
* This script:
|
||||
* 1. Ensures versions are in sync by running generate-version.js
|
||||
* 2. Builds the main package
|
||||
* 3. Builds the CLI
|
||||
* 3. Builds the CLI package
|
||||
* 4. Publishes the main package
|
||||
* 5. Publishes the CLI package
|
||||
*
|
||||
|
|
@ -39,12 +39,12 @@ try {
|
|||
console.log('Building main package...')
|
||||
execSync('npm run build', { stdio: 'inherit', cwd: rootDir })
|
||||
|
||||
// Step 3: Build the CLI
|
||||
console.log('Building CLI...')
|
||||
execSync('npm run build:cli', { stdio: 'inherit', cwd: rootDir })
|
||||
// Step 3: Build the CLI package
|
||||
console.log('Building CLI package...')
|
||||
execSync('npm run build', { stdio: 'inherit', cwd: cliPackageDir })
|
||||
|
||||
// Step 4: Verify the CLI was built successfully
|
||||
const cliPath = path.join(rootDir, 'dist', 'cli.js')
|
||||
const cliPath = path.join(cliPackageDir, 'dist', 'cli.js')
|
||||
if (!fs.existsSync(cliPath)) {
|
||||
console.error(`Error: CLI build failed. File not found at ${cliPath}`)
|
||||
process.exit(1)
|
||||
|
|
|
|||
|
|
@ -3,4 +3,4 @@
|
|||
* Do not modify this file directly.
|
||||
*/
|
||||
|
||||
export const VERSION = '0.9.16';
|
||||
export const VERSION = '0.9.17';
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue