**chore: remove outdated documentation and refactor Rollup configuration**
### Changes: - Removed two documentation files: - `docs/build-time-augmentations.md` - `docs/llm-augmentation.md` - Deleted `rollup.unified.js` and `rollup_config.js` configurations to streamline build setup. - Introduced custom plugins for Node.js module shims and "this" reference fixes directly into the remaining Rollup setup. - Updated Rollup configuration entry and build logic for increased maintainability: - Enabled `inlineDynamicImports` for unified builds. - Enhanced browser compatibility with global Buffer polyfill and environment detection (`isBrowser`, `isNode`, `isServerless`). - Adjusted error logging for module fetch failures in browser environments. ### Purpose: Cleaned outdated and redundant documentation to align with the system's current capabilities. Consolidated and optimized Rollup build setup, ensuring a modern and maintainable configuration while improving cross-environment compatibility and error handling.
This commit is contained in:
parent
c385f3fcff
commit
ace8e0ec15
4 changed files with 0 additions and 912 deletions
|
|
@ -1,217 +0,0 @@
|
||||||
<div align="center">
|
|
||||||
<img src="../brainy.png" alt="Brainy Logo" width="200"/>
|
|
||||||
|
|
||||||
# Build-Time Augmentation Registration
|
|
||||||
</div>
|
|
||||||
|
|
||||||
This document explains how to register custom augmentations at build time using the Brainy library's augmentation
|
|
||||||
registry system.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
Brainy provides a system for registering custom augmentations at build time, similar to Angular's pipeline. This allows
|
|
||||||
you to:
|
|
||||||
|
|
||||||
1. Define custom augmentations in your project
|
|
||||||
2. Register them with the Brainy library during the build process
|
|
||||||
3. Have them automatically available when your application runs
|
|
||||||
|
|
||||||
This approach has several advantages:
|
|
||||||
|
|
||||||
- Better performance as augmentations are available immediately at startup
|
|
||||||
- Improved tree-shaking and bundle optimization
|
|
||||||
- Type safety and better IDE support
|
|
||||||
- No need for dynamic imports or async loading
|
|
||||||
|
|
||||||
## Creating Custom Augmentations
|
|
||||||
|
|
||||||
To create a custom augmentation, you need to implement one of the augmentation interfaces provided by Brainy:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// myTextSenseAugmentation.ts
|
|
||||||
import {
|
|
||||||
registerAugmentation,
|
|
||||||
AugmentationType,
|
|
||||||
BrainyAugmentations
|
|
||||||
} from 'brainy';
|
|
||||||
|
|
||||||
// Define a custom sense augmentation
|
|
||||||
class MyTextSenseAugmentation implements BrainyAugmentations.ISenseAugmentation {
|
|
||||||
name = 'MyTextSenseAugmentation';
|
|
||||||
enabled = true;
|
|
||||||
type = AugmentationType.SENSE;
|
|
||||||
|
|
||||||
// Required IAugmentation methods
|
|
||||||
async initialize() {
|
|
||||||
console.log('Initializing MyTextSenseAugmentation');
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
|
|
||||||
async shutDown() {
|
|
||||||
console.log('Shutting down MyTextSenseAugmentation');
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
|
|
||||||
getStatus() {
|
|
||||||
return {
|
|
||||||
name: this.name,
|
|
||||||
enabled: this.enabled,
|
|
||||||
type: this.type,
|
|
||||||
status: 'ready'
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
// ISenseAugmentation methods
|
|
||||||
async processRawData(rawData, dataType) {
|
|
||||||
console.log(`Processing ${dataType} data`);
|
|
||||||
|
|
||||||
// Your implementation here
|
|
||||||
|
|
||||||
return {
|
|
||||||
success: true,
|
|
||||||
data: { /* processed data */}
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
async listenToFeed(feedUrl, callback) {
|
|
||||||
console.log(`Listening to feed at ${feedUrl}`);
|
|
||||||
|
|
||||||
// Your implementation here
|
|
||||||
|
|
||||||
return {
|
|
||||||
success: true,
|
|
||||||
data: { /* feed info */}
|
|
||||||
};
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Register the augmentation with the registry
|
|
||||||
// This will make it available to the Brainy library at runtime
|
|
||||||
export const myTextSenseAugmentation = registerAugmentation(new MyTextSenseAugmentation());
|
|
||||||
```
|
|
||||||
|
|
||||||
## Registering Augmentations
|
|
||||||
|
|
||||||
There are two ways to register augmentations:
|
|
||||||
|
|
||||||
### 1. Manual Registration
|
|
||||||
|
|
||||||
You can manually register augmentations by calling `registerAugmentation()` in your code:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import {registerAugmentation} from 'brainy';
|
|
||||||
import {MyCustomAugmentation} from './myCustomAugmentation';
|
|
||||||
|
|
||||||
// Register the augmentation
|
|
||||||
const myAugmentation = registerAugmentation(new MyCustomAugmentation());
|
|
||||||
|
|
||||||
// You can also export it for use elsewhere in your application
|
|
||||||
export {myAugmentation};
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Automatic Registration with Build Tools
|
|
||||||
|
|
||||||
For larger projects, you can use build tools like webpack or rollup to automatically discover and register
|
|
||||||
augmentations:
|
|
||||||
|
|
||||||
#### With Webpack
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// webpack.config.js
|
|
||||||
const {createAugmentationRegistryPlugin} = require('brainy');
|
|
||||||
|
|
||||||
module.exports = {
|
|
||||||
// ... other webpack config
|
|
||||||
plugins: [
|
|
||||||
createAugmentationRegistryPlugin({
|
|
||||||
// Pattern to match files containing augmentations
|
|
||||||
pattern: /augmentation\.(js|ts)$/,
|
|
||||||
options: {
|
|
||||||
autoInitialize: true,
|
|
||||||
debug: true
|
|
||||||
}
|
|
||||||
})
|
|
||||||
]
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
#### With Rollup
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// rollup.config.js
|
|
||||||
import {createAugmentationRegistryRollupPlugin} from 'brainy';
|
|
||||||
|
|
||||||
export default {
|
|
||||||
// ... other rollup config
|
|
||||||
plugins: [
|
|
||||||
createAugmentationRegistryRollupPlugin({
|
|
||||||
pattern: /augmentation\.(js|ts)$/,
|
|
||||||
options: {
|
|
||||||
autoInitialize: true,
|
|
||||||
debug: true
|
|
||||||
}
|
|
||||||
})
|
|
||||||
]
|
|
||||||
};
|
|
||||||
```
|
|
||||||
|
|
||||||
## Using Registered Augmentations
|
|
||||||
|
|
||||||
Once augmentations are registered, they are automatically available to the Brainy library. You can use them through the
|
|
||||||
augmentation pipeline:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import {BrainyData, augmentationPipeline, initializeAugmentationPipeline} from 'brainy';
|
|
||||||
|
|
||||||
// Create a new BrainyData instance
|
|
||||||
const db = new BrainyData();
|
|
||||||
await db.init();
|
|
||||||
|
|
||||||
// Initialize the augmentation pipeline with all registered augmentations
|
|
||||||
initializeAugmentationPipeline();
|
|
||||||
|
|
||||||
// Use the pipeline to execute augmentations
|
|
||||||
const senseResults = await augmentationPipeline.executeSensePipeline(
|
|
||||||
'processRawData',
|
|
||||||
['This is some example text to process', 'text']
|
|
||||||
);
|
|
||||||
|
|
||||||
// Process the results
|
|
||||||
for (const resultPromise of senseResults) {
|
|
||||||
const result = await resultPromise;
|
|
||||||
if (result.success) {
|
|
||||||
console.log('Processed data:', result.data);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Best Practices
|
|
||||||
|
|
||||||
1. **Naming Convention**: Use a consistent naming convention for your augmentation files, such as ending them with
|
|
||||||
`augmentation.ts` or `augmentation.js`.
|
|
||||||
|
|
||||||
2. **File Organization**: Keep your augmentations organized in a dedicated directory, such as `src/augmentations/`.
|
|
||||||
|
|
||||||
3. **Type Safety**: Implement the appropriate interfaces for your augmentations to ensure type safety.
|
|
||||||
|
|
||||||
4. **Documentation**: Document your augmentations with JSDoc comments to provide clear usage instructions.
|
|
||||||
|
|
||||||
5. **Testing**: Write tests for your augmentations to ensure they work correctly.
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
If your augmentations are not being registered or are not working as expected, check the following:
|
|
||||||
|
|
||||||
1. Make sure your augmentation implements all required methods from the interface.
|
|
||||||
2. Verify that your augmentation is being registered with `registerAugmentation()`.
|
|
||||||
3. If using build tools, check that your file naming matches the pattern specified in the plugin configuration.
|
|
||||||
4. Enable debug logging in the plugin options to see detailed information about the registration process.
|
|
||||||
5. Check the console for any error messages during initialization.
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
For complete examples, see:
|
|
||||||
|
|
||||||
- [Basic Augmentation Registration](../examples/buildTimeRegistration.js)
|
|
||||||
- [Webpack Configuration](../examples/webpack.config.js)
|
|
||||||
- [Rollup Configuration](../examples/rollup.config.js)
|
|
||||||
|
|
@ -1,443 +0,0 @@
|
||||||
<div align="center">
|
|
||||||
<img src="../brainy.png" alt="Brainy Logo" width="200"/>
|
|
||||||
|
|
||||||
# LLM Augmentation for Brainy
|
|
||||||
</div>
|
|
||||||
|
|
||||||
This document describes the LLM (Language Learning Model) augmentation for Brainy, which enables creating, training, testing, exporting, and deploying language models from the data in Brainy's graph database.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
The LLM augmentation extends Brainy with the ability to create and train language models using the nouns and verbs stored in the graph database. It leverages TensorFlow.js to provide cross-platform compatibility, working in both browser and Node.js environments.
|
|
||||||
|
|
||||||
The augmentation consists of two main components:
|
|
||||||
|
|
||||||
1. **LLMCognitionAugmentation**: A cognition augmentation that provides the core functionality for creating, training, testing, exporting, and deploying LLM models.
|
|
||||||
2. **LLMActivationAugmentation**: An activation augmentation that provides action triggers for the LLM functionality, making it accessible through the augmentation pipeline.
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
- **Create LLM Models**: Create simple sequence models or transformer models with customizable parameters.
|
|
||||||
- **Train Models**: Train models on the nouns and verbs in Brainy's database with configurable training options.
|
|
||||||
- **Test Models**: Evaluate model performance and generate sample predictions.
|
|
||||||
- **Export Models**: Export trained models in various formats (TFJS, JSON) for use in different environments.
|
|
||||||
- **Deploy Models**: Deploy models to browser, Node.js, or cloud environments.
|
|
||||||
- **Generate Text**: Use trained models to generate text based on input prompts.
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
The LLM augmentation is included as an optional component in the Brainy package. No additional installation is required if you have already installed Brainy.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npm install @soulcraft/brainy --legacy-peer-deps
|
|
||||||
```
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Simplified Usage (Recommended for New Users)
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
import { BrainyData, augmentationPipeline } from '@soulcraft/brainy'
|
|
||||||
import { createLLMAugmentations } from '@soulcraft/brainy/src/augmentations/llmAugmentations.js'
|
|
||||||
|
|
||||||
// Initialize Brainy
|
|
||||||
const db = new BrainyData()
|
|
||||||
await db.init()
|
|
||||||
|
|
||||||
// Create LLM augmentations
|
|
||||||
const { cognition, activation } = await createLLMAugmentations({
|
|
||||||
cognitionName: 'my-llm-cognition',
|
|
||||||
activationName: 'my-llm-activation',
|
|
||||||
brainyDb: db
|
|
||||||
})
|
|
||||||
|
|
||||||
// Register augmentations with the pipeline
|
|
||||||
augmentationPipeline.register(cognition)
|
|
||||||
augmentationPipeline.register(activation)
|
|
||||||
|
|
||||||
// Create a model using a preset (tiny, small, medium, large)
|
|
||||||
const createResult = await cognition.createModelFromPreset('small', 'my-model')
|
|
||||||
|
|
||||||
const modelId = createResult.data.modelId
|
|
||||||
|
|
||||||
// Train the model with simplified options (quick, standard, thorough)
|
|
||||||
await cognition.trainModelSimple(modelId, 'standard')
|
|
||||||
|
|
||||||
// Generate text with simplified creativity options (conservative, balanced, creative)
|
|
||||||
const generateResult = await cognition.generateTextSimple(modelId, 'What is a', 'balanced')
|
|
||||||
|
|
||||||
console.log(`Generated text: ${generateResult.data.text}`)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Advanced Usage
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
import { BrainyData, augmentationPipeline } from '@soulcraft/brainy'
|
|
||||||
import { createLLMAugmentations } from '@soulcraft/brainy/src/augmentations/llmAugmentations.js'
|
|
||||||
|
|
||||||
// Initialize Brainy
|
|
||||||
const db = new BrainyData()
|
|
||||||
await db.init()
|
|
||||||
|
|
||||||
// Create LLM augmentations
|
|
||||||
const { cognition, activation } = await createLLMAugmentations({
|
|
||||||
cognitionName: 'my-llm-cognition',
|
|
||||||
activationName: 'my-llm-activation',
|
|
||||||
brainyDb: db
|
|
||||||
})
|
|
||||||
|
|
||||||
// Register augmentations with the pipeline
|
|
||||||
augmentationPipeline.register(cognition)
|
|
||||||
augmentationPipeline.register(activation)
|
|
||||||
|
|
||||||
// Create a model with advanced configuration
|
|
||||||
const createResult = await cognition.createModel({
|
|
||||||
name: 'my-model',
|
|
||||||
modelType: 'simple',
|
|
||||||
vocabSize: 5000,
|
|
||||||
embeddingDim: 64,
|
|
||||||
hiddenDim: 128,
|
|
||||||
numLayers: 1
|
|
||||||
})
|
|
||||||
|
|
||||||
const modelId = createResult.data.modelId
|
|
||||||
|
|
||||||
// Train the model with advanced options
|
|
||||||
await cognition.trainModel(modelId, {
|
|
||||||
maxSamples: 100,
|
|
||||||
validationSplit: 0.2
|
|
||||||
})
|
|
||||||
|
|
||||||
// Generate text with advanced options
|
|
||||||
const generateResult = await cognition.generateText(modelId, 'What is a', {
|
|
||||||
temperature: 0.7
|
|
||||||
})
|
|
||||||
|
|
||||||
console.log(`Generated text: ${generateResult.data.text}`)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Using the Augmentation Pipeline
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
// Create a model through the pipeline
|
|
||||||
const pipelineCreateResult = await augmentationPipeline.executeCognitionPipeline(
|
|
||||||
'createModel',
|
|
||||||
[{
|
|
||||||
name: 'pipeline-model',
|
|
||||||
modelType: 'transformer',
|
|
||||||
numHeads: 2,
|
|
||||||
numLayers: 1
|
|
||||||
}]
|
|
||||||
)
|
|
||||||
|
|
||||||
if (pipelineCreateResult[0] && (await pipelineCreateResult[0]).success) {
|
|
||||||
const pipelineModelId = (await pipelineCreateResult[0]).data.modelId
|
|
||||||
|
|
||||||
// Train the model through the pipeline
|
|
||||||
await augmentationPipeline.executeCognitionPipeline(
|
|
||||||
'trainModel',
|
|
||||||
[pipelineModelId, { maxSamples: 50 }]
|
|
||||||
)
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## CLI Usage
|
|
||||||
|
|
||||||
The LLM functionality is also available through the Brainy CLI. The CLI provides both simplified and advanced commands for working with LLM models.
|
|
||||||
|
|
||||||
### Simplified CLI Commands (Recommended for New Users)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Create a model using a preset (tiny, small, medium, large)
|
|
||||||
brainy llm create-simple [preset] --name my-model
|
|
||||||
|
|
||||||
# Train a model with simplified options (quick, standard, thorough)
|
|
||||||
brainy llm train-simple <modelId> [level]
|
|
||||||
|
|
||||||
# Generate text with simplified creativity options (conservative, balanced, creative)
|
|
||||||
brainy llm generate-simple <modelId> "Your prompt here" [creativity]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Advanced CLI Commands
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Create a model with advanced configuration
|
|
||||||
brainy llm create --name my-model --type simple --vocab-size 5000
|
|
||||||
|
|
||||||
# Train a model with advanced options
|
|
||||||
brainy llm train <modelId> --max-samples 100 --epochs 10
|
|
||||||
|
|
||||||
# Test a model
|
|
||||||
brainy llm test <modelId> --generate-samples
|
|
||||||
|
|
||||||
# Export a model
|
|
||||||
brainy llm export <modelId> --format json --include-metadata
|
|
||||||
|
|
||||||
# Deploy a model
|
|
||||||
brainy llm deploy <modelId> --target browser
|
|
||||||
|
|
||||||
# Generate text with advanced options
|
|
||||||
brainy llm generate <modelId> "Your prompt here" --temperature 0.7
|
|
||||||
```
|
|
||||||
|
|
||||||
## API Reference
|
|
||||||
|
|
||||||
### LLMCognitionAugmentation
|
|
||||||
|
|
||||||
#### Simplified Methods (Recommended for New Users)
|
|
||||||
|
|
||||||
##### Creating Models with Presets
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
createModelFromPreset(presetName: string = 'small', customName?: string): Promise<AugmentationResponse<{ modelId: string, metadata: LLMModelMetadata }>>
|
|
||||||
```
|
|
||||||
|
|
||||||
Creates a new LLM model using a predefined preset.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `presetName`: Name of the preset to use ('tiny', 'small', 'medium', 'large')
|
|
||||||
- `customName`: Optional custom name for the model
|
|
||||||
|
|
||||||
**Returns:**
|
|
||||||
- `modelId`: Unique identifier for the created model
|
|
||||||
- `metadata`: Metadata about the model
|
|
||||||
|
|
||||||
**Preset Details:**
|
|
||||||
- `tiny`: Simple model with minimal parameters, fast but less accurate
|
|
||||||
- `small`: Simple model with balanced parameters, good for most use cases
|
|
||||||
- `medium`: Transformer model with moderate parameters, better accuracy
|
|
||||||
- `large`: Transformer model with more parameters, best accuracy but slower
|
|
||||||
|
|
||||||
##### Training Models with Simplified Options
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
trainModelSimple(modelId: string, trainingLevel: 'quick' | 'standard' | 'thorough' = 'standard'): Promise<AugmentationResponse<{ modelId: string, metadata: LLMModelMetadata, trainingHistory: tf.History, metrics?: { loss: number, accuracy: number, epochs: number } }>>
|
|
||||||
```
|
|
||||||
|
|
||||||
Trains an LLM model with simplified options.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `modelId`: ID of the model to train
|
|
||||||
- `trainingLevel`: Training level ('quick', 'standard', 'thorough')
|
|
||||||
|
|
||||||
**Returns:**
|
|
||||||
- `modelId`: ID of the trained model
|
|
||||||
- `metadata`: Updated metadata about the model
|
|
||||||
- `trainingHistory`: Training history with metrics
|
|
||||||
- `metrics`: Training metrics (loss, accuracy, epochs)
|
|
||||||
|
|
||||||
**Training Level Details:**
|
|
||||||
- `quick`: Faster training with fewer samples (100 samples, 5 epochs)
|
|
||||||
- `standard`: Balanced training (500 samples, 10 epochs)
|
|
||||||
- `thorough`: More thorough training for better results (1000 samples, 20 epochs)
|
|
||||||
|
|
||||||
##### Generating Text with Simplified Creativity Options
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
generateTextSimple(modelId: string, prompt: string, creativity: 'conservative' | 'balanced' | 'creative' = 'balanced'): Promise<AugmentationResponse<{ text: string, tokens?: number, timeMs?: number }>>
|
|
||||||
```
|
|
||||||
|
|
||||||
Generates text using the LLM model with simplified creativity options.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `modelId`: ID of the model to use
|
|
||||||
- `prompt`: Input prompt for text generation
|
|
||||||
- `creativity`: Creativity level ('conservative', 'balanced', 'creative')
|
|
||||||
|
|
||||||
**Returns:**
|
|
||||||
- `text`: Generated text
|
|
||||||
- `tokens`: Number of tokens generated (if available)
|
|
||||||
- `timeMs`: Generation time in milliseconds (if available)
|
|
||||||
|
|
||||||
**Creativity Level Details:**
|
|
||||||
- `conservative`: More predictable, focused output (temperature: 0.3, topK: 3)
|
|
||||||
- `balanced`: Mix of predictability and creativity (temperature: 0.7, topK: 5)
|
|
||||||
- `creative`: More varied, unexpected output (temperature: 1.0, topK: 10)
|
|
||||||
|
|
||||||
#### Advanced Methods
|
|
||||||
|
|
||||||
##### Creating Models
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
createModel(config: Partial<LLMModelConfig>): Promise<AugmentationResponse<{ modelId: string, metadata: LLMModelMetadata }>>
|
|
||||||
```
|
|
||||||
|
|
||||||
Creates a new LLM model with the specified configuration.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `config`: Configuration options for the model
|
|
||||||
- `name`: Name of the model (optional, auto-generated if not provided)
|
|
||||||
- `description`: Description of the model (optional)
|
|
||||||
- `modelType`: Type of model to create ('simple', 'transformer', or 'custom')
|
|
||||||
- `vocabSize`: Size of the vocabulary (default: 10000)
|
|
||||||
- `embeddingDim`: Dimension of the embedding vectors (default: 128)
|
|
||||||
- `hiddenDim`: Dimension of the hidden layers (default: 256)
|
|
||||||
- `numLayers`: Number of layers in the model (default: 2)
|
|
||||||
- `numHeads`: Number of attention heads for transformer models (default: 4)
|
|
||||||
- `dropoutRate`: Dropout rate for regularization (default: 0.1)
|
|
||||||
- `maxSequenceLength`: Maximum sequence length for input (default: 100)
|
|
||||||
- `learningRate`: Learning rate for training (default: 0.001)
|
|
||||||
- `batchSize`: Batch size for training (default: 32)
|
|
||||||
- `epochs`: Number of training epochs (default: 10)
|
|
||||||
- `customModelPath`: Path to a custom model (required for 'custom' modelType)
|
|
||||||
|
|
||||||
**Returns:**
|
|
||||||
- `modelId`: Unique identifier for the created model
|
|
||||||
- `metadata`: Metadata about the model
|
|
||||||
|
|
||||||
#### Training Models
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
trainModel(modelId: string, options: LLMTrainingOptions): Promise<AugmentationResponse<{ modelId: string, metadata: LLMModelMetadata, trainingHistory: tf.History }>>
|
|
||||||
```
|
|
||||||
|
|
||||||
Trains an LLM model on Brainy data.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `modelId`: ID of the model to train
|
|
||||||
- `options`: Training options
|
|
||||||
- `nounTypes`: Types of nouns to include in training (default: all)
|
|
||||||
- `verbTypes`: Types of verbs to include in training (default: all)
|
|
||||||
- `maxSamples`: Maximum number of training samples (default: all)
|
|
||||||
- `validationSplit`: Fraction of data to use for validation (default: 0.2)
|
|
||||||
- `includeMetadata`: Whether to include metadata in training (default: false)
|
|
||||||
- `includeEmbeddings`: Whether to include embeddings in training (default: false)
|
|
||||||
- `augmentData`: Whether to augment training data (default: false)
|
|
||||||
- `earlyStoppingPatience`: Number of epochs with no improvement before stopping (default: 3)
|
|
||||||
|
|
||||||
**Returns:**
|
|
||||||
- `modelId`: ID of the trained model
|
|
||||||
- `metadata`: Updated metadata about the model
|
|
||||||
- `trainingHistory`: Training history with metrics
|
|
||||||
|
|
||||||
#### Testing Models
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
testModel(modelId: string, options: LLMTestingOptions): Promise<AugmentationResponse<{ modelId: string, metrics: Record<string, number>, samples?: Array<{ input: string, expected: string, generated: string }> }>>
|
|
||||||
```
|
|
||||||
|
|
||||||
Tests an LLM model on Brainy data.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `modelId`: ID of the model to test
|
|
||||||
- `options`: Testing options
|
|
||||||
- `testSize`: Number of test samples (default: 100)
|
|
||||||
- `randomSeed`: Random seed for reproducibility (optional)
|
|
||||||
- `metrics`: Metrics to calculate (default: ['accuracy', 'loss'])
|
|
||||||
- `generateSamples`: Whether to generate sample predictions (default: false)
|
|
||||||
- `sampleCount`: Number of samples to generate (default: 5)
|
|
||||||
|
|
||||||
**Returns:**
|
|
||||||
- `modelId`: ID of the tested model
|
|
||||||
- `metrics`: Test metrics (accuracy, loss, etc.)
|
|
||||||
- `samples`: Sample predictions (if generateSamples is true)
|
|
||||||
|
|
||||||
#### Exporting Models
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
exportModel(modelId: string, options: LLMExportOptions): Promise<AugmentationResponse<{ modelId: string, format: string, exportPath?: string, modelJSON?: string }>>
|
|
||||||
```
|
|
||||||
|
|
||||||
Exports an LLM model for deployment.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `modelId`: ID of the model to export
|
|
||||||
- `options`: Export options
|
|
||||||
- `format`: Export format ('tfjs', 'onnx', 'savedmodel', 'json')
|
|
||||||
- `quantize`: Whether to quantize the model (default: false)
|
|
||||||
- `outputPath`: Path to save the exported model (optional)
|
|
||||||
- `includeMetadata`: Whether to include metadata (default: false)
|
|
||||||
- `includeVocab`: Whether to include vocabulary (default: false)
|
|
||||||
|
|
||||||
**Returns:**
|
|
||||||
- `modelId`: ID of the exported model
|
|
||||||
- `format`: Export format
|
|
||||||
- `exportPath`: Path where the model was saved (if outputPath was provided)
|
|
||||||
- `modelJSON`: JSON representation of the model (if outputPath was not provided)
|
|
||||||
|
|
||||||
#### Deploying Models
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
deployModel(modelId: string, options: LLMDeploymentOptions): Promise<AugmentationResponse<{ modelId: string, deploymentTarget: string, deploymentUrl?: string, status: string }>>
|
|
||||||
```
|
|
||||||
|
|
||||||
Deploys an LLM model to the specified target.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `modelId`: ID of the model to deploy
|
|
||||||
- `options`: Deployment options
|
|
||||||
- `target`: Deployment target ('browser', 'node', 'cloud')
|
|
||||||
- `cloudProvider`: Cloud provider for cloud deployment ('aws', 'gcp', 'azure')
|
|
||||||
- `endpoint`: Endpoint URL for cloud deployment
|
|
||||||
- `apiKey`: API key for cloud deployment
|
|
||||||
- `region`: Region for cloud deployment
|
|
||||||
- `containerize`: Whether to containerize the model (default: false)
|
|
||||||
- `autoScale`: Whether to enable auto-scaling (default: false)
|
|
||||||
- `memory`: Memory allocation for deployment
|
|
||||||
- `cpu`: CPU allocation for deployment
|
|
||||||
|
|
||||||
**Returns:**
|
|
||||||
- `modelId`: ID of the deployed model
|
|
||||||
- `deploymentTarget`: Target where the model was deployed
|
|
||||||
- `deploymentUrl`: URL where the model is accessible (for cloud deployments)
|
|
||||||
- `status`: Deployment status
|
|
||||||
|
|
||||||
#### Generating Text
|
|
||||||
|
|
||||||
```javascript
|
|
||||||
generateText(modelId: string, prompt: string, options: { maxLength?: number, temperature?: number, topK?: number }): Promise<AugmentationResponse<string>>
|
|
||||||
```
|
|
||||||
|
|
||||||
Generates text using the LLM model.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `modelId`: ID of the model to use
|
|
||||||
- `prompt`: Input prompt for text generation
|
|
||||||
- `options`: Generation options
|
|
||||||
- `maxLength`: Maximum length of generated text (default: model-dependent)
|
|
||||||
- `temperature`: Temperature for sampling (default: 1.0)
|
|
||||||
- `topK`: Number of top tokens to consider (default: all)
|
|
||||||
|
|
||||||
**Returns:**
|
|
||||||
- Generated text
|
|
||||||
|
|
||||||
### LLMActivationAugmentation
|
|
||||||
|
|
||||||
The activation augmentation provides action triggers for the LLM functionality, making it accessible through the augmentation pipeline. It supports the following actions:
|
|
||||||
|
|
||||||
- `createModel`: Creates a new LLM model
|
|
||||||
- `trainModel`: Trains an LLM model
|
|
||||||
- `testModel`: Tests an LLM model
|
|
||||||
- `exportModel`: Exports an LLM model
|
|
||||||
- `deployModel`: Deploys an LLM model
|
|
||||||
- `generateText`: Generates text using an LLM model
|
|
||||||
|
|
||||||
## Model Types
|
|
||||||
|
|
||||||
### Simple Sequence Model
|
|
||||||
|
|
||||||
A simple sequence model with an embedding layer, LSTM layers, and a dense output layer. This model is suitable for simpler language modeling tasks and requires less computational resources.
|
|
||||||
|
|
||||||
### Transformer Model
|
|
||||||
|
|
||||||
A transformer model with multi-head attention, suitable for more complex language modeling tasks. This model can capture longer-range dependencies in text but requires more computational resources.
|
|
||||||
|
|
||||||
## Limitations
|
|
||||||
|
|
||||||
- The current implementation is focused on training small language models suitable for specific domains rather than large general-purpose models.
|
|
||||||
- Training large models in the browser may be limited by available memory and computational resources.
|
|
||||||
- Cloud deployment functionality is a placeholder and requires additional implementation for specific cloud providers.
|
|
||||||
- ONNX export is not implemented in the current version.
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
See the [llmAugmentationExample.js](../examples/llmAugmentationExample.js) file for a complete example of using the LLM augmentation.
|
|
||||||
|
|
||||||
## Future Enhancements
|
|
||||||
|
|
||||||
- Support for larger model architectures
|
|
||||||
- Improved training efficiency for browser environments
|
|
||||||
- Full implementation of cloud deployment options
|
|
||||||
- Support for ONNX export
|
|
||||||
- Fine-tuning of pre-trained models
|
|
||||||
- More advanced text generation capabilities
|
|
||||||
|
|
@ -1,154 +0,0 @@
|
||||||
import typescript from '@rollup/plugin-typescript';
|
|
||||||
import resolve from '@rollup/plugin-node-resolve';
|
|
||||||
import commonjs from '@rollup/plugin-commonjs';
|
|
||||||
import json from '@rollup/plugin-json';
|
|
||||||
import { terser } from 'rollup-plugin-terser';
|
|
||||||
import replace from '@rollup/plugin-replace';
|
|
||||||
|
|
||||||
// Custom plugin to provide empty shims for Node.js built-in modules in browser environments
|
|
||||||
const nodeModuleShims = () => {
|
|
||||||
return {
|
|
||||||
name: 'node-module-shims',
|
|
||||||
resolveId(source) {
|
|
||||||
// List of Node.js built-in modules to shim
|
|
||||||
const nodeBuiltins = [
|
|
||||||
'fs', 'path', 'util', 'crypto', 'os', 'stream',
|
|
||||||
'http', 'http2', 'https', 'zlib', 'child_process'
|
|
||||||
];
|
|
||||||
|
|
||||||
if (nodeBuiltins.includes(source)) {
|
|
||||||
// Return a virtual module ID for the shim
|
|
||||||
return `\0${source}-shim`;
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
},
|
|
||||||
load(id) {
|
|
||||||
// If this is one of our shims, return an empty module
|
|
||||||
if (id.startsWith('\0') && id.endsWith('-shim')) {
|
|
||||||
console.log(`Providing empty shim for Node.js module: ${id.slice(1, -5)}`);
|
|
||||||
return 'export default {}; export const promises = {};';
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
// Custom plugin to fix 'this' references in specific files
|
|
||||||
const fixThisReferences = () => {
|
|
||||||
return {
|
|
||||||
name: 'fix-this-references',
|
|
||||||
transform(code, id) {
|
|
||||||
// Only transform the specific files that have issues with 'this'
|
|
||||||
if (id.includes('@tensorflow/tfjs-layers/dist/layers/convolutional_recurrent.js')) {
|
|
||||||
// Replace 'this' with 'globalThis' in the problematic code
|
|
||||||
return {
|
|
||||||
code: code.replace(/\bthis\b/g, 'globalThis'),
|
|
||||||
map: { mappings: '' } // Provide an empty sourcemap to avoid warnings
|
|
||||||
};
|
|
||||||
}
|
|
||||||
return null; // Return null to let Rollup handle other files normally
|
|
||||||
}
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
export default {
|
|
||||||
input: 'src/unified.ts', // Use the unified entry point
|
|
||||||
inlineDynamicImports: true,
|
|
||||||
output: [
|
|
||||||
{
|
|
||||||
file: 'dist/unified.js',
|
|
||||||
format: 'es',
|
|
||||||
sourcemap: true,
|
|
||||||
intro: `
|
|
||||||
// Environment detection
|
|
||||||
const isBrowser = typeof window !== 'undefined';
|
|
||||||
const isNode = typeof process !== 'undefined' && process.versions && process.versions.node;
|
|
||||||
const isServerless = !isBrowser && !isNode;
|
|
||||||
|
|
||||||
// Global Buffer polyfill for browser environments
|
|
||||||
if (isBrowser) {
|
|
||||||
try {
|
|
||||||
// Import Buffer from the buffer package
|
|
||||||
import('buffer').then(({ Buffer }) => {
|
|
||||||
globalThis.Buffer = Buffer;
|
|
||||||
}).catch(err => {
|
|
||||||
console.warn('Failed to load Buffer polyfill:', err);
|
|
||||||
});
|
|
||||||
} catch (e) {
|
|
||||||
console.warn('Failed to import buffer package:', e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Global variable to store environment
|
|
||||||
globalThis.__ENV__ = {
|
|
||||||
isBrowser,
|
|
||||||
isNode,
|
|
||||||
isServerless
|
|
||||||
};
|
|
||||||
`
|
|
||||||
},
|
|
||||||
{
|
|
||||||
file: 'dist/unified.min.js',
|
|
||||||
format: 'es',
|
|
||||||
sourcemap: true,
|
|
||||||
plugins: [terser()],
|
|
||||||
intro: `
|
|
||||||
// Environment detection
|
|
||||||
const isBrowser = typeof window !== 'undefined';
|
|
||||||
const isNode = typeof process !== 'undefined' && process.versions && process.versions.node;
|
|
||||||
const isServerless = !isBrowser && !isNode;
|
|
||||||
|
|
||||||
// Global Buffer polyfill for browser environments
|
|
||||||
if (isBrowser) {
|
|
||||||
try {
|
|
||||||
// Import Buffer from the buffer package
|
|
||||||
import('buffer').then(({ Buffer }) => {
|
|
||||||
globalThis.Buffer = Buffer;
|
|
||||||
}).catch(err => {
|
|
||||||
console.warn('Failed to load Buffer polyfill:', err);
|
|
||||||
});
|
|
||||||
} catch (e) {
|
|
||||||
console.warn('Failed to import buffer package:', e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Global variable to store environment
|
|
||||||
globalThis.__ENV__ = {
|
|
||||||
isBrowser,
|
|
||||||
isNode,
|
|
||||||
isServerless
|
|
||||||
};
|
|
||||||
`
|
|
||||||
}
|
|
||||||
],
|
|
||||||
plugins: [
|
|
||||||
// Add environment replacement
|
|
||||||
replace({
|
|
||||||
preventAssignment: true,
|
|
||||||
'process.env.NODE_ENV': JSON.stringify('production')
|
|
||||||
}),
|
|
||||||
// Add our custom plugins
|
|
||||||
fixThisReferences(),
|
|
||||||
nodeModuleShims(),
|
|
||||||
resolve({
|
|
||||||
browser: true,
|
|
||||||
preferBuiltins: false
|
|
||||||
}),
|
|
||||||
commonjs({
|
|
||||||
transformMixedEsModules: true
|
|
||||||
}),
|
|
||||||
json(),
|
|
||||||
typescript({
|
|
||||||
tsconfig: './tsconfig.unified.json',
|
|
||||||
declaration: true,
|
|
||||||
declarationMap: true
|
|
||||||
})
|
|
||||||
],
|
|
||||||
external: [
|
|
||||||
// Add any dependencies you want to exclude from the bundle
|
|
||||||
'@aws-sdk/client-s3',
|
|
||||||
'@smithy/util-stream',
|
|
||||||
'@smithy/node-http-handler',
|
|
||||||
'@aws-crypto/crc32c'
|
|
||||||
]
|
|
||||||
};
|
|
||||||
|
|
@ -1,98 +0,0 @@
|
||||||
import typescript from '@rollup/plugin-typescript'
|
|
||||||
import resolve from '@rollup/plugin-node-resolve'
|
|
||||||
import commonjs from '@rollup/plugin-commonjs'
|
|
||||||
import json from '@rollup/plugin-json'
|
|
||||||
import { terser } from 'rollup-plugin-terser'
|
|
||||||
|
|
||||||
// Custom plugin to fix 'this' references in specific files
|
|
||||||
const fixThisReferences = () => {
|
|
||||||
return {
|
|
||||||
name: 'fix-this-references',
|
|
||||||
transform(code, id) {
|
|
||||||
// Only transform the specific files that have issues with 'this'
|
|
||||||
if (id.includes('@tensorflow/tfjs-layers/dist/layers/convolutional_recurrent.js')) {
|
|
||||||
// Replace 'this' with 'globalThis' in the problematic code
|
|
||||||
return {
|
|
||||||
code: code.replace(/\bthis\b/g, 'globalThis'),
|
|
||||||
map: { mappings: '' } // Provide an empty sourcemap to avoid warnings
|
|
||||||
};
|
|
||||||
}
|
|
||||||
return null; // Return null to let Rollup handle other files normally
|
|
||||||
}
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
// Custom plugin to provide empty shims for Node.js built-in modules
|
|
||||||
const nodeModuleShims = () => {
|
|
||||||
return {
|
|
||||||
name: 'node-module-shims',
|
|
||||||
resolveId(source) {
|
|
||||||
// List of Node.js built-in modules to shim
|
|
||||||
const nodeBuiltins = [
|
|
||||||
'fs', 'path', 'util', 'crypto', 'os', 'stream',
|
|
||||||
'http', 'http2', 'https', 'zlib', 'child_process'
|
|
||||||
];
|
|
||||||
|
|
||||||
if (nodeBuiltins.includes(source)) {
|
|
||||||
// Return a virtual module ID for the shim
|
|
||||||
return `\0${source}-shim`;
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
},
|
|
||||||
load(id) {
|
|
||||||
// If this is one of our shims, return an empty module
|
|
||||||
if (id.startsWith('\0') && id.endsWith('-shim')) {
|
|
||||||
console.log(`Providing empty shim for Node.js module: ${id.slice(1, -5)}`);
|
|
||||||
return 'export default {}; export const promises = {};';
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
export default {
|
|
||||||
input: 'examples/browser_compatible_exports.ts', // Browser-compatible entry point
|
|
||||||
inlineDynamicImports: true,
|
|
||||||
output: [
|
|
||||||
{
|
|
||||||
file: 'dist/brainy.js',
|
|
||||||
format: 'es',
|
|
||||||
sourcemap: true,
|
|
||||||
intro: 'var global = typeof window !== "undefined" ? window : this;',
|
|
||||||
},
|
|
||||||
{
|
|
||||||
file: 'dist/brainy.min.js',
|
|
||||||
format: 'es',
|
|
||||||
sourcemap: true,
|
|
||||||
intro: 'var global = typeof window !== "undefined" ? window : this;',
|
|
||||||
plugins: [terser()]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
plugins: [
|
|
||||||
// Add our custom plugins first to ensure they run before other transformations
|
|
||||||
fixThisReferences(),
|
|
||||||
nodeModuleShims(),
|
|
||||||
resolve({
|
|
||||||
browser: true,
|
|
||||||
preferBuiltins: false
|
|
||||||
}),
|
|
||||||
commonjs({
|
|
||||||
transformMixedEsModules: true
|
|
||||||
}),
|
|
||||||
json(),
|
|
||||||
typescript({
|
|
||||||
tsconfig: './tsconfig.browser.json',
|
|
||||||
declaration: false,
|
|
||||||
declarationMap: false
|
|
||||||
})
|
|
||||||
],
|
|
||||||
external: [
|
|
||||||
// Add any dependencies you want to exclude from the bundle
|
|
||||||
// AWS SDK modules with circular dependencies
|
|
||||||
'@aws-sdk/client-s3',
|
|
||||||
'@smithy/util-stream',
|
|
||||||
'@smithy/node-http-handler',
|
|
||||||
'@aws-crypto/crc32c'
|
|
||||||
// Node.js built-ins are now handled by the nodeModuleShims plugin
|
|
||||||
]
|
|
||||||
}
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue