docs: add Claude Code project guide and verified architecture reference
Contributor-friendly CLAUDE.md with setup, conventions, and architecture overview. Comprehensive skills/architecture.md verified against actual codebase covering all 31 source directories, 38+ exports, and major subsystems (distributed, transactions, neural, CLI, MCP, COW, etc). Remove CLAUDE.md from .gitignore since the new version is designed for public contributors.
This commit is contained in:
parent
d453539a4d
commit
089a4d4141
3 changed files with 277 additions and 3 deletions
165
.claude/skills/architecture.md
Normal file
165
.claude/skills/architecture.md
Normal file
|
|
@ -0,0 +1,165 @@
|
|||
# Brainy Architecture Reference
|
||||
|
||||
## What Is Brainy
|
||||
|
||||
@soulcraft/brainy (v7.17.0) is a Universal Knowledge Protocol -- a Triple Intelligence database combining vector search, graph traversal, and metadata filtering in a single library. Published to npm as a public MIT-licensed package.
|
||||
|
||||
## Core Architecture
|
||||
|
||||
### Storage Layer (`src/storage/`)
|
||||
- **StorageAdapter interface** (`src/coreTypes.ts:576`): The contract ALL storage backends implement. ALWAYS check this interface before adding storage methods.
|
||||
- **BaseStorage** (`src/storage/baseStorage.ts`): Base implementation with built-in type-aware partitioning (TypeAwareStorageAdapter was removed -- functionality merged into BaseStorage).
|
||||
- **Adapters** (`src/storage/adapters/`):
|
||||
- `fileSystemStorage.ts` -- local filesystem
|
||||
- `opfsStorage.ts` -- browser Origin Private File System (formerly browserStorage.ts)
|
||||
- `memoryStorage.ts` -- in-memory
|
||||
- `s3CompatibleStorage.ts` -- generic S3-compatible (AWS, MinIO, etc.)
|
||||
- `r2Storage.ts` -- Cloudflare R2
|
||||
- `gcsStorage.ts` -- Google Cloud Storage
|
||||
- `azureBlobStorage.ts` -- Azure Blob Storage
|
||||
- Supporting: `batchS3Operations.ts`, `optimizedS3Search.ts`
|
||||
- **COW** (`src/storage/cow/`): Copy-on-Write infrastructure for versioning and branching
|
||||
- CommitLog, CommitObject, CommitBuilder, BlobStorage, RefManager, TreeObject
|
||||
|
||||
### Vector Search (`src/hnsw/`)
|
||||
- `hnswIndex.ts` -- HNSW-based approximate nearest neighbor search
|
||||
- `typeAwareHNSWIndex.ts` -- type-partitioned vector search
|
||||
- NOT in `src/intelligence/` (that directory does not exist)
|
||||
|
||||
### Graph Engine (`src/graph/`)
|
||||
- `graphAdjacencyIndex.ts` -- adjacency-based graph representation
|
||||
- `pathfinding.ts` -- relationship traversal and pathfinding
|
||||
- `lsm/` -- LSM tree implementation for graph storage
|
||||
|
||||
### Metadata Index (`src/utils/metadataIndex.ts`)
|
||||
- O(1) exact match via hash indexes
|
||||
- O(log n) range queries via sorted indexes
|
||||
- Roaring bitmap set operations for efficient filtering
|
||||
- Adaptive chunking strategy (`metadataIndexChunking.ts`)
|
||||
- Caching layer (`metadataIndexCache.ts`)
|
||||
|
||||
### Triple Intelligence (`src/triple/`)
|
||||
- `TripleIntelligenceSystem.ts` -- combines vector + graph + metadata into unified queries
|
||||
- Lazy-loaded indexes (loaded on first use, not at startup)
|
||||
|
||||
### Neural/AI Components (`src/neural/`)
|
||||
- Smart Importers (`src/importers/`): CSV, Excel, PDF, DOCX, YAML, JSON, Markdown, Orchestrator
|
||||
- `SmartExtractor.ts` -- entity extraction from unstructured data
|
||||
- `SmartRelationshipExtractor.ts` -- relationship detection
|
||||
- `NeuralEntityExtractor.ts` -- ML-based entity recognition
|
||||
- Natural language processing utilities
|
||||
|
||||
### Distributed Systems (`src/distributed/`)
|
||||
- Distributed Coordinator for multi-node operation
|
||||
- Shard Manager for data partitioning
|
||||
- Cache Synchronization across nodes
|
||||
- Read/Write separation
|
||||
- Network and HTTP transport layers
|
||||
- Storage discovery and shard migration
|
||||
|
||||
### Transaction Management (`src/transaction/`)
|
||||
- TransactionManager for ACID operations
|
||||
- Operations: SaveNoun, AddToHNSW, UpdateMetadata, etc.
|
||||
- Distributed transaction support
|
||||
|
||||
### Integration Hub (`src/integrations/`)
|
||||
- Google Sheets integration
|
||||
- OData (Open Data Protocol)
|
||||
- Server-Sent Events (SSE)
|
||||
- Webhooks
|
||||
- Event bus system
|
||||
|
||||
### Virtual Filesystem (`src/vfs/`)
|
||||
- `VirtualFileSystem.ts` -- full VFS implementation (87 KB)
|
||||
- `PathResolver.ts`, `FSCompat.ts`, `MimeTypeDetector.ts`, `TreeUtils.ts`
|
||||
- Subdirectories: `semantic/` (semantic search), `streams/` (streaming), `importers/`
|
||||
|
||||
### MCP Support (`src/mcp/`)
|
||||
- BrainyMCPAdapter, MCPAugmentationToolset, BrainyMCPService
|
||||
- Model Control Protocol request/response handling
|
||||
|
||||
### Additional Systems
|
||||
- **CLI** (`src/cli/`): Complete command-line tool with interactive mode and catalog system
|
||||
- **Migration** (`src/migration/`): MigrationRunner for database schema migrations
|
||||
- **Embeddings** (`src/embeddings/`): Embedding manager with Candle-WASM Rust source
|
||||
- **Streaming** (`src/streaming/`): Pipeline support with adaptive backpressure
|
||||
- **Versioning** (`src/versioning/`): VersioningAPI for data versioning
|
||||
- **Plugin System**: Registry-based plugin architecture
|
||||
- **Patterns** (`src/patterns/`): 7 pattern library JSON files
|
||||
|
||||
## Type System
|
||||
- **NounType** (42 types, `src/types/graphTypes.ts:850-893`): Person, Organization, Concept, Collection, Document, Task, Project, etc.
|
||||
- **VerbType** (127 types, `src/types/graphTypes.ts:900-1087`): Contains, RelatedTo, PartOf, Creates, DependsOn, MemberOf, etc.
|
||||
- All types in `src/types/`
|
||||
|
||||
## Module Exports (`src/index.ts`)
|
||||
38+ named exports including: Brainy class, configuration types, neural APIs (NeuralImport, NeuralEntityExtractor, SmartExtractor, SmartRelationshipExtractor), distance functions, plugin system, migration system, embedding functions, storage adapters, COW infrastructure, pipeline utilities, graph types, MCP components, integration hub, OData utilities, and more.
|
||||
|
||||
## File Structure
|
||||
```
|
||||
src/
|
||||
├── index.ts # 38+ public exports
|
||||
├── brainy.ts # Main Brainy class (6,500+ lines)
|
||||
├── setup.ts # Initialization polyfills
|
||||
├── coreTypes.ts # StorageAdapter interface + core types
|
||||
├── storage/
|
||||
│ ├── baseStorage.ts # Base storage (includes type-aware)
|
||||
│ ├── adapters/ # All storage backends + cloud adapters
|
||||
│ └── cow/ # Copy-on-Write versioning
|
||||
├── hnsw/ # HNSW vector search
|
||||
├── graph/ # Graph engine + pathfinding + LSM
|
||||
├── triple/ # Triple Intelligence system
|
||||
├── neural/ # Smart extractors + NLP
|
||||
├── importers/ # File format importers (8 types)
|
||||
├── distributed/ # Distributed database (16 files)
|
||||
├── transaction/ # ACID transactions (6 files)
|
||||
├── integrations/ # Sheets, OData, SSE, Webhooks
|
||||
├── vfs/ # Virtual filesystem + semantic search
|
||||
├── mcp/ # Model Control Protocol
|
||||
├── cli/ # Command-line interface
|
||||
├── migration/ # Schema migrations
|
||||
├── embeddings/ # Embedding manager + Candle-WASM
|
||||
├── streaming/ # Pipeline + backpressure
|
||||
├── versioning/ # Versioning API
|
||||
├── types/ # TypeScript type definitions
|
||||
├── utils/ # Metadata index, logging, etc.
|
||||
├── config/ # Configuration system
|
||||
├── patterns/ # Pattern library
|
||||
├── api/ # API layer
|
||||
├── interfaces/ # Interface definitions
|
||||
├── shared/ # Shared utilities
|
||||
├── data/ # Data utilities
|
||||
├── errors/ # Error handling
|
||||
├── critical/ # Critical error handling
|
||||
├── universal/ # Universal utilities
|
||||
├── import/ # Import functionality
|
||||
└── scripts/ # Build scripts
|
||||
```
|
||||
|
||||
## Initialization
|
||||
`brainy.ts` `init()` method performs initialization cascade:
|
||||
1. Load plugins
|
||||
2. Initialize storage
|
||||
3. Enable COW (Copy-on-Write)
|
||||
4. Set up embeddings
|
||||
5. Initialize caches
|
||||
6. Set up graph indexes
|
||||
7. Initialize VFS
|
||||
8. Set up transaction manager
|
||||
9. Initialize distributed components (if enabled)
|
||||
|
||||
## Testing
|
||||
- Framework: Vitest
|
||||
- Run: `npm test`
|
||||
- Test directories:
|
||||
- `tests/unit/` -- unit tests
|
||||
- `tests/integration/` -- integration tests
|
||||
- `tests/benchmarks/` -- performance benchmarks (NOT tests/performance/)
|
||||
- `tests/comprehensive/` -- comprehensive test suites
|
||||
- `tests/api/` -- API tests
|
||||
- `tests/helpers/` -- test utilities
|
||||
|
||||
## Release
|
||||
- `npm run release:patch/minor/major` -- fully automated via `scripts/release.sh`
|
||||
- `npm run release:dry` -- preview without changes
|
||||
- Uses conventional commits for changelog generation
|
||||
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -52,14 +52,12 @@ temp/
|
|||
|
||||
# Planning and instruction files
|
||||
plan.md
|
||||
CLAUDE.md
|
||||
|
||||
# Package files
|
||||
*.tgz
|
||||
|
||||
# Private/confidential files
|
||||
PLAN.md
|
||||
CLAUDE.md
|
||||
INTERNAL_NOTES.md
|
||||
TODO_PRIVATE.md
|
||||
*.tar.gz
|
||||
|
|
@ -81,7 +79,6 @@ models-cache/
|
|||
|
||||
# Development planning files (not for commit)
|
||||
PLAN.md
|
||||
CLAUDE.md
|
||||
|
||||
# Backup folders
|
||||
backup-*
|
||||
|
|
|
|||
112
CLAUDE.md
Normal file
112
CLAUDE.md
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
# Brainy - Claude Code Project Guide
|
||||
|
||||
This file provides guidance for Claude Code (and human contributors) when working on the Brainy codebase.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraft/brainy` on npm under the MIT license.
|
||||
|
||||
## Getting Started
|
||||
|
||||
```bash
|
||||
npm install # Install dependencies
|
||||
npm run build # Build the project
|
||||
npm test # Run test suite (Vitest)
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
Full architecture reference: `.claude/skills/architecture.md`
|
||||
|
||||
### Core Systems
|
||||
- **Storage** (`src/storage/`): Pluggable storage backends via StorageAdapter interface (`src/coreTypes.ts`)
|
||||
- **Vector Search** (`src/hnsw/`): HNSW approximate nearest neighbor search
|
||||
- **Graph Engine** (`src/graph/`): Relationship traversal with adjacency index and pathfinding
|
||||
- **Metadata Index** (`src/utils/metadataIndex.ts`): O(1) exact match, O(log n) range queries
|
||||
- **Triple Intelligence** (`src/triple/`): Unified query combining all three intelligence types
|
||||
- **Virtual Filesystem** (`src/vfs/`): Full VFS with semantic search
|
||||
|
||||
### Type System
|
||||
- **NounType** (42 types): Entity classification -- Person, Concept, Collection, Document, Task, etc.
|
||||
- **VerbType** (127 types): Relationship types -- Contains, RelatedTo, PartOf, Creates, DependsOn, etc.
|
||||
- Defined in `src/types/graphTypes.ts`
|
||||
|
||||
## Code Standards
|
||||
|
||||
### TypeScript
|
||||
- Strict mode enabled
|
||||
- Target: ES2020, NodeNext module resolution
|
||||
- All new code must be TypeScript
|
||||
- Follow existing patterns -- read related code before writing
|
||||
|
||||
### Quality
|
||||
- All code must compile without errors
|
||||
- All code must have working tests that exercise real behavior
|
||||
- No stub returns (`return {} as any`)
|
||||
- No incomplete implementations with TODO comments
|
||||
- If something can't be fully implemented, throw an explicit error rather than faking it
|
||||
|
||||
### Verification Before Code Changes
|
||||
1. Check that interfaces and methods actually exist before using them
|
||||
2. Check that type properties are in the type definitions
|
||||
3. Run `npm test` -- tests must pass
|
||||
4. Run `npm run build` -- build must succeed
|
||||
|
||||
### Testing
|
||||
- Framework: Vitest
|
||||
- Tests in `tests/` (unit, integration, benchmarks, comprehensive)
|
||||
- Use in-memory storage for speed where possible
|
||||
- Tests must exercise real behavior, not mock it
|
||||
- Benchmarks are in `tests/benchmarks/` (not tests/performance/)
|
||||
|
||||
## Commit Conventions
|
||||
|
||||
Use [Conventional Commits](https://www.conventionalcommits.org/):
|
||||
|
||||
```
|
||||
feat: add new feature (minor version bump)
|
||||
fix: resolve bug (patch version bump)
|
||||
docs: update documentation (patch version bump)
|
||||
perf: improve performance (patch version bump)
|
||||
refactor: restructure code (patch version bump)
|
||||
test: add/update tests (patch version bump)
|
||||
```
|
||||
|
||||
**Important:** Never use `BREAKING CHANGE` in commit messages. Major version bumps are manual decisions only (`npm run release:major`).
|
||||
|
||||
## Release Process
|
||||
|
||||
Fully automated via `scripts/release.sh`:
|
||||
|
||||
```bash
|
||||
npm run release:dry # Preview (no changes)
|
||||
npm run release:patch # Bug fixes
|
||||
npm run release:minor # New features
|
||||
npm run release:major # Breaking changes (rare, manual decision)
|
||||
```
|
||||
|
||||
The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release.
|
||||
|
||||
## Performance Claims
|
||||
|
||||
When documenting performance characteristics:
|
||||
- **MEASURED**: Cite the test file and line number
|
||||
- **PROJECTED**: Clearly label as extrapolated from tested scale
|
||||
- Never claim a performance figure without context or evidence
|
||||
|
||||
## Debugging
|
||||
|
||||
When a bug persists through 2+ fix attempts, switch to systematic debugging:
|
||||
1. Add comprehensive logging at every step
|
||||
2. Test with production-like data
|
||||
3. Trace the complete execution path
|
||||
4. Check both library code and consumer code
|
||||
5. Verify with actual test execution before declaring fixed
|
||||
|
||||
## Key Paths
|
||||
|
||||
- Main class: `src/brainy.ts`
|
||||
- Public API: `src/index.ts` (38+ exports)
|
||||
- Storage interface: `src/coreTypes.ts`
|
||||
- Type definitions: `src/types/`
|
||||
- Strategy/planning docs: `.strategy/` (gitignored, not public)
|
||||
Loading…
Add table
Add a link
Reference in a new issue