docs(8.0): Phase F — deep clean across 21 docs

Aligned every public doc to the 8.0 contract: filesystem + memory adapters
only, vector index provider terminology (config.vector with recall +
quantization + persistMode knobs), no cloud storage adapters, no closed-
source product names.

Tier 1 — heavier rewrites:
- docs/architecture/storage-architecture.md
- docs/architecture/data-storage-architecture.md
- docs/architecture/distributed-storage.md DELETED — content was 100%
  cloud-coordination examples with no 8.0 substance.
- docs/guides/distributed-system.md DELETED — same reason; no inbound refs.
- docs/SCALING.md rewritten for single-node guidance.
- docs/PLUGINS.md, docs/augmentations/{COMPLETE-REFERENCE,README}.md:
  HnswProvider→VectorIndexProvider, hnsw→vector key.
- docs/PERFORMANCE.md, docs/BATCHING.md cloud-detection + sharding
  sections replaced with single-node vector tuning + filesystem framing.

Tier 2 — surgical renames + cloud-section deletions:
- architecture/{index,initialization-and-rebuild,overview}.md
- transactions.md, DEVELOPER_LEARNING_PATH.md
- vfs/{VFS_API_GUIDE,COMMON_PATTERNS}.md
- api/README.md, guides/{inspection,import-flow}.md

Tier 3 — light edits:
- docs/README.md, architecture/augmentation-system-audit.md

MIGRATION-V3-TO-V4.md untouched (internal migration doc, no stale terms).
This commit is contained in:
David Snelling 2026-06-09 16:13:35 -07:00
parent 2626ab8d62
commit adda1570f3
22 changed files with 570 additions and 2658 deletions

View file

@ -5,7 +5,7 @@ public: true
category: guides
template: guide
order: 4
description: Replace any Brainy subsystem — distance functions, embeddings, HNSW index, metadata index, aggregation — with a custom implementation or native Rust via Cortex.
description: Replace any Brainy subsystem — distance functions, embeddings, vector index, metadata index, aggregation — with a custom implementation or optional native acceleration.
next:
- cortex/comparison
- guides/storage-adapters
@ -13,7 +13,7 @@ next:
# Plugin Development Guide
Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cortex` provides native Rust acceleration, and it's the same system available to any developer.
Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cortex` provides optional native acceleration, and it's the same system available to any developer.
## Architecture Overview
@ -28,7 +28,7 @@ Plugins are **opt-in** — brainy never auto-imports packages. You must explicit
```typescript
const brain = new Brainy({
plugins: ['@soulcraft/cortex'] // explicitly load cortex
plugins: ['@soulcraft/cortex'] // explicitly load the native acceleration package
})
```
@ -109,7 +109,7 @@ Each key has a specific expected signature. Brainy checks for these during `init
#### `distance`
**Type:** `(a: number[], b: number[]) => number`
Replaces the default cosine distance function used in HNSW search and neural APIs. This is the highest-impact single provider — it's called for every vector comparison.
Replaces the default cosine distance function used in vector search and neural APIs. This is the highest-impact single provider — it's called for every vector comparison.
```typescript
context.registerProvider('distance', (a: number[], b: number[]): number => {
@ -151,10 +151,10 @@ context.registerProvider('embedBatch', async (texts: string[]) => {
### Index Providers
#### `hnsw`
**Type:** `(config: object, distanceFunction: Function, options: object) => HNSWIndex-compatible`
#### `vector`
**Type:** `(config: object, distanceFunction: Function, options: object) => VectorIndexProvider-compatible`
Factory function that creates an HNSW index instance. The returned object must implement the `HNSWIndex` public API:
Factory function that creates a vector index instance. The returned object must implement the `VectorIndexProvider` public API:
- `addItem(item: { id: string, vector: number[] }): Promise<string>`
- `search(queryVector: number[], k: number, filter?, options?): Promise<Array<[string, number]>>`
@ -174,12 +174,12 @@ Factory function that creates an HNSW index instance. The returned object must i
- `setUseParallelization(boolean): void`
For type-aware indexes (separate graph per noun type), also implement:
- `getIndexForType(type: string): HNSWIndex` (duck-typed detection)
- `getIndexForType(type: string): VectorIndexProvider` (duck-typed detection)
- `search(queryVector, k, type?, filter?, options?): Promise<Array<[string, number]>>`
```typescript
context.registerProvider('hnsw', (config, distanceFn, options) => {
return new MyNativeHNSWIndex(config, distanceFn, options)
context.registerProvider('vector', (config, distanceFn, options) => {
return new MyNativeVectorIndex(config, distanceFn, options)
})
```
@ -216,7 +216,7 @@ context.registerProvider('aggregation', (storage) => {
})
```
When provided by a native plugin like `@soulcraft/cortex`, this enables:
When provided by an optional native acceleration plugin (such as `@soulcraft/cortex`), this enables:
- Compiled source filters (vs per-entity JS object traversal)
- Precise MIN/MAX via sorted data structures (vs lazy recompute)
- Parallel aggregate rebuild across CPU cores
@ -227,7 +227,7 @@ When provided by a native plugin like `@soulcraft/cortex`, this enables:
#### `cache`
**Type:** `UnifiedCache`
Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and HNSW vector caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`).
Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`).
```typescript
import type { UnifiedCache } from '@soulcraft/brainy/internals'
@ -252,7 +252,7 @@ Native msgpack encode/decode for SSTable serialization.
### Analytics Providers (Native-Only)
These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when a native plugin like `@soulcraft/cortex` is installed.
These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cortex`) is installed.
Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it.
@ -338,11 +338,11 @@ console.log(diag)
// embeddings: { source: 'plugin' },
// embedBatch: { source: 'plugin' },
// distance: { source: 'plugin' },
// hnsw: { source: 'default' },
// vector: { source: 'default' },
// ...
// },
// indexes: {
// hnsw: { size: 0, type: 'TypeAwareHNSWIndex' },
// vector: { size: 0, type: 'JsHnswVectorIndex' },
// metadata: { type: 'MetadataIndexManager', initialized: true },
// graph: { type: 'GraphAdjacencyIndex', initialized: true, wiredToStorage: true }
// }
@ -361,7 +361,7 @@ When a plugin is active, brainy automatically logs a provider summary after `ini
```
[brainy] Plugin activated: @soulcraft/cortex
[brainy] Providers: 8/10 native (@soulcraft/cortex) | default: hnsw, cache
[brainy] Providers: 8/10 native (@soulcraft/cortex) | default: vector, cache
```
This tells you at a glance how many subsystems are accelerated and which ones are falling back to JavaScript. The log respects `config.silent`.