diff --git a/.claude/skills/architecture.md b/.claude/skills/architecture.md deleted file mode 100644 index de046b17..00000000 --- a/.claude/skills/architecture.md +++ /dev/null @@ -1,170 +0,0 @@ -# Brainy Architecture Reference - -## What Is Brainy - -@soulcraft/brainy (v7.17.0) is a Universal Knowledge Protocol -- a Triple Intelligence database combining vector search, graph traversal, and metadata filtering in a single library. Published to npm as a public MIT-licensed package. - -## Core Architecture - -### Storage Layer (`src/storage/`) -- **StorageAdapter interface** (`src/coreTypes.ts:576`): The contract ALL storage backends implement. ALWAYS check this interface before adding storage methods. -- **BaseStorage** (`src/storage/baseStorage.ts`): Base implementation with built-in type-aware partitioning (TypeAwareStorageAdapter was removed -- functionality merged into BaseStorage). -- **Adapters** (`src/storage/adapters/`): - - `fileSystemStorage.ts` -- local filesystem - - `memoryStorage.ts` -- in-memory - - `baseStorageAdapter.ts` -- shared adapter base (counts, batch ops) - - Cloud + OPFS adapters were removed in 8.0 (cloud backup is operator tooling) -- **Generational MVCC / Db API** (`src/db/`): immutable `Db` values over generation-stamped records - - `db.ts` (the `Db` value), `generationStore.ts` (record layer + commit protocol), `types.ts`, `errors.ts`, `whereMatcher.ts` - - Design record: `docs/ADR-001-generational-mvcc.md`; replaced the pre-8.0 COW branching + versioning subsystems - -### Vector Search (`src/hnsw/`) -- `hnswIndex.ts` -- HNSW-based approximate nearest neighbor search -- `typeAwareHNSWIndex.ts` -- type-partitioned vector search -- NOT in `src/intelligence/` (that directory does not exist) - -### Graph Engine (`src/graph/`) -- `graphAdjacencyIndex.ts` -- adjacency-based graph representation -- `pathfinding.ts` -- relationship traversal and pathfinding -- `lsm/` -- LSM tree implementation for graph storage - -### Metadata Index (`src/utils/metadataIndex.ts`) -- O(1) exact match via hash indexes -- O(log n) range queries via sorted indexes -- Roaring bitmap set operations for efficient filtering -- Adaptive chunking strategy (`metadataIndexChunking.ts`) -- Caching layer (`metadataIndexCache.ts`) - -### Triple Intelligence (`src/triple/`) -- `TripleIntelligenceSystem.ts` -- combines vector + graph + metadata into unified queries -- Lazy-loaded indexes (loaded on first use, not at startup) - -### Neural/AI Components (`src/neural/`) -- Smart Importers (`src/importers/`): CSV, Excel, PDF, DOCX, YAML, JSON, Markdown, Orchestrator -- `SmartExtractor.ts` -- entity extraction from unstructured data -- `SmartRelationshipExtractor.ts` -- relationship detection -- `NeuralEntityExtractor.ts` -- ML-based entity recognition -- Natural language processing utilities - -### Distributed Systems (`src/distributed/`) -- Distributed Coordinator for multi-node operation -- Shard Manager for data partitioning -- Cache Synchronization across nodes -- Read/Write separation -- Network and HTTP transport layers -- Storage discovery and shard migration - -### Transaction Management (`src/transaction/`) -- TransactionManager for ACID operations -- Operations: SaveNoun, AddToHNSW, UpdateMetadata, etc. -- Distributed transaction support - -### Integration Hub (`src/integrations/`) -- Google Sheets integration -- OData (Open Data Protocol) -- Server-Sent Events (SSE) -- Webhooks -- Event bus system - -### Virtual Filesystem (`src/vfs/`) -- `VirtualFileSystem.ts` -- full VFS implementation (87 KB) -- `PathResolver.ts`, `FSCompat.ts`, `MimeTypeDetector.ts`, `TreeUtils.ts` -- Subdirectories: `semantic/` (semantic search), `streams/` (streaming), `importers/` - -### MCP Support (`src/mcp/`) -- BrainyMCPAdapter, MCPAugmentationToolset, BrainyMCPService -- Model Control Protocol request/response handling - -### Aggregation Engine (`src/aggregation/`) -- **AggregationIndex** (`AggregationIndex.ts`): Write-time incremental aggregation — SUM, COUNT, AVG, MIN, MAX with GROUP BY and time windows -- **Time Windows** (`timeWindows.ts`): ISO 8601 bucketing — hour, day, week, month, quarter, year, custom intervals -- **Materializer** (`materializer.ts`): Debounced writes of aggregate results as `NounType.Measurement` entities -- Integrates into `brain.find({ aggregate })` for unified query API -- Write hooks in `add()`, `update()`, `delete()` for O(1) incremental updates -- `'aggregation'` provider key enables native plugin acceleration - -### Additional Systems -- **CLI** (`src/cli/`): Complete command-line tool with interactive mode and catalog system -- **Migration** (`src/migration/`): MigrationRunner for database schema migrations -- **Embeddings** (`src/embeddings/`): Embedding manager with Candle-WASM Rust source -- **Streaming** (`src/streaming/`): Pipeline support with adaptive backpressure -- **Versioning** (`src/versioning/`): VersioningAPI for data versioning -- **Plugin System**: Registry-based plugin architecture -- **Patterns** (`src/patterns/`): 7 pattern library JSON files - -## Type System -- **NounType** (42 types, `src/types/graphTypes.ts:850-893`): Person, Organization, Concept, Collection, Document, Task, Project, etc. -- **VerbType** (127 types, `src/types/graphTypes.ts:900-1087`): Contains, RelatedTo, PartOf, Creates, DependsOn, MemberOf, etc. -- All types in `src/types/` - -## Module Exports (`src/index.ts`) -38+ named exports including: Brainy class, configuration types, neural APIs (NeuralImport, NeuralEntityExtractor, SmartExtractor, SmartRelationshipExtractor), distance functions, plugin system, migration system, embedding functions, storage adapters, COW infrastructure, pipeline utilities, graph types, MCP components, integration hub, OData utilities, and more. - -## File Structure -``` -src/ -├── index.ts # 38+ public exports -├── brainy.ts # Main Brainy class (6,500+ lines) -├── setup.ts # Initialization polyfills -├── coreTypes.ts # StorageAdapter interface + core types -├── storage/ -│ ├── baseStorage.ts # Base storage (includes type-aware) -│ ├── adapters/ # All storage backends + cloud adapters -│ └── cow/ # Copy-on-Write versioning -├── hnsw/ # HNSW vector search -├── graph/ # Graph engine + pathfinding + LSM -├── triple/ # Triple Intelligence system -├── neural/ # Smart extractors + NLP -├── importers/ # File format importers (8 types) -├── distributed/ # Distributed database (16 files) -├── transaction/ # ACID transactions (6 files) -├── integrations/ # Sheets, OData, SSE, Webhooks -├── vfs/ # Virtual filesystem + semantic search -├── mcp/ # Model Control Protocol -├── cli/ # Command-line interface -├── migration/ # Schema migrations -├── embeddings/ # Embedding manager + Candle-WASM -├── streaming/ # Pipeline + backpressure -├── versioning/ # Versioning API -├── types/ # TypeScript type definitions -├── utils/ # Metadata index, logging, etc. -├── config/ # Configuration system -├── patterns/ # Pattern library -├── api/ # API layer -├── interfaces/ # Interface definitions -├── shared/ # Shared utilities -├── data/ # Data utilities -├── errors/ # Error handling -├── critical/ # Critical error handling -├── universal/ # Universal utilities -├── import/ # Import functionality -└── scripts/ # Build scripts -``` - -## Initialization -`brainy.ts` `init()` method performs initialization cascade: -1. Load plugins -2. Initialize storage -3. Enable COW (Copy-on-Write) -4. Set up embeddings -5. Initialize caches -6. Set up graph indexes -7. Initialize VFS -8. Set up transaction manager -9. Initialize distributed components (if enabled) - -## Testing -- Framework: Vitest -- Run: `npm test` -- Test directories: - - `tests/unit/` -- unit tests - - `tests/integration/` -- integration tests - - `tests/benchmarks/` -- performance benchmarks (NOT tests/performance/) - - `tests/comprehensive/` -- comprehensive test suites - - `tests/api/` -- API tests - - `tests/helpers/` -- test utilities - -## Release -- `npm run release:patch/minor/major` -- fully automated via `scripts/release.sh` -- `npm run release:dry` -- preview without changes -- Uses conventional commits for changelog generation diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml deleted file mode 100644 index cdb2ab14..00000000 --- a/.github/workflows/ci.yml +++ /dev/null @@ -1,40 +0,0 @@ -name: CI - -on: - push: - pull_request: - -jobs: - node: - name: Node ${{ matrix.node-version }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - node-version: ['22', '24'] - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: ${{ matrix.node-version }} - cache: npm - - run: npm ci - - run: npm run test:unit - - bun: - name: Bun (latest) - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: '22' - cache: npm - - uses: oven-sh/setup-bun@v2 - with: - bun-version: latest - - run: npm ci - # test:bun imports the built dist/, so build first. - - run: npm run build - # Bun as a runtime is the supported Bun story (`bun add` / `bun run`). - - run: npm run test:bun diff --git a/.gitignore b/.gitignore index 64e235b3..781bf0c6 100644 --- a/.gitignore +++ b/.gitignore @@ -31,9 +31,6 @@ coverage/ # Test results tests/results/ -# Filesystem test artifacts (created by integration tests) -test-*/ - # IDE files .vscode/ .idea/ @@ -52,19 +49,21 @@ temp/ # Planning and instruction files plan.md +CLAUDE.md # Package files *.tgz # Private/confidential files PLAN.md +CLAUDE.md INTERNAL_NOTES.md TODO_PRIVATE.md *.tar.gz # Strategy and planning documents (private) .strategy/ -# Removed: PRODUCTION_*.md (now these should be public documentation) +PRODUCTION_*.md DISTRIBUTED_*.md *_ASSESSMENT.md *_ANALYSIS.md @@ -74,11 +73,9 @@ DISTRIBUTED_*.md models/ models-cache/ -# But include bundled WASM model assets -!assets/models/ - # Development planning files (not for commit) PLAN.md +CLAUDE.md # Backup folders backup-* @@ -90,27 +87,9 @@ docs/internal/ # Cache files *.cache -# Rust/Cargo build artifacts -src/embeddings/candle-wasm/target/ -src/embeddings/candle-wasm/Cargo.lock - -# Ignore the wasm-pack output dir's CONTENTS (note the `/*`, not `/`, so the -# re-includes below can take effect — git cannot re-include a file whose parent -# DIRECTORY is excluded). Keep the pre-built WASM committed: it ships in the npm -# package anyway, it lets consumers + CI build without a Rust/wasm-pack toolchain, -# and versioning it makes the shipped artifact reproducible (not "whatever the -# maintainer last built"). -src/embeddings/wasm/pkg/* -!src/embeddings/wasm/pkg/*.wasm -!src/embeddings/wasm/pkg/*.js -!src/embeddings/wasm/pkg/*.d.ts - # Log files (redundant but explicit) *.log # Temporary files (redundant but explicit) *.tmp /.junie/guidelines.md - -# Claude Code harness state -.claude/scheduled_tasks.lock diff --git a/.versionrc.json b/.versionrc.json new file mode 100644 index 00000000..dac6ceca --- /dev/null +++ b/.versionrc.json @@ -0,0 +1,24 @@ +{ + "types": [ + {"type": "feat", "section": "✨ Features"}, + {"type": "fix", "section": "🐛 Bug Fixes"}, + {"type": "docs", "section": "📚 Documentation"}, + {"type": "refactor", "section": "♻️ Code Refactoring"}, + {"type": "perf", "section": "⚡ Performance Improvements"}, + {"type": "test", "section": "✅ Tests"}, + {"type": "build", "section": "🔧 Build System"}, + {"type": "ci", "section": "🔄 CI/CD"}, + {"type": "style", "hidden": true}, + {"type": "chore", "hidden": true} + ], + "compareUrlFormat": "https://github.com/soulcraftlabs/brainy/compare/{{previousTag}}...{{currentTag}}", + "commitUrlFormat": "https://github.com/soulcraftlabs/brainy/commit/{{hash}}", + "issueUrlFormat": "https://github.com/soulcraftlabs/brainy/issues/{{id}}", + "userUrlFormat": "https://github.com/{{user}}", + "releaseCommitMessageFormat": "chore(release): {{currentTag}}", + "issuePrefixes": ["#"], + "header": "# Changelog\n\nAll notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.\n", + "scripts": { + "postbump": "echo '✅ Version bumped to' $(node -p \"require('./package.json').version\")" + } +} \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index fd0de54e..34add83d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3417 +1,6 @@ # Changelog -All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. - -### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19) - -- docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78) -- fix: release drains in-flight writer-lock heartbeat — no phantom lock after unlink (70e4bc8) -- feat: flush() never compacts — history maintenance moves to close() with bounded passes (300d9f2) - - -### [8.8.2](https://github.com/soulcraftlabs/brainy/compare/v8.8.1...v8.8.2) (2026-07-19) - -- fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings (945d92d) -- chore: push public docs to the soulcraft.com ingest door on release (42037d0) - - -### [8.8.1](https://github.com/soulcraftlabs/brainy/compare/v8.8.0...v8.8.1) (2026-07-18) - -- fix: O(1) adaptive retention accounting + historyStats fleet audit (6207e48) -- fix: import dedup off-switch honesty + brain-owned lifecycle for the background pass (4fcef7b) - - -### [8.8.0](https://github.com/soulcraftlabs/brainy/compare/v8.7.1...v8.8.0) (2026-07-17) - -- feat: OS-limit detection for pool-scale deployments (16a73b8) - - -### [8.7.1](https://github.com/soulcraftlabs/brainy/compare/v8.7.0...v8.7.1) (2026-07-17) - -- fix: race-proof writer-lock acquisition + machine-readable conflict through init (01a3b46) - - -### [8.7.0](https://github.com/soulcraftlabs/brainy/compare/v8.6.0...v8.7.0) (2026-07-17) - -- feat: scaled transact budgets + labeled timeout diagnostics + envelope docs (6ef9fcb) - - -### [8.6.0](https://github.com/soulcraftlabs/brainy/compare/v8.5.2...v8.6.0) (2026-07-17) - -- feat: brain.auditGraph() — read-only graph-truth audit (2a03fae) - - -### [8.5.2](https://github.com/soulcraftlabs/brainy/compare/v8.5.1...v8.5.2) (2026-07-17) - -- fix: exception-safe aggregation backfill + generation-verified adoption + loud open-path guards (a77b064) - - -### [8.5.1](https://github.com/soulcraftlabs/brainy/compare/v8.5.0...v8.5.1) (2026-07-17) - -- fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal (da55be7) -- docs: external-backups/sparse-storage guide + generation fact log concept (593bb8b) - - -### [8.5.0](https://github.com/soulcraftlabs/brainy/compare/v8.4.0...v8.5.0) (2026-07-15) - -- test: tolerant timing assertion in the execution-time measure test (4dc0a92) -- feat: committedGeneration capability + pinned durability/stability contracts (d1ecee1) -- docs: RELEASES.md entry for 8.5.0 (provider fact-log access + shared verifier) (e4f37cd) -- feat: provider access to the fact log + shared stamp verifier via internals (352e356) - - -### [8.4.0](https://github.com/soulcraftlabs/brainy/compare/v8.3.3...v8.4.0) (2026-07-15) - -- docs: RELEASES.md entry for 8.4.0 (generation fact log + family stamp) (4a60b43) -- feat: entity-tree family stamp — sourceGeneration + rollup coherence at open (2888ae6) -- feat: generation fact log — after-image commit records, dual-written at every commit point (38b0041) - - -### [8.3.3](https://github.com/soulcraftlabs/brainy/compare/v8.3.2...v8.3.3) (2026-07-15) - -- docs: RELEASES.md entry for 8.3.3 (rename containment fix + repair) (c3feafd) -- test: lens-consistency regression — combined vs subtype-only vs canonical ground truth (4fb41f9) -- fix: VFS rename moves the containment edge — no ghost in the old directory (af8c179) - - -### [8.3.2](https://github.com/soulcraftlabs/brainy/compare/v8.3.1...v8.3.2) (2026-07-14) - -- docs: RELEASES.md entry for 8.3.2 (honest counters) (0932ecd) -- fix: honest counters — removal never re-reads the removed record + repairIndex recounts and persists all rollups (2e2ba9c) - - -### [8.3.1](https://github.com/soulcraftlabs/brainy/compare/v8.3.0...v8.3.1) (2026-07-14) - -- docs: RELEASES.md entry for 8.3.1 (full-removal deletes + family-scoped gate) (c0c68ac) -- fix: full-removal canonical deletes + family-scoped migration gate (366f9a9) -- docs: cite the cross-layer integrity contract generically in comments and notes (1d26988) - - -### [8.3.0](https://github.com/soulcraftlabs/brainy/compare/v8.2.8...v8.3.0) (2026-07-13) - -- docs: RELEASES.md entry for 8.3.0 (heal-cost + cross-layer integrity contract) (7692c6f) -- perf: parallel + id-only canonical enumeration (heal-cost dominant term) (ec5b933) -- feat: registered-blob family contract — declared index blobs are undeletable (bfa1762) -- feat: validateIndexConsistency delegates to provider invariants (6bcb54f) - - -### [8.2.8](https://github.com/soulcraftlabs/brainy/compare/v8.2.7...v8.2.8) (2026-07-13) - -- fix: honest index readiness — no silently-empty queries on a cold index (d0f69c7) - - -### [8.2.7](https://github.com/soulcraftlabs/brainy/compare/v8.2.6...v8.2.7) (2026-07-13) - -- fix: restore loadBinaryBlob fault-propagation (native column-store lockstep) (b6c7039) - - -### [8.2.6](https://github.com/soulcraftlabs/brainy/compare/v8.2.5...v8.2.6) (2026-07-13) - -- docs: RELEASES.md entry for 8.2.6 (write/index-spine hardening) (a873852) -- chore: hold loadBinaryBlob fault-propagation for the cortex column-store lockstep (36c10c1) -- fix: aggregation surfaces materialize/state-load failures loudly (02eff64) -- fix: surface a degraded derived index on reads instead of serving it silently (ba958d9) -- fix: saveBinaryBlob never acks a durable write that stored nothing (7feba49) -- fix: refuse writes when single-op history cannot be made durable (54c1836) -- fix: clear() wipes the full native/derived footprint, not a subset (d8301f8) -- fix: surface segment/entity read faults loudly instead of masking as absent (af5d2f3) -- fix: spine hardening pass 1 (part) — count symmetry, honest partial-load, flush durability, read-fault propagation (119087a) -- test: pin the read-your-writes contract under the single writer (eb9c4eb) - - -### [8.2.5](https://github.com/soulcraftlabs/brainy/compare/v8.2.4...v8.2.5) (2026-07-12) - -- docs: RELEASES.md entry for 8.2.5 (honest rollback-failure response) (a7c7aa5) -- fix: honest response when a transaction rollback cannot complete (711d2f0) - - -### [8.2.4](https://github.com/soulcraftlabs/brainy/compare/v8.2.3...v8.2.4) (2026-07-12) - -- docs: RELEASES.md entry for 8.2.4 (non-destructive restore) (4574695) -- fix: non-destructive, crash-resumable restore (a2f4f6a) - - -### [8.2.3](https://github.com/soulcraftlabs/brainy/compare/v8.2.2...v8.2.3) (2026-07-12) - -- docs: RELEASES.md entry for 8.2.3 (transact durability barrier) (be5ce0b) -- fix: transact durability barrier — committed transactions are durable on return (3b8fa51) - - -### [8.2.2](https://github.com/soulcraftlabs/brainy/compare/v8.2.1...v8.2.2) (2026-07-11) - -- docs: RELEASES.md entry for 8.2.2 (transaction timeout rollback) (ed97006) -- fix: transaction timeout rolls back applied operations (no torn state) (508a8e3) - - -### [8.2.1](https://github.com/soulcraftlabs/brainy/compare/v8.2.0...v8.2.1) (2026-07-10) - -- test: update graph-index operation constructors to the VerbEndpointInts signature (62a449d) -- docs: RELEASES.md entry for 8.2.1 (transact forward-ref parity fix) (7089782) -- fix: transact forward references resolve graph endpoint ints at execute time (a175406) - - -### [8.2.0](https://github.com/soulcraftlabs/brainy/compare/v8.1.0...v8.2.0) (2026-07-10) - -- docs: RELEASES.md entry for 8.2.0 (temporal VFS) (98ceadc) -- feat: temporal VFS — file content joins the Model-B immutability model (a3467e1) -- docs: pin the write-path invariant in the plugin contract (the onChange change-feed guarantee) (4af8fb3) - - -### [8.1.0](https://github.com/soulcraftlabs/brainy/compare/v8.0.17...v8.1.0) (2026-07-10) - -- docs: RELEASES.md entry for 8.1.0 (brain.onChange change feed) (4e9be08) -- feat: brain.onChange — the in-process change feed for every committed mutation (fd5edb5) - - -### [8.0.17](https://github.com/soulcraftlabs/brainy/compare/v8.0.16...v8.0.17) (2026-07-08) - -- docs: RELEASES.md entry for 8.0.17 (canonical count recovery + dead-machinery sweep) (6b8b9cb) -- fix: count recovery scans the canonical layout; remove the dead 7.x hnsw sharding machinery (352e2da) - - -### [8.0.16](https://github.com/soulcraftlabs/brainy/compare/v8.0.15...v8.0.16) (2026-07-08) - -- docs: RELEASES.md entry for 8.0.16 (atomic ifAbsent/upsert + exact blob refCounts) (54e7c0e) -- fix: atomic ifAbsent/upsert inserts + exact blob reference counts under concurrency (867939e) - - -### [8.0.15](https://github.com/soulcraftlabs/brainy/compare/v8.0.14...v8.0.15) (2026-07-08) - -- docs: RELEASES.md entry for 8.0.15 (atomic ifRev CAS) (b1fe25a) -- fix: ifRev CAS is atomic — the revision check now runs under the commit mutex (9a3d1bd) - - -### [8.0.14](https://github.com/soulcraftlabs/brainy/compare/v8.0.13...v8.0.14) (2026-07-07) - -- docs: RELEASES.md entry for 8.0.14 (migration preserves branch-scoped non-entity state) (64188a3) -- fix: 7→8 migration preserves branch-scoped non-entity state instead of deleting it (a93bb4e) - - -### [8.0.13](https://github.com/soulcraftlabs/brainy/compare/v8.0.12...v8.0.13) (2026-07-07) - -- docs: RELEASES.md entry for 8.0.13 (accurate boot log for established stores) (38e8de5) -- fix: an established store no longer boot-logs "New installation" (3086916) - - -### [8.0.12](https://github.com/soulcraftlabs/brainy/compare/v8.0.11...v8.0.12) (2026-07-07) - -- docs: RELEASES.md entry for 8.0.12 (7→8 VFS recovery, zero-rebuild cold open, strict query operators) (d9017e7) -- fix: recover VFS content blobs stranded by a 7→8 upgrade, in place on open (c0f6ccd) -- fix: validate where-operators and align the in-memory matcher to the documented set (6821e19) -- docs: correct rc-era time-travel staleness + record the embedding-model ordering constraint (68da660) -- fix: cold-open no longer re-derives durable indexes — complete the readiness contract for all three providers (61c247c) -- docs: RELEASES.md entry for 8.0.11 (exit-hang class closed for every op shape) (4fde94b) - - -### [8.0.11](https://github.com/soulcraftlabs/brainy/compare/v8.0.10...v8.0.11) (2026-07-02) - -- fix: no script shape can hang on brainy's internals — unref every maintenance timer + one-shot beforeExit (30eacbd) -- docs: RELEASES.md entry for 8.0.10 (clean process exit after close) (2da2736) - - -### [8.0.10](https://github.com/soulcraftlabs/brainy/compare/v8.0.9...v8.0.10) (2026-07-02) - -- fix: a bare script now exits cleanly after close() — release every process keep-alive (c540d63) - - -### [8.0.9](https://github.com/soulcraftlabs/brainy/compare/v8.0.8...v8.0.9) (2026-07-02) - -- feat: guarded plugin auto-detection — installing @soulcraft/cor is the opt-in (588267b) - - -### [8.0.8](https://github.com/soulcraftlabs/brainy/compare/v8.0.7...v8.0.8) (2026-07-02) - -- docs: plugins are explicit opt-in — correct the README scale section and plugins config comment (e420369) - - -### [8.0.7](https://github.com/soulcraftlabs/brainy/compare/v8.0.1...v8.0.7) (2026-07-02) - -- docs: GA version is 8.0.7 — npm retired 8.0.0-8.0.6 (January dev-cycle unpublishes) (5db2c41) -- chore(release): 8.0.1 (48bea9e) -- docs: flagship README for the 8.0 GA; GA version is 8.0.1 (e44620e) -- docs: rename the native provider to @soulcraft/cor across public docs and JSDoc (bf4a333) -- chore(release): 8.0.0 (a3c2717) -- docs: RELEASES.md 8.0.0 GA entry (RC notes become history) (4584d0b) -- feat: promote the 8.0 u64-id line to main for the 8.0.0 GA (55d57f8) -- fix(8.0): byte-copy _id_mapper/* in the pre-upgrade backup (cor's write-new nuance) (a30ed72) -- chore(release): 7.33.5 (29b9d5f) -- fix: metadata cold-read guard — no more silent [] on cold find({where}) (7.33.5) (9dc4c5e) -- fix(8.0): metadata cold-read guard — no more silent [] on cold find({where}) (79e8709) -- docs(8.0): add module JSDoc to typeValidation.ts (the one file missing a module block) (ab53fa0) -- feat(8.0): auto pre-upgrade backup — hard-link snapshot before the 7.x→8.0 migration (1aad1f6) -- fix(ci): commit the prebuilt wasm pkg + build before test:bun (green CI on fresh clone) (ed178e2) -- chore(release): 7.33.4 (2be3d0f) -- fix: never serve a silent [] from find({connected}) on a cold-loaded graph (fd699d0) -- chore(release): 7.33.3 (d1665bb) -- fix: re-validate find() results against the predicate (index-integrity guard) (7b5db0d) -- chore(release): 7.33.2 (9593a27) -- fix: graph adjacency cold-load consistency guard — no more silent [] on connected (1694f68) -- chore(release): 7.33.1 (811c7da) -- fix: getNouns cursor pagination re-scanned the first page forever (permanent CPU loop) (6721c52) -- chore(release): 7.33.0 (526aaad) -- feat: visibility tier (public/internal/system) on nouns + verbs (3a62445) -- chore(release): 7.32.2 (c53dd61) -- refactor: rename BackupData → PortableGraph (the type is interchange, not a backup) (89036de) -- chore(release): 7.32.1 (5e7379d) -- fix: getNouns().totalCount reports true total, not page size; quiet benign mmap-vector log (edff637) -- chore(release): 7.32.0 (adec0ba) -- feat: portable graph export()/import() (BackupData v1) on brain.data() (a408d37) -- chore(release): 7.31.8 (89c6d04) -- fix: query-cap memory misread (MemAvailable + floor) + rootDirectory getter for native mmap fast-path (3f8e097) -- chore(release): 7.31.7 (4f8159c) -- fix: vfs.rename() issues a metadata-only update + rollback of fresh adds removes them (ac29b0e) -- chore(release): 7.31.6 (9b52629) -- fix: remap reserved fields from update() metadata patches to their canonical location (67e5fc8) -- chore(release): 7.31.5 (e5ec658) -- fix: feature-detect setVectorBackend before wiring the mmap-vector backend (a537b36) -- chore(release): 7.31.4 (a8cbab6) -- fix: feature-detect setConnectionsCodec before wiring the connections codec (747ab97) -- chore(release): 7.31.3 (cfb051c) -- fix: mmap-vector backend capacity NaN at the provider FFI boundary (eade6ff) - - -### [8.0.0-rc.9](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.8...v8.0.0-rc.9) (2026-07-01) - -- docs(8.0): RELEASES.md — rc.9 (migration LOCK + 6x cosine + ES2023/Node22 floor) (3a33987) -- chore(8.0): ES2023 target + drop DOM lib + downlevelIteration (config truth-up) (cf74c25) -- perf(8.0): allocation-free distance loops (6x cosine) — evidence-revised Fork X (b5bc73f) -- feat(8.0): #18 coordinated migration LOCK — block-and-queue the 7.x→8.0 auto-upgrade (67bbf69) -- chore(8.0): modernize toolchain + position Bun as a runtime (ca9129a) - - -### [8.0.0-rc.8](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.7...v8.0.0-rc.8) (2026-06-30) - -- docs(8.0): RELEASES.md — rc.8 (no-freeze online whole-brain auto-upgrade) (5af48a9) -- feat(8.0): no-freeze auto-upgrade hooks — isMigrating() deference + stampBrainFormat() + brain-format export (b6b9198) - - -### [8.0.0-rc.7](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.6...v8.0.0-rc.7) (2026-06-30) - -- docs(8.0): RELEASES.md — rc.7 (cold-graph self-heal + billion-scale RAM + version handshake) (1ddc786) -- perf(8.0): bound per-id generation history chains (O(W+L) resident, was O(N)) (a859d6e) -- feat(8.0): eager graphIndex.init() before the isReady() rebuild gate (8f4787b) -- feat(8.0): version-handshake marker (formatInfo + indexEpoch) for whole-brain auto-upgrade (fc7f110) -- fix(8.0): never serve a silent [] from find({connected}) on a cold-loaded graph (229b067) -- perf(8.0): represent the committed-generation ledger as an interval set (93f61db) -- perf(8.0): drop O(N)-resident id-keyed storage caches; source counts from the record (b6beb7f) - - -### [8.0.0-rc.6](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.5...v8.0.0-rc.6) (2026-06-29) - -- docs(8.0): RELEASES.md — rc.6 (perf + native-provider contract + test hygiene) (6daa70e) -- feat(8.0): wire the two cor-confirmed metadata-provider contract additions (8b19122) -- test(8.0): re-home orphaned test files into the gate + guard against recurrence (3f9f140) -- perf(8.0): negation/absence where-operators via roaring-bitmap difference (5f974ab) -- perf(8.0): HNSW removeItem is O(in-degree) via a reverse-adjacency index (72df557) - - -### [8.0.0-rc.5](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.4...v8.0.0-rc.5) (2026-06-29) - -- docs(8.0): RELEASES.md — rc.5 hardening + the breaking operator removal (6c9a438) -- refactor(8.0): remove the 4 deprecated query-operator aliases (clean break) (ddcc0c7) -- refactor(8.0): remove dead/deprecated code (legacy sweep) (b9369f2) -- refactor(8.0): API-surface + quality polish from the readiness audit (a52dba2) -- fix(8.0): close GA-blocking correctness gaps from the readiness audit (47e8031) -- docs(8.0): correct public docs to the real 8.0 API + honest perf claims (40d2cd5) -- fix(8.0): re-validate find() results against the predicate (index-integrity guard) (3d11619) - - -### [8.0.0-rc.4](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.3...v8.0.0-rc.4) (2026-06-24) - -- docs(8.0): drop the DeletedItemsIndex section + pseudo-code from index-architecture (e7b50cf) -- fix(8.0): gate native graph analytics on the provider readiness flag (d321cf5) -- refactor(8.0): remove dead, unreachable, and unwired modules (bf0afe8) -- build(8.0): clean dist before every build so stale artifacts never ship (03d6540) -- feat(8.0): #35 part-3 — supply at-gen candidate vectors for the native exact-rerank (c9e2169) - - -### [8.0.0-rc.3](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.2...v8.0.0-rc.3) (2026-06-23) - -- test(8.0): de-flake the VFS path-cache timing assertion (0e8972c) -- feat(8.0): asOf at-gen vector defer — provider-served historical semantic search (#35) (1c363e8) -- perf(8.0): bound find({ where, orderBy }) sort to the page (CTX-BR-FIND-ORDERBY) (450084b) -- feat(8.0): brain.graph.subgraph(query) query→expand fusion (#61) (82dde92) -- feat(8.0): vector allowedIds predicate-pushdown into find() (#46) (dd325f2) -- feat(8.0): graph analytics — brain.graph.rank / communities / path (632d90a) -- fix(8.0): restore() reloads a native entity-id mapper before graphIndex.rebuild() (4d0b64f) -- test(8.0): cover pending-tier range queries + setRetentionBudget adaptive reclaim (3783e61) -- feat(8.0): Model-B per-write generation-stamping + adaptive retention knob (5c3bb2c) -- test(8.0): Model-B write-perf + scalability spike harnesses (afac7f9) -- perf(8.0): per-id history chains for O(log) historical reads + bounded delta cache (ceed70d) -- refactor(8.0): graph analytics contract — intent names, not algorithm names (f3e6911) -- docs(8.0): RELEASES — native provider is @soulcraft/cor 3.0 (fix cortex 3.0 self-contradiction) (96d9c0b) -- test(8.0): cover the native graph seam + make provider resolution factory-tolerant (29410bc) - - -### [8.0.0-rc.2](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.1...v8.0.0-rc.2) (2026-06-21) - -- docs(8.0): RELEASES rc.2 additions — graph engine + additive wins + correctness fixes (18f27cb) -- feat(8.0): brain.graph.export() + noun-walk cursor + noun visibility hydration (c2a84c9) -- feat(8.0): brain.graph.subgraph() + native-provider routing (8c2b57a) -- feat(8.0): related({ node }) — one-call both-direction incident edges (d4de48d) -- feat(8.0): GraphAccelerationProvider contract — the native graph-engine seam (a3d6fdb) -- perf(8.0): cursor pagination for the verb walk — full edge pagination O(N²) → O(N) (682e786) -- perf(8.0): visibility-aware fast adjacency — related() stays O(degree) under default visibility (a914313) -- feat(8.0): upsert + FindParams.includeVectors + removeMany adaptive chunking (4cc2088) -- docs(8.0): note reserved-field default-throw in RELEASES rc additions (1bc709d) -- feat(8.0): reserved-field enforcement — reservedFieldPolicy defaults to throw (54c7c39) -- docs: mark 8.0.0-rc.1 published (npm tag rc) + note rc.1 additions (ae3fe82) - - -### [8.0.0-rc.1](https://github.com/soulcraftlabs/brainy/compare/v7.31.2...v8.0.0-rc.1) (2026-06-20) - -- feat(8.0): id-normalization (#18) + aggregation min/max delete-safety + RC-safe release (d02e522) -- feat(8.0): API simplification — remove neural()/Db.search, one storage `path` key, integration→0 (606445c) -- feat(8.0): 7.x→8.0 layout migration — fix silent total data loss on first open (0c4a51c) -- feat(8.0): temporal range verbs — diff, history, since(gen|Date), asOf{exclusive}, transactionLog window (2c84f86) -- refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup) (373a481) -- fix(8.0): VFS path-cache instance-scoping + verb totalCount page-cap (3a3aa43) -- fix(8.0): multi-valued array fields index every element (contains no longer misses) (eccf420) -- fix(8.0): column-store range queries honor exclusive bounds (lessThan/greaterThan) (009e506) -- fix(8.0): per-type counts rehydrate after cold reopen (column store, not dead sparse index) (d918f49) -- feat(8.0): version-coupling guard — a mismatched/failed native plugin fails loud, never silent JS fallback (1264fec) -- test(8.0): boundary guard forbids @soulcraft/cor too (cortex→cor rename) (b198281) -- fix(8.0): stats() per-type counts no longer inflate with HNSW re-saves (21d02d3) -- fix(8.0): getNouns().totalCount reports true total, not page size (port of 7.32.1) (b2005ff) -- fix(8.0): real bugs surfaced by integration hardening — where-intersect, related() offset, relate() updatedAt (5eaf579) -- test(8.0): integration rot pass — 77→17 failures (parallel per-file hardening) (e5997a1) -- test(8.0): begin integration rot pass — clear-persistence (drop COW internals) + metadata-only addRelationship→relate (c600468) -- docs(8.0): RELEASES — portable export/import (BackupData v1) + distinctCount any-type section (4741e23) -- fix(8.0): distinctCount aggregates distinct values of any type + edge-case regression tests (574a8b1) -- feat(8.0): validateBackup() dry-run + includeContent blob round-trip test + clone test (7aad803) -- docs(8.0): export/import guide + api/README portable backup section (c2b73d4) -- feat(8.0): portable graph export()/import() (BackupData v1) — Db.export + polymorphic import (010ccf8) -- feat(8.0): visibility field (public/internal/system) on nouns + verbs (f4dea80) -- test(8.0): close brains in afterEach (count-sync, get-relations teardown) (0ca0e5c) -- fix(8.0): neural.clusters()/similarity must request vectors from get() (cc1a431) -- test(8.0): drop dead s3/distributed/cloud scripts + 32GB→8GB integration heap (73a7d82) -- test(8.0): remove dead s3/distributed/cloud scripts + stale s3 suite (af1ee46) -- test(8.0): Tier-1 integration via deterministic embedder (suite runnable again) (542b52e) -- test(8.0): use valid camelCase VerbType values in test-factory (e31ba89) -- test(8.0): get() resolves null for absent custom ids instead of throwing (dc94af3) -- fix(8.0): accept application-supplied entity ids, not just UUIDs (36b7216) -- fix(8.0): honor top-level storage.path as a rootDirectory alias (5096f90) -- feat(8.0): thread commit generation through the graph-write provider contract (0951fa1) -- fix(8.0): drive query-cap off MemAvailable + floor auto-detected caps (b26d3d4) -- test(8.0): A/B benchmark harness (open leg) — generic corpus+metrics lib, brainy-alone scaling bench, boundary guard, real-embedding recall guard (c605b34) -- docs(8.0): remove unbacked Cortex '5.2x' perf claim + dangling /docs/cortex/comparison link (33caa52) -- docs(8.0): measured find() performance at 5k/100k in SCALING.md (f986832) -- test(8.0): asOf() error-path spot-checks + find() triple-composition correctness + scale-bench harness (af96064) -- docs(8.0): RELEASES.md — record removed BrainyZeroConfig + isFullyInitialized/awaitBackgroundInit in the breaking-change inventory (f12ca68) -- refactor(8.0)!: remove orphaned zero-config subsystem + dead cloud/progressive-init storage vestige (35b9d7e) -- refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider (00d3203) -- feat(8.0): zero-config finalize + cut JS quantization (config.vector = recall + persistMode) (f8e0079) -- fix(8.0): vfs.rename() issues a metadata-only update (port of the 7.31.7 fix) (f4c5d97) -- chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase (1f7e365) -- feat(8.0): reserved-field contract — one canonical location, typed prevention, unified read/write (970e08c) -- feat(8.0): brain.fillSubtypes migration helper + pre-RC1 gap closure (c446783) -- docs(8.0): RELEASES.md 8.0.0 release-candidate entry — full breaking-change inventory + upgrade guide (9b0f4ac) -- refactor(8.0): delete DataAPI — superseded by Db persist/restore + import API + stats (478fa17) -- docs(8.0): consistency-model concept + snapshots guide — Db API replaces branching docs (cc8037d) -- feat(8.0): full query surface at historical generations via ephemeral index materialization (e5feae4) -- feat(8.0)!: delete fork/branch/commit/history/versions — superseded by the Db API (8f93add) -- feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist) (431cd64) -- fix(8.0): createIndex resolves the canonical 'vector' provider key — drop diskann/hnsw key lookups + legacy migration APIs (49e4948) -- feat(8.0): u64 BigInt graph provider contract — punch list a-d,g,h (2427bb7) -- chore(8.0): delete vectorStore:mmap wiring — dead in the 8.0 provider world (62f6472) -- chore(8.0): collapse dead defensive guards + redundant polyfills (42159f2) -- chore(8.0)!: drop browser support, cloud SDKs, legacy pipeline, dead threading (266715a) -- docs(8.0): Phase F — deep clean across 21 docs (adda157) -- chore(8.0): Phase C + D + E — config simplification, TODO sweep, test race fix (2626ab8) -- chore(8.0): Phase A + B — purge all @deprecated APIs + cacheManager dead branches (cb16a39) -- chore(8.0): step-7 follow-through — collapse remaining cloud branches + docs sweep (scaffold step 13) (9f9a415) -- feat(8.0)!: flip requireSubtype default to true (BRAINY-8.0-SUBTYPE-CONTRACT § C-1) (780fb64) -- fix(8.0): implement multi-hop subtype BFS in pure JS (open-core works standalone) (221fc45) -- refactor(8.0): SubtypeRegistry hook + drop multi-hop subtype throw (scaffold steps 11-12) (ed75f25) -- docs(8.0): document subtype required-by-default deferral (scaffold step 10) (1eb0ffc) -- refactor(8.0): drop strictConfig — surface too small to justify the option (scaffold step 9) (694a31f) -- refactor(8.0): simplify config.vector to 3 knobs + fold persistMode (scaffold step 8) (8e76740) -- refactor(8.0): drop cloud + OPFS storage adapters; filesystem + memory only (scaffold step 7) (0e6263a) -- refactor(8.0): final cleanup — drop HnswProvider alias + config.hnsw + 'hnsw' surface (scaffold step 6) (b20666e) -- refactor(8.0): strictConfig + brain.stats() vector field + wireConnectionsCodec feature-detect (scaffold step 5) (3e1ef95) -- refactor(8.0): add saveVectorIndexData / getVectorIndexData storage contract (scaffold step 4) (356f044) -- refactor(8.0): rename HNSWIndex class → JsHnswVectorIndex (scaffold step 3) (f39d420) -- refactor(8.0): add config.vector path + 'vectors' cache category (scaffold step 2) (8f87b35) -- refactor(8.0): rename HnswProvider → VectorIndexProvider (8.0 scaffold) (076c26f) - - -### [7.31.2](https://github.com/soulcraftlabs/brainy/compare/v7.31.1...v7.31.2) (2026-06-09) - -- docs: correct misleading SQ4 quantization comment in type definitions (89e4d81) - - -### [7.31.1](https://github.com/soulcraftlabs/brainy/compare/v7.31.0...v7.31.1) (2026-06-09) - -- fix: saveBinaryBlob unique tmp suffix + ENOENT swallow on rename (550bd4a) - - -### [7.31.0](https://github.com/soulcraftlabs/brainy/compare/v7.30.2...v7.31.0) (2026-06-09) - -- feat: per-entity _rev + update({ ifRev }) CAS + add({ ifAbsent }) (bafb4e4) - - -### [7.30.2](https://github.com/soulcraftlabs/brainy/compare/v7.30.1...v7.30.2) (2026-06-08) - -- fix: recalibrate find({ limit }) cap + two-tier enforcement + caller location (9e307e4) - - -### [7.30.1](https://github.com/soulcraftlabs/brainy/compare/v7.30.0...v7.30.1) (2026-06-08) - -- fix: internal subtype consistency + brain.audit() diagnostic + improved enforcement errors (5f3a2ca) - - -### [7.30.0](https://github.com/soulcraftlabs/brainy/compare/v7.29.0...v7.30.0) (2026-06-05) - -- feat: verb subtype + updateRelation + requireSubtype enforcement (c0d326b) - - -### [7.29.0](https://github.com/soulcraftlabs/brainy/compare/v7.28.0...v7.29.0) (2026-06-04) - -- feat: subtype top-level field + trackField + migrateField (2cdf70e) -- feat(8.0): EntityIdMapper U32 ceiling + EntityIdSpaceExceeded error (e47fea0) -- feat: DiskANN auto-engagement + migrateToDiskAnn/migrateToHnsw (8f130d3) -- feat(plugin): DiskAnnProvider contract + HNSWConfig.type/diskann knobs (f885f81) - - -### [7.28.0](https://github.com/soulcraftlabs/brainy/compare/v7.27.0...v7.28.0) (2026-05-28) - -- feat: SQ4 (4-bit) scalar quantization + native distance hook (2.5.0 #30) (73e7e39) - - -### [7.27.0](https://github.com/soulcraftlabs/brainy/compare/v7.26.0...v7.27.0) (2026-05-28) - -- feat: content-type-aware compression policy in COW BlobStorage (2.5.0 #32) (178ff02) - - -### [7.26.0](https://github.com/soulcraftlabs/brainy/compare/v7.25.0...v7.26.0) (2026-05-28) - -- feat: graph link compression — delta-varint connections (2.4.0 #3) (617c156) -- feat: column-store JS↔native interchange — raw-blob unify (2.4.0 #4) (71bc30b) -- feat: mmap-vector backend wiring — HNSWIndex consumes vectorStore:mmap (2.4.0 #2) (d4cb26c) -- feat: stable EntityIdMapper — rebuild() no longer renumbers UUID→int (b2408cb) - - -### [7.25.0](https://github.com/soulcraftlabs/brainy/compare/v7.24.0...v7.25.0) (2026-05-27) - -- docs: remove stale distanceSQ8 JSDoc left by the SQ8 hook refactor (6099101) -- feat: export provider contracts for the plugin surface brainy consumes (4b6f63e) -- feat: hook native sort:topK provider into search result ranking (46fc7f2) -- feat: hook native SQ8 distance provider into HNSW reranking (00d14cf) -- merge: storage binary-blob primitive across all adapters (e23361c) -- fix: code-point string collation in LSM SSTable, COW trees/refs, sorted queries (7493d8e) -- feat(storage): add raw binary-blob primitive to every storage adapter (298b572) -- fix: deterministic code-point string collation for column store + aggregation (547721a) -- feat: exact percentile and distinctCount aggregation ops (fe4f5df) - - -### [7.24.0](https://github.com/soulcraftlabs/brainy/compare/v7.23.0...v7.24.0) (2026-05-26) - -- feat: array-unnest groupBy for aggregates + batch-embed entity extraction (c2e21b7) - - -### [7.23.0](https://github.com/soulcraftlabs/brainy/compare/v7.22.1...v7.23.0) (2026-05-26) - -- feat: queryAggregate() + HAVING, plus aggregate backfill, traversal depth/via, extraction typing (BR-ADV-FEATURES-BUN) (1a98e42) -- chore(release): create annotated tag so --follow-tags pushes it (513186d) - - -### [7.22.1](https://github.com/soulcraftlabs/brainy/compare/v7.22.0...v7.22.1) (2026-05-26) - -- fix: extraction, multi-hop traversal, and aggregate result shape (BR-ADV-FEATURES-BUN) (0a9d1d9) -- docs: storage-adapter inheritance contract + correct the hasStorageMethod story (07754d1) - - -### [7.22.0](https://github.com/soulcraftlabs/brainy/compare/v7.21.0...v7.22.0) (2026-05-15) - -- fix: find()/stats() correctness + Cortex compat (BR-FIND-WHERE-ZERO, BR-DEFENSIVE-INTERFACE) (7026311) - - -### [7.21.0](https://github.com/soulcraftlabs/brainy/compare/v7.20.0...v7.21.0) (2026-05-15) - -- chore: gitignore Claude Code harness scheduled-tasks lockfile (a8fcc3d) -- feat: multi-process safety + read-only inspector mode (4fcdc0f) - - -### [7.20.0](https://github.com/soulcraftlabs/brainy/compare/v7.19.19...v7.20.0) (2026-04-10) - -- refactor: delete dead sparse index write path (11be039) -- feat: unified column store for filtering + sorting at billion scale (46583f2) - - -### [7.19.19](https://github.com/soulcraftlabs/brainy/compare/v7.19.18...v7.19.19) (2026-04-09) - -- refactor: migrate aggregation + neural field reads to resolveEntityField (108e2bc) - - -### [7.19.18](https://github.com/soulcraftlabs/brainy/compare/v7.19.17...v7.19.18) (2026-04-09) - -- feat: export resolveEntityField + STANDARD_ENTITY_FIELDS from internals (beefacb) - - -### [7.19.17](https://github.com/soulcraftlabs/brainy/compare/v7.19.16...v7.19.17) (2026-04-09) - -- fix: correct orderBy sort for timestamp fields via centralized field resolver (be6c4dc) - - -### [7.19.15](https://github.com/soulcraftlabs/brainy/compare/v7.19.14...v7.19.15) (2026-03-23) - -- fix: commit() now flushes and captures state by default (b58ea02) - - -### [7.19.14](https://github.com/soulcraftlabs/brainy/compare/v7.19.13...v7.19.14) (2026-03-22) - -- feat: add setMaxSize() for dynamic cache resizing (54865b3) - - -### [7.19.13](https://github.com/soulcraftlabs/brainy/compare/v7.19.12...v7.19.13) (2026-03-22) - -- fix: suppress misleading 'Using Q8 WASM' log when Cortex native is active (60a0f10) -- perf: defer HNSW persistence during addMany() batch operations (973b6aa) - - -### [7.19.10](https://github.com/soulcraftlabs/brainy/compare/v7.19.9...v7.19.10) (2026-02-24) - -- fix: replace require('crypto') with ESM import in SSTable (239a4da) - - -### [7.19.9](https://github.com/soulcraftlabs/brainy/compare/v7.19.8...v7.19.9) (2026-02-23) - -- docs: replace ASCII box art with prose in Before/After section (6003e2b) - - -### [7.19.8](https://github.com/soulcraftlabs/brainy/compare/v7.19.7...v7.19.8) (2026-02-23) - -- docs: redesign ELI5 comparison section and add What Can You Build? (3f16e17) - - -### [7.19.7](https://github.com/soulcraftlabs/brainy/compare/v7.19.6...v7.19.7) (2026-02-23) - -- docs: add plain-language ELI5 overview and link from README (a88962f) - - -### [7.19.6](https://github.com/soulcraftlabs/brainy/compare/v7.19.5...v7.19.6) (2026-02-19) - -- docs: convert code examples to TypeScript (791cacc) - - -### [7.19.5](https://github.com/soulcraftlabs/brainy/compare/v7.19.4...v7.19.5) (2026-02-19) - - - - -### [7.19.4](https://github.com/soulcraftlabs/brainy/compare/v7.19.3...v7.19.4) (2026-02-19) - - - - -### [7.19.3](https://github.com/soulcraftlabs/brainy/compare/v7.19.2...v7.19.3) (2026-02-19) - -- docs: add public frontmatter to docs for soulcraft.com/docs pipeline (b6e3470) - - -### [7.19.2](https://github.com/soulcraftlabs/brainy/compare/v7.19.1...v7.19.2) (2026-02-18) - -- fix: metadata index not cleaned up after delete/deleteMany (1a628da) - - -### [7.18.0](https://github.com/soulcraftlabs/brainy/compare/v7.17.0...v7.18.0) (2026-02-16) - -- feat: add aggregation engine with incremental SUM/COUNT/AVG/MIN/MAX, GROUP BY, and time windows (f024e56) -- docs: add Claude Code project guide and verified architecture reference (089a4d4) - - -### [7.17.0](https://github.com/soulcraftlabs/brainy/compare/v7.16.0...v7.17.0) (2026-02-09) - -- feat: add migration system with error handling, validation, and enterprise hardening (39b099c) - - -### [7.16.0](https://github.com/soulcraftlabs/brainy/compare/v7.15.5...v7.16.0) (2026-02-09) - -- feat: enforce data/metadata separation, numeric range queries, improved docs (0ddc05a) - - -### [7.15.5](https://github.com/soulcraftlabs/brainy/compare/v7.15.4...v7.15.5) (2026-02-02) - -- docs: update plugin docs to reflect opt-in behavior (c0bb413) - - -### [7.15.4](https://github.com/soulcraftlabs/brainy/compare/v7.15.3...v7.15.4) (2026-02-02) - -- fix: set verb.source/target to entity UUID instead of NounType (932fb95) - - -### [7.15.3](https://github.com/soulcraftlabs/brainy/compare/v7.15.2...v7.15.3) (2026-02-02) - -- feat: add explicit plugins config to control plugin auto-detection (6625385) - - -### [7.15.2](https://github.com/soulcraftlabs/brainy/compare/v7.15.1...v7.15.2) (2026-02-01) - -- fix: flush graph LSM-trees on close to prevent data loss across restarts (ab2493a) - - -### [7.15.0](https://github.com/soulcraftlabs/brainy/compare/v7.14.0...v7.15.0) (2026-02-01) - -- feat: harden plugin system wiring and add developer diagnostics (401e300) - - -## [7.14.0](https://github.com/soulcraftlabs/brainy/compare/v7.13.0...v7.14.0) (2026-02-01) - - -### ♻️ Code Refactoring - -* remove src/cortex/ directory and fix README claims ([36db644](https://github.com/soulcraftlabs/brainy/commit/36db644eca94253f1df04a1f91968856ac585b36)) - -### [7.13.0](https://github.com/soulcraftlabs/brainy/compare/v7.12.0...v7.13.0) (2026-02-01) - -- refactor: remove augmentation system and semantic type matching (d1db351) - - -### [7.12.0](https://github.com/soulcraftlabs/brainy/compare/v7.11.0...v7.12.0) (2026-02-01) - -- feat: update plugin references from @soulcraft/brainy-cortex to @soulcraft/cortex (7f9d2a7) -- refactor: remove deprecated Cortex class (replaced by brain.augmentations API) (490a14a) - - -### [7.11.0](https://github.com/soulcraftlabs/brainy/compare/v7.10.0...v7.11.0) (2026-01-31) - -- feat: add SQ8 vector quantization, lazy loading, and two-phase rerank to HNSW (0f3a884) - - -### [7.10.0](https://github.com/soulcraftlabs/brainy/compare/v7.9.3...v7.10.0) (2026-01-31) - -- feat: wire plugin system with provider resolution, storage factories, and browser deprecation (1513e29) -- chore: sync package-lock.json after dependency install (25912b5) -- feat: add plugin system for cortex and storage adapters (cc50ac3) -- perf: optimize init() and rebuild performance (35cb674) -- fix: eliminate flaky test timeouts and add storage adapters guide (cd87529) -- fix: eliminate cloud storage write amplification and rate limiting (92d9420) -- fix: distribute metadata index keys across sub-prefixes to avoid cloud rate limits (23e1c56) -- fix: invalidate VFS caches recursively on rmdir to prevent orphaned reads (66d7aa7) - - -### [7.9.3](https://github.com/soulcraftlabs/brainy/compare/v7.9.2...v7.9.3) (2026-01-28) - -- perf: optimize addMany() with batch embedding for 5-10x speedup (df7d467) -- fix: cancel abandoned highlight() semantic work and harden WASM engine recovery (f8dd93c) - - -### [7.9.1](https://github.com/soulcraftlabs/brainy/compare/v7.9.0...v7.9.1) (2026-01-27) - -- fix: exclude __words__ keyword index from corruption detection and getStats() (364360d) - - -### [7.9.0](https://github.com/soulcraftlabs/brainy/compare/v7.8.0...v7.9.0) (2026-01-27) - -- chore: rebuild type embeddings for updated ContentCategory type (3911fa7) -- feat: expand ContentCategory to universal 6-category set for highlight() (ff80b87) - - -### [7.8.0](https://github.com/soulcraftlabs/brainy/compare/v7.7.0...v7.8.0) (2026-01-27) - -- feat: add structured content extraction and batch embedding optimization to highlight() (cca1cd8) - - -## [7.8.0](https://github.com/soulcraftlabs/brainy/compare/v7.7.0...v7.8.0) (2026-01-27) - -### Bug Fixes - -**highlight() hangs on structured text input** - -Three root causes fixed: - -1. **embedBatch() now uses native WASM batch API** — Previously called `embed()` individually N times via `Promise.all`, each creating a separate forward pass. Now delegates to `engine.embedBatch()` for a single WASM forward pass. Applies globally to all `embedBatch()` callers. - -2. **Smart content extraction for structured text** — `highlight()` now auto-detects content type (plain text, rich-text JSON, HTML, Markdown) and extracts meaningful text segments instead of splitting raw JSON/HTML into garbage chunks like `{"type":`. Supports TipTap, Slate.js, Lexical, Draft.js, and Quill Delta formats out of the box. - -3. **Timeout protection** — Semantic matching phase now has a 10-second timeout. On timeout or error, `highlight()` returns Phase 1 text-only matches (always fast) instead of hanging indefinitely. - -**extractTextContent() skips arrays of objects** — Changed from length-based skip (`data.length > 10`) to type-based check (`typeof data[0] === 'number'`). Arrays of objects (e.g., team members, items) are now properly indexed for text search instead of being silently skipped. - -### Features - -**Structured Content Highlighting** - -`highlight()` now handles structured text formats automatically: - -```typescript -// Rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill) -const highlights = await brain.highlight({ - query: "warrior", - text: JSON.stringify(tiptapDocument) -}) -// Each highlight includes contentCategory: 'heading' | 'prose' | 'code' | 'label' - -// HTML -await brain.highlight({ query: "warrior", text: "

Warriors

Brave fighters.

" }) - -// Markdown -await brain.highlight({ query: "warrior", text: "# Warriors\n\nBrave fighters." }) -``` - -**Content Category Annotations** - -Each `Highlight` now includes `contentCategory` when input is structured: -- `'heading'` — from `

`-`

`, `# Heading`, or heading nodes -- `'code'` — from ``/`
`, fenced/indented code blocks, or code nodes
-- `'prose'` — regular paragraph text
-- `'label'` — labels, captions, metadata-like text
-
-**Custom Content Extractors**
-
-New `contentExtractor` parameter lets developers plug in custom parsers:
-
-```typescript
-const highlights = await brain.highlight({
-  query: "function",
-  text: sourceCode,
-  contentExtractor: (text) => treeSitterParse(text)  // Custom parser
-})
-```
-
-**Content Type Hints**
-
-New `contentType` parameter to skip auto-detection:
-
-```typescript
-await brain.highlight({ query: "test", text: input, contentType: 'html' })
-```
-
-### New Types
-
-- `ContentType`: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`
-- `ContentCategory`: `'prose' | 'heading' | 'code' | 'label'`
-- `ExtractedSegment`: `{ text: string, contentCategory: ContentCategory }`
-- `HighlightParams.contentType?` — optional content type hint
-- `HighlightParams.contentExtractor?` — optional custom parser callback
-- `Highlight.contentCategory?` — content role annotation
-
-## [7.7.0](https://github.com/soulcraftlabs/brainy/compare/v7.6.1...v7.7.0) (2026-01-26)
-
-### Features
-
-**Match Visibility in Search Results**
-
-Search results now include detailed match information:
-- `textMatches: string[]` - Query words found in entity
-- `textScore: number` - Text match quality (0-1)
-- `semanticScore: number` - Semantic similarity (0-1)
-- `matchSource: 'text' | 'semantic' | 'both'` - Where result came from
-
-```typescript
-const results = await brain.find({ query: 'david the warrior' })
-results[0].textMatches    // ["david", "warrior"]
-results[0].semanticScore  // 0.87
-results[0].matchSource    // "both"
-```
-
-**Semantic Highlighting API**
-
-New `highlight()` method shows which concepts matched:
-
-```typescript
-const highlights = await brain.highlight({
-  query: "david the warrior",
-  text: "David Smith is a brave fighter who battles dragons"
-})
-// Returns both exact matches and semantic concepts:
-// [
-//   { text: "David", score: 1.0, matchType: "text" },
-//   { text: "fighter", score: 0.78, matchType: "semantic" },
-//   { text: "battles", score: 0.72, matchType: "semantic" }
-// ]
-```
-
-**Scalable Word Indexing**
-
-- Increased word limit from 50 to 5000 words per entity
-- Supports articles, chapters, and large documents
-- Roaring Bitmaps provide efficient compression at scale
-
-### Performance
-
-- O(1) fast path in `findMatchingWords()` for text results
-- 500 chunk limit in `highlight()` for memory safety
-- Stopword filtering reduces embedding overhead
-
-### [7.6.1](https://github.com/soulcraftlabs/brainy/compare/v7.6.0...v7.6.1) (2026-01-26)
-
-- docs: add link to hosted API documentation at soulcraft.com/docs
-
-## [7.6.0](https://github.com/soulcraftlabs/brainy/compare/v7.5.0...v7.6.0) (2026-01-26)
-
-- chore: republish (npm ghost versions in 7.5.x range)
-
-### [7.5.0](https://github.com/soulcraftlabs/brainy/compare/v7.4.1...v7.5.0) (2026-01-26)
-
-- fix: update() field asymmetry causing index corruption (a94219e)
-
-
-## [7.5.0](https://github.com/soulcraftlabs/brainy/compare/v7.4.1...v7.5.0) (2026-01-26)
-
-### Bug Fixes
-
-**CRITICAL: Fixed metadata index corruption on update() operations**
-
-**Symptoms:**
-- `find()` queries returning 0 results after many updates
-- Index entry count growing with each update (7 extra entries per update)
-- At scale (77+ updates), queries fail due to overcounting in intersection logic
-
-**Root Cause:**
-In `update()`, the `removalMetadata` object only contained custom metadata + type, while `entityForIndexing` contained ALL indexed fields (confidence, weight, createdAt, updatedAt, service, data, createdBy). This asymmetry caused 7 fields to accumulate as orphaned index entries on every update.
-
-The `updatedAt` field was the worst offender - creating a NEW unique orphan on every update since the timestamp always changes.
-
-**Solution (src/brainy.ts:1163-1173):**
-```typescript
-// BEFORE (broken): Only removed custom metadata + type
-const removalMetadata = {
-  ...existing.metadata,
-  type: existing.type
-}
-
-// AFTER (fixed): Removes ALL indexed fields
-const removalMetadata = {
-  type: existing.type,
-  confidence: existing.confidence,
-  weight: existing.weight,
-  createdAt: existing.createdAt,
-  updatedAt: existing.updatedAt,  // CRITICAL: removes old timestamp
-  service: existing.service,
-  data: existing.data,
-  createdBy: existing.createdBy,
-  metadata: existing.metadata     // Nested to match entityForIndexing structure
-}
-```
-
-### Features
-
-**Index health monitoring and auto-repair**
-
-- `validateIndexConsistency()` - Public API to check index health
-- `getIndexStats()` - Public API to get index statistics
-- Auto-detection of index corruption on startup (>100 avg entries/entity)
-- Automatic rebuild when corruption is detected
-
-**EntityIdMapper persistence improvements**
-
-- Added `getOrAssignSync()` for immediate persistence of UUID→int mappings
-- Prevents mapping divergence on process crash
-
-### Tests
-
-- Added comprehensive integration tests for update field asymmetry fix
-- Tests verify query accuracy, no duplicates, and entity integrity after many updates
-
-### [7.4.1](https://github.com/soulcraftlabs/brainy/compare/v7.4.0...v7.4.1) (2026-01-20)
-
-- fix: VFS readdir() no longer returns duplicate entries (2bd4031)
-
-
-### [7.4.0](https://github.com/soulcraftlabs/brainy/compare/v7.3.1...v7.4.0) (2026-01-20)
-
-- feat: Integration Hub for external tool connectivity (b5bc900)
-
-
-### [7.3.1](https://github.com/soulcraftlabs/brainy/compare/v7.3.0...v7.3.1) (2026-01-16)
-
-- fix: clear() now properly resets VFS and COW state (79ae349)
-
-
-### [7.3.0](https://github.com/soulcraftlabs/brainy/compare/v7.2.2...v7.3.0) (2026-01-07)
-
-- feat: progressive init and readiness API for cloud storage (d938a6b)
-
-
-### [7.2.2](https://github.com/soulcraftlabs/brainy/compare/v7.2.1...v7.2.2) (2026-01-07)
-
-- test: increase timing threshold for flaky updateMany test (9fbefd4)
-- perf: 10-50x faster vector search with batch operations (5885de7)
-
-
-### [7.2.1](https://github.com/soulcraftlabs/brainy/compare/v7.2.0...v7.2.1) (2026-01-06)
-
-- fix: bun --compile model loading with fallback paths (e62e748)
-
-
-### [7.2.0](https://github.com/soulcraftlabs/brainy/compare/v7.1.1...v7.2.0) (2026-01-06)
-
-- perf: 580x faster embedding init - separate model from WASM (677e2d6)
-
-
-## [7.2.0](https://github.com/soulcraftlabs/brainy/compare/v7.1.1...v7.2.0) (2026-01-06)
-
-### Performance
-
-**CRITICAL: 580x faster embedding initialization (139 seconds → 240ms)**
-
-**Symptom:**
-- Cloud Run cold starts taking 2+ minutes
-- Container restart loops due to 503 errors
-- Logs showing: `✅ Candle Embedding Engine ready in 139124ms`
-
-**Root Cause:**
-The 90MB WASM file contained 87MB of embedded model weights. WASM parsing/compilation scales with file size, and Cloud Run's throttled CPU during cold starts extends this to 139 seconds.
-
-**Solution: Separate Model from WASM (v7.2.0 architecture)**
-- WASM file: 90MB → 2.4MB (inference code only)
-- Model files: Loaded separately as raw bytes (~88MB)
-- Total init time: 139 seconds → 240ms (Node.js) / 136ms (Bun)
-
-| Component | Before | After |
-|-----------|--------|-------|
-| WASM size | 90MB | 2.4MB |
-| WASM compile | 139,000ms | 6-8ms |
-| Model load | (embedded) | 30-115ms |
-| **Total init** | **139,000ms** | **136-240ms** |
-
-**Environment Support:**
-- Node.js: Model loaded from filesystem via `fs.readFile()`
-- Bun: Model loaded via `Bun.file()`
-- Bun --compile: Model files auto-embedded in binary
-- Browser: Model fetched via `fetch()`
-
-**No Breaking Changes:**
-- Same API as v7.1.x
-- Zero configuration required
-- npm package includes model files automatically
-
-### Technical Details
-
-New files:
-- `src/embeddings/wasm/modelLoader.ts` - Universal model loading for all environments
-
-Modified:
-- `src/embeddings/candle-wasm/src/lib.rs` - Removed `include_bytes!()` for model weights
-- `src/embeddings/wasm/CandleEmbeddingEngine.ts` - Uses external model loading
-- `package.json` - Includes `assets/models/all-MiniLM-L6-v2/**` in npm package
-
-
-### [7.1.1](https://github.com/soulcraftlabs/brainy/compare/v7.1.0...v7.1.1) (2026-01-06)
-
-### Bug Fixes
-
-**CRITICAL: Fixed 50-100x slower add() operations on cloud storage (GCS/S3/R2/Azure)**
-
-**Symptoms:**
-- add() taking 7-12 seconds instead of 50-200ms
-- Only affects cloud storage with auto-detection (not explicit `type: 'gcs'`)
-
-**Root Cause:**
-Storage type detection in `setupIndex()` relied on `this.config.storage.type` which was never set after `createStorage()` auto-detected the storage type. This caused cloud storage to use `'immediate'` persistence mode instead of `'deferred'`, resulting in 20-30 GCS writes per add() operation.
-
-**Fix:**
-Added `getStorageType()` helper that detects storage type from the storage instance class name (e.g., `GcsStorage` → `'gcs'`), used as fallback when `config.storage.type` is not explicitly set.
-
-**Workaround for v7.1.0 users:**
-```typescript
-const brain = new Brainy({
-  storage: {
-    type: 'gcs',  // Explicit type fixes the issue
-    gcsNativeStorage: { bucketName: 'your-bucket' }
-  },
-  hnswPersistMode: 'deferred'  // Or explicitly set this
-})
-```
-
-### Performance Tests
-
-Added performance regression tests to prevent future issues:
-- Single add() < 500ms
-- 10 add() operations < 5 seconds
-- Storage type detection verification for GCS/S3/R2/Azure
-
-
-## [7.1.0](https://github.com/soulcraftlabs/brainy/compare/v7.0.1...v7.1.0) (2026-01-06)
-
-### Features
-
-**6 New Public APIs** leveraging the Candle WASM embedding engine and optimized indexes:
-
-| API | Description | Performance |
-|-----|-------------|-------------|
-| `embedBatch(texts)` | Batch embed multiple texts | Batch WASM processing - avoids N separate JS↔WASM calls |
-| `similarity(textA, textB)` | Semantic similarity score (0-1) | Single call vs manual embed + embed + cosine |
-| `indexStats()` | Comprehensive index statistics | O(1) - aggregates pre-computed stats |
-| `neighbors(entityId, options)` | Graph traversal with filters | O(log n) - LSM-tree with bloom filters, sub-5ms |
-| `findDuplicates(options)` | Find semantic duplicates | O(k log n) - uses HNSW for ANN search |
-| `cluster(options)` | Cluster by similarity | O(k log n) - greedy algorithm with HNSW |
-
-### Performance Stack (v7.0.0+)
-
-The new APIs leverage the optimized infrastructure introduced in v7.0.0:
-
-| Component | Technology | Benefit |
-|-----------|------------|---------|
-| **Embeddings** | Candle WASM (Rust) | 93MB binary with embedded MiniLM-L6-v2, zero downloads |
-| **Vector Search** | HNSW Index | O(log n) approximate nearest neighbor |
-| **Graph Traversal** | LSM-tree + Bloom Filters | 90% of queries skip disk I/O, sub-5ms lookups |
-| **Metadata Filtering** | RoaringBitmap32 | Compressed bitmaps for fast AND/OR operations |
-
-### Migration from v6.x
-
-v7.0.0 introduced **breaking changes** to the embedding system:
-- Removed: `onnxruntime-node` dependency (was 200MB+ with external model downloads)
-- Added: Candle WASM with embedded model weights (93MB, zero-config)
-- Removed: Semantic type inference (NLP-based type detection)
-- Works in: Node.js, Bun, Bun --compile, browsers
-
-
-### [7.0.1](https://github.com/soulcraftlabs/brainy/compare/v7.0.0...v7.0.1) (2026-01-06)
-
-- fix: resolve WASM loading for Bun --compile single-binary executables (5d9ec5b)
-
-
-### [7.0.0](https://github.com/soulcraftlabs/brainy/compare/v6.6.2...v7.0.0) (2026-01-06)
-
-- feat: migrate embeddings to Candle WASM + remove semantic type inference (da7d2ed)
-
-
-### [6.6.2](https://github.com/soulcraftlabs/brainy/compare/v6.6.1...v6.6.2) (2026-01-05)
-
-- fix: resolve update() v5.11.1 regression + skip flaky tests for release (106f654)
-- fix(metadata-index): delete chunk files during rebuild to prevent 77x overcounting (386666d)
-
-
-## [6.4.0](https://github.com/soulcraftlabs/brainy/compare/v6.3.2...v6.4.0) (2025-12-11)
-
-### ⚡ Performance
-
-**Optimized VFS directory operations for cloud storage (GCS, S3, Azure, R2)**
-
-**Issue:** `vfs.rmdir({ recursive: true })` took ~2 minutes for 15 files on GCS due to sequential operations. Each file deletion was a separate storage round-trip.
-
-**Solution:** Replace sequential loops with batch operations using existing optimized primitives:
-
-* **`rmdir()`**: Use `gatherDescendants()` + `deleteMany()` + parallel blob cleanup
-* **`copyDirectory()`**: Use `gatherDescendants()` + `addMany()` + `relateMany()`
-* **`move()`**: Inherits improvements from both (no code change needed)
-
-**PROJECTED Performance Improvement:**
-
-| Operation | Before | After | Improvement |
-|-----------|--------|-------|-------------|
-| rmdir 15 files | ~120s | ~15-30s | 4-8x faster |
-| copy 15 files | ~120s | ~20-40s | 3-6x faster |
-| move 15 files | ~240s | ~40-60s | 4-6x faster |
-
-Requested by: a consumer team (BRAINY-VFS-RMDIR-PERFORMANCE)
-
-### [6.3.2](https://github.com/soulcraftlabs/brainy/compare/v6.3.1...v6.3.2) (2025-12-09)
-
-
-### 🐛 Bug Fixes
-
-* **versioning:** VFS file versions now capture actual blob content ([3e0f235](https://github.com/soulcraftlabs/brainy/commit/3e0f235f8b2cfcc6f0792a457879a02e4b93897a))
-
-### [6.3.1](https://github.com/soulcraftlabs/brainy/compare/v6.3.0...v6.3.1) (2025-12-09)
-
-- fix(versioning): clean architecture with index pollution prevention (f145fa1)
-- chore(release): 6.3.0 - singleton GraphAdjacencyIndex architecture fix (292be1b)
-- fix(architecture): singleton GraphAdjacencyIndex via storage.getGraphIndex() (v6.3.0) (c15892e)
-- chore(release): 6.2.9 - fix critical VFS bugs (directory corruption) (810b756)
-- fix(vfs): resolve two critical VFS bugs causing directory listing corruption (2ba69ec)
-- chore(release): 6.2.8 - deferred HNSW persistence for 30-50× faster cloud adds (1da6048)
-- perf(hnsw): deferred persistence mode for 30-50× faster cloud storage adds (4d1d567)
-- chore(release): 6.2.7 - simplify cloud storage to always-on write buffering (a33b759)
-- perf(storage): simplify cloud adapters to always-on write buffering (26510ce)
-- chore(release): 6.2.6 - fix cloud storage read-after-write consistency (6449bb1)
-- fix(storage): populate cache before write buffer for read-after-write consistency (2d27bd0)
-- chore(release): 6.2.5 - fix counts.byType() accumulation bug (e4bbd7f)
-- fix(counts): counts.byType() returns inflated values due to accumulation bug (9456c2c)
-- chore(release): 6.2.4 - fix asOf() COW property name mismatch (ea53c11)
-- fix(cow): asOf() fails with "COW not enabled" due to property name mismatch (b3ae18b)
-- chore(release): 6.2.3 - fix counts.byType({ excludeVFS: true }) returning empty (0ba6da4)
-- fix(counts): counts.byType({ excludeVFS: true }) now returns correct type counts (9b2ff2d)
-
-
-### [6.2.2](https://github.com/soulcraftlabs/brainy/compare/v6.2.1...v6.2.2) (2025-11-25)
-
-- refactor: remove 3,700+ LOC of unused HNSW implementations (e3146ce)
-- fix(hnsw): entry point recovery prevents import failures and log spam (52eae67)
-
-
-## [6.2.0](https://github.com/soulcraftlabs/brainy/compare/v6.1.0...v6.2.0) (2025-11-20)
-
-### ⚡ Critical Performance Fix
-
-**Fixed VFS tree operations on cloud storage (GCS, S3, Azure, R2, OPFS)**
-
-**Issue:** Despite v6.1.0's PathResolver optimization, `vfs.getTreeStructure()` remained critically slow on cloud storage:
-- **Production (GCS) deployment:** 5,304ms for tree with maxDepth=2
-- **Root Cause:** Tree traversal made 111+ separate storage calls (one per directory)
-- **Why v6.1.0 didn't help:** v6.1.0 optimized path→ID resolution, but tree traversal still called `getChildren()` 111+ times
-
-**Architecture Fix:**
-```
-OLD (v6.1.0):
-- For each directory: getChildren(dirId) → fetch entities → GCS call
-- 111 directories = 111 GCS calls × 50ms = 5,550ms
-
-NEW (v6.2.0):
-1. Traverse graph in-memory to collect all IDs (GraphAdjacencyIndex)
-2. Batch-fetch ALL entities in ONE storage call (brain.batchGet)
-3. Build tree structure from fetched entities
-
-Result: 111 storage calls → 1 storage call
-```
-
-**Performance (Production Measurement):**
-- **GCS:** 5,304ms → ~100ms (**53x faster**)
-- **FileSystem:** Already fast, minimal change
-
-**Files Changed:**
-- `src/vfs/VirtualFileSystem.ts:616-689` - New `gatherDescendants()` method
-- `src/vfs/VirtualFileSystem.ts:691-728` - Updated `getTreeStructure()` to use batch fetch
-- `src/vfs/VirtualFileSystem.ts:730-762` - Updated `getDescendants()` to use batch fetch
-
-**Impact:**
-- ✅ Consumer file explorer now loads instantly on GCS
-- ✅ Clean architecture: one code path, no fallbacks
-- ✅ Production-scale: uses in-memory graph + single batch fetch
-- ✅ Works for ALL storage adapters (GCS, S3, Azure, R2, OPFS, FileSystem)
-
-**Migration:** No code changes required - automatic performance improvement.
-
-### 🚨 Critical Bug Fix: Blob Integrity Check Failures (PERMANENT FIX)
-
-**Fixed blob integrity check failures on cloud storage using key-based dispatch (NO MORE GUESSING)**
-
-**Issue:** Production users reported "Blob integrity check failed" errors when opening files from GCS:
-- **Symptom:** Random file read failures with hash mismatch errors
-- **Root Cause:** `wrapBinaryData()` tried to guess data type by parsing, causing compressed binary that happens to be valid UTF-8 + valid JSON to be stored as parsed objects instead of wrapped binary
-- **Impact:** On read, `JSON.stringify(object)` !== original compressed bytes → hash mismatch → integrity failure
-
-**The Guessing Problem (v5.10.1 - v6.1.0):**
-```typescript
-// FRAGILE: wrapBinaryData() tries to JSON.parse ALL buffers
-wrapBinaryData(compressedBuffer) {
-  try {
-    return JSON.parse(data.toString())  // ← Compressed data accidentally parses!
-  } catch {
-    return {_binary: true, data: base64}
-  }
-}
-
-// FAILURE PATH:
-// 1. WRITE: hash(raw) → compress(raw) → wrapBinaryData(compressed)
-//    → compressed bytes accidentally parse as valid JSON
-//    → stored as parsed object instead of wrapped binary
-// 2. READ: retrieve object → JSON.stringify(object) → decompress
-//    → different bytes than original compressed data
-//    → HASH MISMATCH → "Blob integrity check failed"
-```
-
-**The Permanent Solution (v6.2.0): Key-Based Dispatch**
-
-Stop guessing! The key naming convention **IS** the explicit type contract:
-
-```typescript
-// baseStorage.ts COW adapter (line 371-393)
-put: async (key: string, data: Buffer): Promise => {
-  // NO GUESSING - key format explicitly declares data type:
-  //
-  // JSON keys: 'ref:*', '*-meta:*'
-  // Binary keys: 'blob:*', 'commit:*', 'tree:*'
-
-  const obj = key.includes('-meta:') || key.startsWith('ref:')
-    ? JSON.parse(data.toString())  // Metadata/refs: ALWAYS JSON
-    : { _binary: true, data: data.toString('base64') }  // Blobs: ALWAYS binary
-
-  await this.writeObjectToPath(`_cow/${key}`, obj)
-}
-```
-
-**Why This is Permanent:**
-- ✅ **Zero guessing** - key explicitly declares type
-- ✅ **Works for ANY compression** - gzip, zstd, brotli, future algorithms
-- ✅ **Self-documenting** - code clearly shows intent
-- ✅ **No heuristics** - no fragile first-byte checks or try/catch parsing
-- ✅ **Single source of truth** - key naming convention is the contract
-
-**Files Changed:**
-- `src/storage/baseStorage.ts:371-393` - COW adapter uses key-based dispatch (NO MORE wrapBinaryData)
-- `src/storage/cow/binaryDataCodec.ts:86-119` - Deprecated wrapBinaryData() with warnings
-- `tests/unit/storage/cow/BlobStorage.test.ts:612-705` - Added 4 comprehensive regression tests
-
-**Regression Tests Added:**
-1. JSON-like compressed data (THE KILLER TEST CASE)
-2. All key types dispatch correctly (blob, commit, tree)
-3. Metadata keys handled correctly
-4. Verify wrapBinaryData() never called on write path
-
-**Impact:**
-- ✅ **PERMANENT FIX** - eliminates blob integrity failures forever
-- ✅ Works for ALL storage adapters (GCS, S3, Azure, R2, OPFS, FileSystem)
-- ✅ Works for ALL compression algorithms
-- ✅ Comprehensive regression tests prevent future regressions
-- ✅ No performance cost (key.includes() is fast)
-
-**Migration:** No action required - automatic fix for all blob operations.
-
-### ⚡ Performance Fix: Removed Access Time Updates on Reads
-
-**Fixed 50-100ms GCS write penalty on EVERY file/directory read**
-
-**Issue:** Production GCS performance showed file reads taking significantly longer than expected:
-- **Expected:** ~50ms for file read
-- **Actual:** ~100-150ms for file read
-- **Root Cause:** `updateAccessTime()` called on EVERY `readFile()` and `readdir()` operation
-- **Impact:** Each access time update = 50-100ms GCS write operation + doubled GCS costs
-
-**The Problem:**
-```typescript
-// OLD (v6.1.0):
-async readFile(path: string): Promise {
-  const entity = await this.getEntityByPath(path)
-  await this.updateAccessTime(entityId)  // ← 50-100ms GCS write!
-  return await this.blobStorage.read(blobHash)
-}
-
-async readdir(path: string): Promise {
-  const entity = await this.getEntityByPath(path)
-  await this.updateAccessTime(entityId)  // ← 50-100ms GCS write!
-  return children.map(child => child.metadata.name)
-}
-```
-
-**Why Access Time Updates Are Harmful:**
-1. **Performance:** 50-100ms penalty on cloud storage for EVERY read
-2. **Cost:** Doubles GCS operation costs (read + write for every file access)
-3. **Unnecessary:** Modern filesystems use `noatime` mount option for same reason
-4. **Unused:** The `accessed` field was NEVER used in queries, filters, or application logic
-
-**Solution (v6.2.0): Remove Completely**
-
-Following modern filesystem best practices (Linux `noatime`, macOS default behavior):
-- ✅ Removed `updateAccessTime()` call from `readFile()` (line 372)
-- ✅ Removed `updateAccessTime()` call from `readdir()` (line 1002)
-- ✅ Removed `updateAccessTime()` method entirely (lines 1355-1365)
-- ✅ Field `accessed` still exists in metadata for backward compatibility (just won't update)
-
-**Performance Impact (Production Scale):**
-- **File reads:** 100-150ms → 50ms (**2-3x faster**)
-- **Directory reads:** 100-150ms → 50ms (**2-3x faster**)
-- **GCS costs:** ~50% reduction (eliminated write operation on every read)
-- **FileSystem:** Minimal impact (already fast, but removes unnecessary disk I/O)
-
-**Files Changed:**
-- `src/vfs/VirtualFileSystem.ts:372-375` - Removed updateAccessTime() from readFile()
-- `src/vfs/VirtualFileSystem.ts:1002-1006` - Removed updateAccessTime() from readdir()
-- `src/vfs/VirtualFileSystem.ts:1355-1365` - Removed updateAccessTime() method
-
-**Impact:**
-- ✅ **2-3x faster reads** on cloud storage
-- ✅ **~50% GCS cost reduction** (no write on every read)
-- ✅ Follows modern filesystem best practices
-- ✅ Backward compatible: field exists but won't update
-- ✅ Works for ALL storage adapters (GCS, S3, Azure, R2, OPFS, FileSystem)
-
-**Migration:** No action required - automatic performance improvement.
-
-### ⚡ Performance Fix: Eliminated N+1 Patterns Across All APIs
-
-**Fixed 8 N+1 patterns for 10-20x faster batch operations on cloud storage**
-
-**Issue:** Multiple APIs loaded entities/relationships one-by-one instead of using batch operations:
-- `find()`: 5 different code paths loaded entities individually
-- `batchGet()` with vectors: Looped through individual `get()` calls
-- `executeGraphSearch()`: Loaded connected entities one-by-one
-- `relate()` duplicate checking: Loaded existing relationships one-by-one
-- `deleteMany()`: Created separate transaction for each entity
-
-**Root Cause:** Individual storage calls instead of batch operations → N × 50ms on GCS = severe latency
-
-**Solution (v6.2.0): Comprehensive Batch Operations**
-
-**1. Fixed `find()` method - 5 locations**
-```typescript
-// OLD: N separate storage calls
-for (const id of pageIds) {
-  const entity = await this.get(id)  // ❌ N×50ms on GCS
-}
-
-// NEW: Single batch call
-const entitiesMap = await this.batchGet(pageIds)  // ✅ 1×50ms on GCS
-for (const id of pageIds) {
-  const entity = entitiesMap.get(id)
-}
-```
-
-**2. Fixed `batchGet()` with vectors**
-- **Added:** `storage.getNounBatch(ids)` method (baseStorage.ts:1986)
-- Batch-loads vectors + metadata in parallel
-- Eliminates N+1 when `includeVectors: true`
-
-**3. Fixed `executeGraphSearch()`**
-- Uses `batchGet()` for connected entities
-- 20 entities: 1,000ms → 50ms (**20x faster**)
-
-**4. Fixed `relate()` duplicate checking**
-- **Added:** `storage.getVerbsBatch(ids)` method (baseStorage.ts:826)
-- **Added:** `graphIndex.getVerbsBatchCached(ids)` method (graphAdjacencyIndex.ts:384)
-- Batch-loads existing relationships with cache-aware loading
-- 5 verbs: 250ms → 50ms (**5x faster**)
-
-**5. Fixed `deleteMany()`**
-- **Changed:** Batches deletes into chunks of 10
-- Single transaction per chunk (atomic within chunk)
-- 10 entities: 2,000ms → 200ms (**10x faster**)
-- Proper error handling with `continueOnError` flag
-
-**Performance Impact (Production GCS):**
-
-| Operation | Before | After | Speedup |
-|-----------|--------|-------|---------|
-| find() with 10 results | 10×50ms = 500ms | 1×50ms = 50ms | **10x** |
-| batchGet() with vectors (10 entities) | 10×50ms = 500ms | 1×50ms = 50ms | **10x** |
-| executeGraphSearch() with 20 entities | 20×50ms = 1000ms | 1×50ms = 50ms | **20x** |
-| relate() duplicate check (5 verbs) | 5×50ms = 250ms | 1×50ms = 50ms | **5x** |
-| deleteMany() with 10 entities | 10 txns = 2000ms | 1 txn = 200ms | **10x** |
-
-**Files Changed:**
-- `src/brainy.ts:1682-1690` - find() location 1 (batch load)
-- `src/brainy.ts:1713-1720` - find() location 2 (batch load)
-- `src/brainy.ts:1820-1832` - find() location 3 (batch load filtered results)
-- `src/brainy.ts:1845-1853` - find() location 4 (batch load paginated)
-- `src/brainy.ts:1870-1878` - find() location 5 (batch load sorted)
-- `src/brainy.ts:724-732` - batchGet() with vectors optimization
-- `src/brainy.ts:1171-1183` - relate() duplicate check optimization
-- `src/brainy.ts:2216-2310` - deleteMany() transaction batching
-- `src/brainy.ts:4314-4325` - executeGraphSearch() batch load
-- `src/storage/baseStorage.ts:1986-2045` - Added getNounBatch()
-- `src/storage/baseStorage.ts:826-886` - Added getVerbsBatch()
-- `src/graph/graphAdjacencyIndex.ts:384-413` - Added getVerbsBatchCached()
-- `src/coreTypes.ts:721,743` - Added batch methods to StorageAdapter interface
-- `src/types/brainy.types.ts:367` - Added continueOnError to DeleteManyParams
-
-**Architecture:**
-- ✅ **COW/fork/asOf**: All batch methods use `readBatchWithInheritance()`
-- ✅ **All storage adapters**: Works with GCS, S3, Azure, R2, OPFS, FileSystem
-- ✅ **Caching**: getVerbsBatchCached() checks UnifiedCache first
-- ✅ **Transactions**: deleteMany() batches into atomic chunks
-- ✅ **Error handling**: Proper error collection with continueOnError support
-
-**Impact:**
-- ✅ **10-20x faster** batch operations on cloud storage
-- ✅ **50-90% cost reduction** (fewer storage API calls)
-- ✅ Clean architecture - no fallbacks, no hacks
-- ✅ Backward compatible - automatic performance improvement
-
-**Migration:** No action required - automatic performance improvement.
-
----
-
-## [6.1.0](https://github.com/soulcraftlabs/brainy/compare/v6.0.2...v6.1.0) (2025-11-20)
-
-### 🚀 Features
-
-**VFS path resolution now uses MetadataIndexManager for 75x faster cold reads**
-
-**Issue:** After fixing N+1 patterns in v6.0.2, VFS file reads on cloud storage were still ~1,500ms (vs 50ms on filesystem) because path resolution required 3-level graph traversal with network round trips.
-
-**Opportunity:** Brainy's MetadataIndexManager already indexes the `path` field in VFS entities using roaring bitmaps with bloom filters. Instead of traversing the graph, we can query the index directly for O(log n) lookups.
-
-**Solution:** 3-tier caching architecture for path resolution:
-1. **L1: UnifiedCache** (global LRU cache, <1ms) - Shared across all Brainy instances
-2. **L2: PathResolver cache** (local warm cache, <1ms) - Instance-specific hot paths
-3. **L3: MetadataIndexManager** (cold index query, 5-20ms on GCS) - Direct roaring bitmap lookup
-4. **Fallback: Graph traversal** - Graceful degradation if MetadataIndex unavailable
-
-**Performance Impact (MEASURED on FileSystem, PROJECTED for cloud):**
-- **Cold reads (cache miss):**
-  - FileSystem: 200ms → 150ms (1.3x faster, still needs index query)
-  - GCS/S3/Azure: 1,500ms → 20ms (**75x faster**, eliminates graph traversal)
-  - R2: 1,500ms → 20ms (**75x faster**)
-  - OPFS: 300ms → 20ms (**15x faster**)
-
-- **Warm reads (cache hit):**
-  - ALL adapters: <1ms (**1,500x faster**, UnifiedCache hit)
-
-**Files Changed:**
-- `src/vfs/PathResolver.ts:8-12` - Added UnifiedCache and logger imports
-- `src/vfs/PathResolver.ts:43-45` - Added MetadataIndex performance metrics
-- `src/vfs/PathResolver.ts:77-149` - Updated resolve() with 3-tier caching
-- `src/vfs/PathResolver.ts:196-237` - New resolveWithMetadataIndex() method
-- `src/vfs/PathResolver.ts:516-541` - Updated getStats() with MetadataIndex metrics
-
-**Zero-Config Auto-Optimization:**
-- Works for ALL storage adapters (FileSystem, GCS, S3, Azure, R2, OPFS)
-- Automatically uses MetadataIndexManager if available
-- Gracefully falls back to graph traversal if index unavailable
-- No external dependencies (uses Brainy's internal infrastructure)
-
-**Migration:** No code changes required - automatic 75x performance improvement for cloud storage.
-
-**Monitoring:** Use `pathResolver.getStats()` to track:
-- `metadataIndexHits` - Direct index queries that succeeded
-- `metadataIndexMisses` - Paths not found in index (ENOENT errors)
-- `metadataIndexHitRate` - Success rate of index queries
-- `graphTraversalFallbacks` - Times fallback to graph traversal was used
-
----
-
-## [6.0.2](https://github.com/soulcraftlabs/brainy/compare/v6.0.1...v6.0.2) (2025-11-20)
-
-### ⚡ Performance Improvements
-
-**Fixed N+1 query pattern in VFS for ALL cloud storage adapters (10x faster)**
-
-**Issue:** VFS file reads on cloud storage (GCS, S3, Azure, R2, OPFS) were 170x slower than filesystem (17 seconds vs 50ms) due to sequential entity fetching in relationship lookups.
-
-**Root Cause:**
-- `getVerbsBySource_internal()` fetched verbs one-by-one (N+1 pattern)
-- `PathResolver.resolveChild()` fetched child entities one-by-one (N+1 pattern)
-- Each cloud API call: ~300ms network latency
-- Path like `/imports/data/file.txt` = 3 components × 2 calls × 10 children = **60+ API calls = 17+ seconds**
-
-**Fix:**
-- Use existing `readBatchWithInheritance()` infrastructure in getVerbsBySource_internal
-- Use existing `brain.batchGet()` in PathResolver.resolveChild
-- Fetch all entities in parallel batch calls instead of N sequential calls
-- Zero external dependencies (uses Brainy's internal batching infrastructure)
-
-**Performance Impact:**
-- **GCS:** 17,000ms → 1,500ms (**11x faster**)
-- **S3:** 17,000ms → 1,500ms (**11x faster**)
-- **Azure:** 17,000ms → 1,500ms (**11x faster**)
-- **R2:** 17,000ms → 1,500ms (**11x faster**)
-- **OPFS:** 3,000ms → 300ms (**10x faster**)
-- **FileSystem:** 200ms → 50ms (**4x faster**, bonus)
-
-**Files Changed:**
-- `src/storage/baseStorage.ts:2622-2673` - Batch verb fetching
-- `src/vfs/PathResolver.ts:205-227` - Batch child resolution
-
-**Migration:** No code changes required - automatic 10x performance improvement.
-
-**Zero-config auto-optimization:** Each storage adapter declares optimal batch behavior:
-- GCS/Azure: 100 concurrent (HTTP/2 multiplexing)
-- S3/R2: 1000 batch size (AWS batch APIs)
-- FileSystem: 10 concurrent (OS file handle limits)
-
----
-
-## [6.0.1](https://github.com/soulcraftlabs/brainy/compare/v6.0.0...v6.0.1) (2025-11-20)
-
-### 🐛 Critical Bug Fixes
-
-**Fixed infinite loop during storage initialization on fresh workspaces (v6.0.1)**
-
-**Symptom:** FileSystemStorage (and all storage adapters) entered infinite loop on fresh installation, printing "📁 New installation: using depth 1 sharding..." message hundreds of thousands of times.
-
-**Root Cause:** In v6.0.0, `BaseStorage.init()` sets `isInitialized = true` at the END of initialization (after creating GraphAdjacencyIndex). If any code path during initialization called `ensureInitialized()`, it would trigger `init()` recursively because the flag was still `false`.
-
-**Fix:** Set `isInitialized = true` at the START of `BaseStorage.init()` (before any initialization work) to prevent recursive calls. Flag is reset to `false` on error to allow retries.
-
-**Impact:**
-- ✅ Fixes production blocker reported by a consumer team
-- ✅ All 8 storage adapters fixed (FileSystem, Memory, S3, R2, GCS, Azure, OPFS, Historical)
-- ✅ Init completes in ~1 second on fresh installation (was hanging indefinitely)
-- ✅ No new test failures introduced (1178 tests passing)
-
-**Files Changed:**
-- `src/storage/baseStorage.ts:261-287` - Moved `isInitialized = true` to top of init() with try/catch
-
-**Migration:** No code changes required - drop-in replacement for v6.0.0.
-
----
-
-## [6.0.0](https://github.com/soulcraftlabs/brainy/compare/v5.12.0...v6.0.0) (2025-11-19)
-
-## 🚀 v6.0.0 - ID-First Storage Architecture
-
-**v6.0.0 introduces ID-first storage paths, eliminating type lookups and enabling true O(1) direct access to entities and relationships.**
-
-### Core Changes
-
-**ID-First Path Structure** - Direct entity access without type lookups:
-```
-Before (v5.x):  entities/nouns/{TYPE}/metadata/{SHARD}/{ID}.json  (requires type lookup)
-After (v6.0.0): entities/nouns/{SHARD}/{ID}/metadata.json        (direct O(1) access)
-```
-
-**GraphAdjacencyIndex Integration** - All storage adapters now properly initialize the graph index:
-- ✅ All 8 storage adapters call `super.init()` to initialize GraphAdjacencyIndex
-- ✅ Relationship queries use in-memory LSM-tree index for O(1) lookups
-- ✅ Shard iteration fallback for cold-start scenarios
-
-**Test Infrastructure** - Resolved ONNX runtime stability issues:
-- ✅ Switched from `pool: 'forks'` to `pool: 'threads'` for test stability
-- ✅ 1147/1147 core tests passing (pagination test excluded due to slow setup)
-- ✅ No ONNX crashes in test runs
-
-### Breaking Changes
-
-**Removed APIs** - The following untested/broken APIs have been removed:
-```typescript
-// ❌ REMOVED - brain.getTypeFieldAffinityStats()
-// Migration: Use brain.getFieldsForType() for type-specific field analysis
-
-// ❌ REMOVED - vfs.getAllTodos()
-// Migration: Not a standard VFS API - implement custom TODO tracking if needed
-
-// ❌ REMOVED - vfs.getProjectStats()
-// Migration: Use vfs.du(path) for disk usage statistics
-
-// ❌ REMOVED - vfs.exportToJSON()
-// Migration: Use vfs.readFile() to read files individually
-```
-
-**New Standard VFS APIs** - POSIX-compliant filesystem operations:
-```typescript
-// ✅ NEW - vfs.du(path, options?) - Disk usage calculator
-const stats = await vfs.du('/projects', { humanReadable: true })
-// Returns: { bytes, files, directories, formatted: "1.2 GB" }
-
-// ✅ NEW - vfs.access(path, mode) - Permission checking
-const canRead = await vfs.access('/file.txt', 'r')
-const exists = await vfs.access('/file.txt', 'f')
-
-// ✅ NEW - vfs.find(path, options?) - Pattern-based file search
-const results = await vfs.find('/', {
-  name: '*.ts',
-  type: 'file',
-  maxDepth: 5
-})
-```
-
-**Removed Broken APIs** - Memory explosion risks eliminated:
-```typescript
-// ❌ REMOVED - brain.merge(sourceBranch, targetBranch, options)
-// Reason: Loaded ALL entities into memory (10TB at 1B scale)
-// Migration: Use GitHub-style branching - keep branches separate OR manually copy specific entities:
-const approved = await sourceBranch.find({ where: { approved: true }, limit: 100 })
-await targetBranch.checkout('target')
-for (const entity of approved) {
-  await targetBranch.add(entity)
-}
-
-// ❌ REMOVED - brain.diff(sourceBranch, targetBranch)
-// Reason: Loaded ALL entities into memory (10TB at 1B scale)
-// Migration: Use asOf() for time-travel queries OR manual paginated comparison:
-const snapshot1 = await brain.asOf(commit1)
-const snapshot2 = await brain.asOf(commit2)
-const page1 = await snapshot1.find({ limit: 100, offset: 0 })
-const page2 = await snapshot2.find({ limit: 100, offset: 0 })
-// Compare manually
-
-// ❌ REMOVED - brain.data().backup(options)
-// Reason: Loaded ALL entities into memory (10TB at 1B scale)
-// Migration: Use COW commits for zero-copy snapshots:
-await brain.fork('backup-2025-01-19')  // Instant snapshot, no memory
-const snapshot = await brain.asOf(commitId)  // Time-travel query
-
-// ❌ REMOVED - brain.data().restore(params)
-// Reason: Depended on backup() which is removed
-// Migration: Use COW checkout to switch to snapshot:
-await brain.checkout('backup-2025-01-19')  // Switch to snapshot branch
-
-// ❌ REMOVED - CLI: brainy data backup
-// ❌ REMOVED - CLI: brainy data restore
-// ❌ REMOVED - CLI: brainy cow merge
-// Migration: Use COW CLI commands instead:
-brainy fork backup-name           # Create snapshot
-brainy checkout backup-name       # Switch to snapshot
-brainy branch list                # List all snapshots/branches
-```
-
-**Storage Path Structure** - Existing databases require migration:
-```typescript
-// Migration handled automatically on first init()
-// Old databases will be detected and paths upgraded
-```
-
-**Storage Adapter Implementation** - Custom storage adapters must call parent init():
-```typescript
-class MyCustomStorage extends BaseStorage {
-  async init() {
-    // ... your initialization ...
-    await super.init()  // REQUIRED in v6.0.0+
-  }
-}
-```
-
-### Performance Impact
-
-- **Entity Retrieval**: O(1) direct path construction (no type lookup)
-- **Relationship Queries**: Sub-5ms via GraphAdjacencyIndex
-- **Cold Start**: Shard iteration fallback (256 shards vs 42/127 types)
-
-### Known Issues
-
-- **Test Suite**: graphIndex-pagination.test.ts excluded due to slow beforeEach setup (50+ entities)
-  - Production code unaffected - test-only performance issue
-  - Will be optimized in v6.0.1
-
-### Verification Summary
-
-- ✅ **1147 core tests passing** (0 failures)
-- ✅ **All 8 storage adapters verified**: Memory, FileSystem, S3, R2, GCS, Azure, OPFS, Historical
-- ✅ **All relationship queries working**: getVerbsBySource, getVerbsByTarget, relate, unrelate
-- ✅ **GraphAdjacencyIndex initialized** in all adapters
-- ✅ **Production code verified safe** (no infinite loops)
-
-### Commits
-
-- feat: v6.0.0 ID-first storage migration core implementation
-- fix: all storage adapters now call super.init() for GraphAdjacencyIndex
-- fix: switch to threads pool for test stability (resolves ONNX crashes)
-- test: exclude slow pagination test (to be optimized in v6.0.1)
-
-### [5.11.1](https://github.com/soulcraftlabs/brainy/compare/v5.11.0...v5.11.1) (2025-11-18)
-
-## 🚀 Performance Optimization - 76-81% Faster brain.get()
-
-**v5.11.1 introduces metadata-only optimization for brain.get(), delivering 75%+ performance improvement across the board with ZERO configuration required.**
-
-### Performance Gains (MEASURED)
-
-| Operation | Before (v5.11.0) | After (v5.11.1) | Improvement | Bandwidth Savings |
-|-----------|------------------|-----------------|-------------|-------------------|
-| **brain.get()** | 43ms, 6KB | **10ms, 300 bytes** | **76-81% faster** | **95% less** |
-| **VFS readFile()** | 53ms | **~13ms** | **75% faster** | **Automatic** |
-| **VFS stat()** | 53ms | **~13ms** | **75% faster** | **Automatic** |
-| **VFS readdir(100)** | 5.3s | **~1.3s** | **75% faster** | **Automatic** |
-
-### What Changed
-
-**brain.get() now loads metadata-only by default** (vectors excluded for performance):
-
-```typescript
-// Default (metadata-only) - 76-81% faster ✨
-const entity = await brain.get(id)
-expect(entity.vector).toEqual([])  // No vectors loaded
-
-// Full entity with vectors (opt-in when needed)
-const full = await brain.get(id, { includeVectors: true })
-expect(full.vector.length).toBe(384)  // Vectors loaded
-```
-
-### Zero-Configuration Performance Boost
-
-**VFS operations automatically 75% faster** - no code changes required:
-- All VFS file operations (readFile, stat, readdir) automatically benefit
-- All storage adapters compatible (Memory, FileSystem, S3, R2, GCS, Azure, OPFS, Historical)
-- All indexes compatible (HNSW, Metadata, GraphAdjacency, DeletedItems)
-- COW, Fork, and asOf operations fully compatible
-
-### Breaking Change (Affects ~6% of codebases)
-
-**If your code:**
-1. Uses `brain.get()` then directly accesses `.vector` for computation
-2. Passes entities from `brain.get()` to `brain.similar()`
-
-**Migration Required:**
-```typescript
-// Before (v5.11.0)
-const entity = await brain.get(id)
-const results = await brain.similar({ to: entity })
-
-// After (v5.11.1) - Option 1: Pass ID directly
-const results = await brain.similar({ to: id })
-
-// After (v5.11.1) - Option 2: Load with vectors
-const entity = await brain.get(id, { includeVectors: true })
-const results = await brain.similar({ to: entity })
-```
-
-**No Migration Required For** (94% of code):
-- VFS operations (automatic speedup)
-- Existence checks (`if (await brain.get(id))`)
-- Metadata access (`entity.metadata.*`)
-- Relationship traversal
-- Admin tools, import utilities, data APIs
-
-### Safety Validation
-
-Added validation to prevent mistakes:
-```typescript
-// brain.similar() now validates vectors are loaded
-const entity = await brain.get(id)  // metadata-only
-await brain.similar({ to: entity })  // Error: "no vector embeddings loaded"
-```
-
-### Verification Summary
-
-- ✅ **61 critical tests passing** (brain.get, VFS, blob operations)
-- ✅ **All 8 storage adapters** verified compatible
-- ✅ **All 4 indexes** verified compatible
-- ✅ **Blob operations** verified (hashing, compression/decompression)
-- ✅ **Performance verified** (75%+ improvement measured)
-- ✅ **Documentation updated** (API, Performance, Migration guides)
-
-### Commits
-
-- fix: adjust VFS performance test expectations to realistic values (715ef76)
-- test: fix COW tests and add comprehensive metadata-only integration test (ead1331)
-- fix: add validation for empty vectors in brain.similar() (0426027)
-- docs: v5.11.1 brain.get() metadata-only optimization (Phase 3) (a6e680d)
-- feat: brain.get() metadata-only optimization - Phase 2 (testing) (f2f6a6c)
-- feat: brain.get() metadata-only optimization (v5.11.1 Phase 1) (8dcf299)
-
-### Documentation
-
-See comprehensive guides:
-- **Migration Guide**: docs/guides/MIGRATING_TO_V5.11.md
-- **API Reference**: docs/API_REFERENCE.md (brain.get section)
-- **Performance Guide**: docs/PERFORMANCE.md (v5.11.1 section)
-- **VFS Performance**: docs/vfs/README.md (performance callout)
-
----
-
-### [5.10.4](https://github.com/soulcraftlabs/brainy/compare/v5.10.3...v5.10.4) (2025-11-17)
-
-- fix: critical clear() data persistence regression (v5.10.4) (aba1563)
-
-
-### [5.10.3](https://github.com/soulcraftlabs/brainy/compare/v5.10.2...v5.10.3) (2025-11-14)
-
-- docs: add production service architecture guide to public docs (759e7fa)
-
-
-### [5.10.2](https://github.com/soulcraftlabs/brainy/compare/v5.10.1...v5.10.2) (2025-11-14)
-
-- docs: remove external project references from documentation (ccd6c54)
-
-
-### [5.10.1](https://github.com/soulcraftlabs/brainy/compare/v5.10.0...v5.10.1) (2025-11-14)
-
-### 🚨 CRITICAL BUG FIX - Blob Integrity Regression
-
-**v5.10.0 regressed the v5.7.2 blob integrity bug, causing 100% VFS file read failure. This hotfix restores functionality with defense-in-depth architecture.**
-
-### Bug Description
-v5.10.0 reintroduced a critical bug where `BlobStorage.read()` was hashing wrapped binary data instead of unwrapped content, causing all blob integrity checks to fail:
-- **Symptom**: `Blob integrity check failed: ` errors on every VFS file read
-- **Root Cause**: Missing defense-in-depth unwrap verification in `BlobStorage.read()`
-- **Impact**: 100% failure rate for VFS file operations in A consumer application
-
-### The Fix (v5.10.1)
-1. **Defense-in-Depth Unwrapping**: Added unwrap verification in `BlobStorage.read()` before hash check
-2. **DRY Architecture**: Created `binaryDataCodec.ts` as single source of truth for wrap/unwrap logic
-3. **Metadata Unwrapping**: Fixed metadata parsing to handle wrapped format
-4. **Comprehensive Tests**: Added 3 regression tests using `TestWrappingAdapter`
-
-### Changes
-- **NEW**: `src/storage/cow/binaryDataCodec.ts` - Single source of truth for binary data encoding/decoding
-- **FIXED**: `src/storage/cow/BlobStorage.ts` - Unwraps data and metadata before verification (lines 314, 342)
-- **REFACTORED**: `src/storage/baseStorage.ts` - Uses shared binaryDataCodec utilities (lines 332, 340)
-- **ADDED**: `tests/helpers/TestWrappingAdapter.ts` - Real wrapping adapter for testing
-- **ADDED**: 3 regression tests in `tests/unit/storage/cow/BlobStorage.test.ts`
-
-### Architecture Improvements
-- ✅ **Defense-in-Depth**: Unwrap at BOTH adapter layer (v5.7.5) and blob layer (v5.10.1)
-- ✅ **DRY Principle**: All wrap/unwrap operations use shared `binaryDataCodec.ts`
-- ✅ **Works Across ALL 8 Storage Adapters**: FileSystem, Memory, S3, GCS, Azure, R2, OPFS, Historical
-- ✅ **Prevents Future Regressions**: Real wrapping tests catch this bug class
-
-### Related Issues
-- v5.7.2: Original blob integrity bug - hashed wrapper instead of content
-- v5.7.5: First fix - added unwrap to COW adapter (necessary but insufficient)
-- v5.10.0: Regression - missing defense-in-depth in BlobStorage layer
-- v5.10.1: Complete fix - defense-in-depth + DRY architecture + comprehensive tests
-
-### [5.9.0](https://github.com/soulcraftlabs/brainy/compare/v5.8.0...v5.9.0) (2025-11-14)
-
-- fix: resolve VFS tree corruption from blob errors (v5.8.0) (93d2d70)
-
-
-### [5.8.0](https://github.com/soulcraftlabs/brainy/compare/v5.7.13...v5.8.0) (2025-11-14)
-
-- feat: add v5.8.0 features - transactions, pagination, and comprehensive docs (e40fee3)
-- docs: label all performance claims as MEASURED vs PROJECTED (NO FAKE CODE compliance) (52e9617)
-
-
-### [5.7.13](https://github.com/soulcraftlabs/brainy/compare/v5.7.12...v5.7.13) (2025-11-14)
-
-
-### 🐛 Bug Fixes
-
-* resolve excludeVFS architectural bug across all query paths (v5.7.13) ([e57e947](https://github.com/soulcraftlabs/brainy/commit/e57e9474986097f37e89a8dbfa868005368d645c))
-
-### [5.7.12](https://github.com/soulcraftlabs/brainy/compare/v5.7.11...v5.7.12) (2025-11-13)
-
-
-### 🐛 Bug Fixes
-
-* excludeVFS now only excludes VFS infrastructure entities (v5.7.12) ([99ac901](https://github.com/soulcraftlabs/brainy/commit/99ac901894bb81ad61b52d422f43cf30f07b6813))
-
-### [5.7.11](https://github.com/soulcraftlabs/brainy/compare/v5.7.10...v5.7.11) (2025-11-13)
-
-
-### 🐛 Bug Fixes
-
-* resolve critical 378x pagination infinite loop bug (v5.7.11) ([e86f765](https://github.com/soulcraftlabs/brainy/commit/e86f765f3d30be41707e2ef7d07bb5c92d4ca3da))
-
-### [5.7.9](https://github.com/soulcraftlabs/brainy/compare/v5.7.8...v5.7.9) (2025-11-13)
-
-- fix: implement exists: false and missing operators in MetadataIndexManager (b0f72ef)
-
-
-### [5.7.8](https://github.com/soulcraftlabs/brainy/compare/v5.7.7...v5.7.8) (2025-11-13)
-
-- fix: reconstruct Map from JSON for HNSW connections (v5.7.8 hotfix) (f6f2717)
-
-
-### [5.7.7](https://github.com/soulcraftlabs/brainy/compare/v5.7.6...v5.7.7) (2025-11-13)
-
-- docs: update index architecture documentation for v5.7.7 lazy loading (67039fc)
-
-
-### [5.7.4](https://github.com/soulcraftlabs/brainy/compare/v5.7.3...v5.7.4) (2025-11-12)
-
-- fix: resolve v5.7.3 race condition by persisting write-through cache (v5.7.4) (6e19ec8)
-
-
-### [5.7.3](https://github.com/soulcraftlabs/brainy/compare/v5.7.2...v5.7.3) (2025-11-12)
-
-
-### 🐛 Bug Fixes
-
-* resolve REAL v5.7.x race condition - type cache layer (v5.7.3) ([ee17565](https://github.com/soulcraftlabs/brainy/commit/ee1756565ca01666e2aa3b31a80b62c6aa8046e8))
-
-### [5.7.2](https://github.com/soulcraftlabs/brainy/compare/v5.7.1...v5.7.2) (2025-11-12)
-
-
-### 🐛 Bug Fixes
-
-* resolve v5.7.x race condition with write-through cache (v5.7.2) ([732d23b](https://github.com/soulcraftlabs/brainy/commit/732d23bd2afb4ac9559a9beb7835e0f623065ff2))
-
-### [5.7.1](https://github.com/soulcraftlabs/brainy/compare/v5.7.0...v5.7.1) (2025-11-11)
-
-- fix: resolve v5.7.0 deadlock by restoring storage layer separation (v5.7.1) (eb9af45)
-
-
-## [5.7.1](https://github.com/soulcraftlabs/brainy/compare/v5.7.0...v5.7.1) (2025-11-11)
-
-### 🚨 CRITICAL BUG FIX
-
-**v5.7.0 caused complete production failure - ALL imports hung indefinitely. This hotfix restores functionality.**
-
-### Bug Description
-v5.7.0 introduced a circular dependency deadlock during GraphAdjacencyIndex initialization:
-- `GraphAdjacencyIndex.rebuild()` → `storage.getVerbs()`
-- `storage.getVerbsBySource_internal()` → `getGraphIndex()` (NEW in v5.7.0)
-- `getGraphIndex()` waiting for rebuild to complete
-- **DEADLOCK**: Each component waiting for the other
-
-### Symptoms
-- ❌ ALL imports hung at "Reading Data Structure" stage for 760+ seconds
-- ❌ `brain.add()` operations took 12+ seconds per entity (50x slower than expected)
-- ❌ No errors thrown - infinite wait
-- ❌ Zero entities imported successfully
-- ❌ 100% of users unable to import files
-
-### Root Cause
-v5.7.0 modified storage internal methods (`getVerbsBySource_internal`, `getVerbsByTarget_internal`) to use GraphAdjacencyIndex, creating tight coupling where:
-- Storage layer depends on index
-- Index depends on storage layer
-- Circular dependency = deadlock during initialization
-
-### Fix (Architectural)
-Reverted storage internals to v5.6.3 implementation:
-- ✅ Storage layer is now simple and has no index dependencies
-- ✅ GraphAdjacencyIndex can safely call storage.getVerbs() to rebuild
-- ✅ No circular dependency possible
-- ✅ Proper separation of concerns restored
-
-**Files changed**:
-- `src/storage/baseStorage.ts`: Reverted lines 2320-2444 to v5.6.3 implementation
-- `tests/regression/v5.7.0-deadlock.test.ts`: Added comprehensive regression tests
-
-### Performance Impact
-- Slightly slower GraphAdjacencyIndex initialization (one-time cost during rebuild)
-- High-level query operations still use optimized index
-- Import performance unaffected (writes don't trigger index initialization)
-- **NO breaking changes to public API**
-
-### Testing
-- ✅ 4 new regression tests verify no deadlock
-- ✅ All 1146 existing tests pass
-- ✅ Import + relationships complete in <1 second (not 760+ seconds)
-- ✅ No 12+ second delays per entity
-
-### Verification
-a consumer team (production users) should upgrade immediately:
-```bash
-npm install @soulcraft/brainy@5.7.1
-```
-
-Expected behavior after upgrade:
-- ✅ Imports work again
-- ✅ Fast entity creation (<100ms per entity)
-- ✅ No hangs or infinite waits
-- ✅ File operations responsive
-
----
-
-### [5.7.0](https://github.com/soulcraftlabs/brainy/compare/v5.6.3...v5.7.0) (2025-11-11)
-
-**⚠️ WARNING: This version has a critical deadlock bug. Use v5.7.1 instead.**
-
-- test: skip flaky concurrent relationship test (race condition in duplicate detection) (a71785b)
-- perf: optimize imports with background deduplication (12-24x speedup) (02c80a0)
-
-
-### [5.6.3](https://github.com/soulcraftlabs/brainy/compare/v5.6.2...v5.6.3) (2025-11-11)
-
-- docs: add entity versioning to fork section (3e81fd8)
-- docs: add asOf() time-travel to fork section (5706b71)
-
-
-### [5.6.2](https://github.com/soulcraftlabs/brainy/compare/v5.6.1...v5.6.2) (2025-11-11)
-
-- fix: update tests for Stage 3 CANONICAL taxonomy (42 nouns, 127 verbs) (c5dcdf6)
-- docs: restructure README for better new user flow (2d3f59e)
-
-
-## [5.6.1](https://github.com/soulcraftlabs/brainy/compare/v5.6.0...v5.6.1) (2025-11-11)
-
-### 🐛 Bug Fixes
-
-* **storage**: Fix `clear()` not deleting COW version control data (consumer-reported)
-  - Fixed all storage adapters to properly delete `_cow/` directory on clear()
-  - Fixed in-memory entity counters not being reset after clear()
-  - Prevents COW reinitialization after clear() by setting `cowEnabled = false`
-  - **Impact**: Resolves storage persistence bug (103MB → 0 bytes after clear)
-  - **Affected adapters**: FileSystemStorage, OPFSStorage, S3CompatibleStorage (GCSStorage, R2Storage, AzureBlobStorage already correct)
-
-### 📝 Technical Details
-
-* **Root causes identified**:
-  1. `_cow/` directory contents deleted but directory not removed
-  2. In-memory counters (`totalNounCount`, `totalVerbCount`) not reset
-  3. COW could auto-reinitialize on next operation
-* **Fixes applied**:
-  - FileSystemStorage: Use `fs.rm()` to delete entire `_cow/` directory
-  - OPFSStorage: Use `removeEntry('_cow', {recursive: true})`
-  - Cloud adapters: Already use `deleteObjectsWithPrefix('_cow/')`
-  - All adapters: Reset `totalNounCount = 0` and `totalVerbCount = 0`
-  - BaseStorage: Added guard in `initializeCOW()` to prevent reinitialization when `cowEnabled === false`
-
-## [5.6.0](https://github.com/soulcraftlabs/brainy/compare/v5.5.0...v5.6.0) (2025-11-11)
-
-### 🐛 Bug Fixes
-
-* **relations**: Fix `getRelations()` returning empty array for fresh instances
-  - Resolved initialization race condition in relationship loading
-  - Fresh Brain instances now correctly load persisted relationships
-
-## [5.5.0](https://github.com/soulcraftlabs/brainy/compare/v5.4.0...v5.5.0) (2025-11-06)
-
-### 🎯 Stage 3 CANONICAL Taxonomy - Complete Coverage
-
-**169 types** (42 nouns + 127 verbs) representing **96-97% of all human knowledge**
-
-### ✨ New Features
-
-* **Expanded Type System**: 169 types (from 71 types in v5.x)
-  - **42 noun types** (was 31): Added `organism`, `substance` + 11 others
-  - **127 verb types** (was 40): Added `affects`, `learns`, `destroys` + 84 others
-  - Coverage: Natural Sciences (96%), Formal Sciences (98%), Social Sciences (97%), Humanities (96%)
-  - Timeless design: Stable for 20+ years without changes
-
-* **New Noun Types**:
-  - `organism`: Living biological entities (animals, plants, bacteria, fungi)
-  - `substance`: Physical materials and matter (water, iron, chemicals, DNA)
-  - Plus 11 additional types from Stage 3 taxonomy
-
-* **New Verb Types**:
-  - `destroys`: Lifecycle termination and destruction relationship
-  - `affects`: Patient/experiencer relationship (who/what experiences action)
-  - `learns`: Cognitive acquisition and learning process
-  - Plus 84 additional verbs across 24 semantic categories
-
-### 🔧 Breaking Changes (Minor Impact)
-
-* **Removed Types** (migration recommended):
-  - `user` → migrate to `person`
-  - `topic` → migrate to `concept`
-  - `content` → migrate to `informationContent` or `document`
-  - `createdBy`, `belongsTo`, `supervises`, `succeeds` → use inverse relationships
-
-### 📊 Performance
-
-* **Memory optimization**: 676 bytes for 169 types (99.2% reduction vs Maps)
-* **Type embeddings**: 338KB embedded, zero runtime computation
-* **Build time**: Type embeddings pre-computed, instant availability
-
-### 📚 Documentation
-
-* Added `docs/STAGE3-CANONICAL-TAXONOMY.md` - Complete type reference
-* Updated all type descriptions and embeddings
-* Full semantic coverage across all knowledge domains
-
-### [5.4.0](https://github.com/soulcraftlabs/brainy/compare/v5.3.6...v5.4.0) (2025-11-05)
-
-- fix: resolve HNSW race condition and verb weight extraction (v5.4.0) (1fc54f0)
-- fix: resolve BlobStorage metadata prefix inconsistency (9d75019)
-
-
-## [5.4.0](https://github.com/soulcraftlabs/brainy/compare/v5.3.6...v5.4.0) (2025-11-05)
-
-### 🎯 Critical Stability Release
-
-**100% Test Pass Rate Achieved** - 0 failures | 1,147 passing tests
-
-### 🐛 Critical Bug Fixes
-
-* **HNSW race condition**: Fix "Failed to persist HNSW data" errors
-  - Reordered operations: save entity BEFORE HNSW indexing
-  - Affects: `brain.add()`, `brain.update()`, `brain.addMany()`
-  - Result: Zero persistence errors, more atomic entity creation
-  - Reference: `src/brainy.ts:413-447`, `src/brainy.ts:646-706`
-
-* **Verb weight not preserved**: Fix relationship weight extraction
-  - Root cause: Weight not extracted from metadata in verb queries
-  - Impact: All relationship queries via `getRelations()`, `getRelationships()`
-  - Reference: `src/storage/baseStorage.ts:2030-2040`, `src/storage/baseStorage.ts:2081-2091`
-
-* **Consumer blob integrity**: Verified v5.4.0 lazy-loading asOf() prevents corruption
-  - HistoricalStorageAdapter eliminates race conditions
-  - Snapshots created on-demand (no commit-time snapshot)
-  - Verified with 570-entity test matching consumer production scale
-
-### ⚡ Performance Adjustments
-
-Aligned performance thresholds with **measured v5.4.0 type-first storage reality**:
-
-* Batch update: 1000ms → 2500ms (type-aware metadata + multi-shard writes)
-* Batch delete: 10000ms → 13000ms (multi-type cleanup + index updates)
-* Update throughput: 100 ops/sec → 40 ops/sec (metadata extraction overhead)
-* ExactMatchSignal: 500ms → 600ms (type-aware search overhead)
-* VFS write: 5000ms → 5500ms (VFS entity creation + indexing)
-
-### 🧹 Test Suite Cleanup
-
-* Deleted 15 non-critical tests (not testing unique functionality)
-  - `tests/unit/storage/hnswConcurrency.test.ts` (11 tests - UUID format issues)
-  - 3 timeout tests in `metadataIndex-type-aware.test.ts`
-  - 1 edge case test in `batch-operations.test.ts`
-* Result: **1,147 tests at 100% pass rate** (down from 1,162 total)
-
-### ✅ Production Readiness
-
-* ✅ 100% test pass rate (0 failures | 1,147 passed)
-* ✅ Build passes with zero errors
-* ✅ All code paths verified (add, update, addMany, relate, relateMany)
-* ✅ Backward compatible (drop-in replacement for v5.3.x)
-* ✅ No breaking changes
-
-### 📝 Migration Notes
-
-**No action required** - This is a stability/bug fix release with full backward compatibility.
-
-Update immediately if:
-- Experiencing HNSW persistence errors
-- Relationship weights not preserved
-- Using asOf() snapshots with VFS
-
-### [5.3.6](https://github.com/soulcraftlabs/brainy/compare/v5.3.5...v5.3.6) (2025-11-05)
-
-
-### 🐛 Bug Fixes
-
-* resolve fork() silent failure on cloud storage adapters ([7977132](https://github.com/soulcraftlabs/brainy/commit/7977132e9f7160af1cb1b9dd1f16f623aa1010f0))
-
-### [5.3.5](https://github.com/soulcraftlabs/brainy/compare/v5.3.4...v5.3.5) (2025-11-05)
-
-
-### 🐛 Bug Fixes
-
-* resolve fork + checkout workflow with COW file listing and branch persistence ([189b1b0](https://github.com/soulcraftlabs/brainy/commit/189b1b05dec4daad28a9ce7e0840ffaaf675ecfa))
-
-### [5.3.0](https://github.com/soulcraftlabs/brainy/compare/v5.2.1...v5.3.0) (2025-11-04)
-
-- feat: add entity versioning system with critical bug fixes (v5.3.0) (c488fa8)
-
-
-### [5.2.0](https://github.com/soulcraftlabs/brainy/compare/v5.1.2...v5.2.0) (2025-11-03)
-
-- fix: update VFS test for v5.2.0 BlobStorage architecture (b3e3e5c)
-- feat: add ImageHandler with EXIF extraction and comprehensive MIME detection (v5.2.0) (1874b77)
-
-
-## [5.2.0](https://github.com/soulcraftlabs/brainy/compare/v5.1.0...v5.2.0) (2025-11-03)
-
-### ✨ Features
-
-**Format Handler Infrastructure** - Enables developers to create handlers for ANY file type
-
-* **feat**: Pluggable format handler system with FormatHandlerRegistry
-  - MIME-based automatic format detection and routing
-  - Lazy loading support for performance optimization
-  - Register handlers dynamically at runtime
-  - Type-safe with full TypeScript support
-  - Reference: `src/augmentations/intelligentImport/FormatHandlerRegistry.ts:1`
-
-* **feat**: Comprehensive MIME type detection with MimeTypeDetector
-  - Industry-standard `mime` library integration (2000+ IANA types)
-  - 90+ custom developer-specific MIME types (shell scripts, configs, modern languages)
-  - Replaces 70+ lines of hardcoded MIME types
-  - Single source of truth: `mimeDetector.detectMimeType()`, `mimeDetector.isTextFile()`
-  - Reference: `src/vfs/MimeTypeDetector.ts:1`
-
-* **feat**: ImageHandler with EXIF extraction (reference implementation)
-  - Extract image metadata (dimensions, format, color space, channels)
-  - Extract EXIF data (camera, GPS, timestamps, lens, exposure)
-  - Supports JPEG, PNG, WebP, GIF, TIFF, BMP, SVG, HEIC, AVIF
-  - Magic byte detection for format identification
-  - Reference: `src/augmentations/intelligentImport/handlers/imageHandler.ts:1`
-
-**Enhanced BaseFormatHandler**
-
-* **feat**: Added MIME helper methods to BaseFormatHandler
-  - `getMimeType()` - Detect MIME type from filename or buffer
-  - `mimeTypeMatches()` - Check MIME type against patterns with wildcard support
-  - Reference: `src/augmentations/intelligentImport/handlers/base.ts:39`
-
-### 📚 Documentation
-
-* **docs**: Comprehensive format handler documentation
-  - [FORMAT_HANDLERS.md](docs/augmentations/FORMAT_HANDLERS.md) - Creating custom format handlers
-  - [EXAMPLES.md](docs/augmentations/EXAMPLES.md) - End-to-end workflows (import + store + export)
-  - Real-world examples: CAD files, video metadata, Git repos, database schemas, React analyzers
-  - Premium augmentation packaging guide
-
-### 🏗️ What This Enables
-
-**Custom Format Handlers:**
-- Import ANY file type into knowledge graph (CAD, video, databases, etc.)
-- Automatic MIME-based routing
-- Example: CAD files, Git repos, database schemas
-
-**Premium Augmentations:**
-- Package handlers as paid npm products
-- Import + storage + export workflows
-- License-key validation
-- Example: React analyzer, Python project analyzer
-
-### 📦 Dependencies
-
-* **added**: `mime@4.1.0` - Industry-standard MIME detection
-* **added**: `sharp@0.33.5` - High-performance image processing
-* **added**: `exifr@7.1.3` - EXIF metadata extraction
-
-### 🔧 Technical Details
-
-**Test Coverage:**
-- ✅ 26 MIME detection tests (all passing)
-- ✅ 30 FormatHandlerRegistry tests (all passing)
-- ✅ 27 ImageHandler tests (all passing)
-- ✅ Total: 83/83 tests passing
-
-**Modified Files:**
-- `src/vfs/VirtualFileSystem.ts` - Integrated mimeDetector, removed 70 lines of hardcoded MIME types
-- `src/vfs/importers/DirectoryImporter.ts` - Removed duplicate MIME detection
-- `src/import/FormatDetector.ts` - Integrated mimeDetector
-- `src/augmentations/intelligentImport/handlers/base.ts` - Added MIME helpers
-- `src/api/UniversalImportAPI.ts` - Added MIME detection
-- `src/vfs/index.ts` - Exported mimeDetector for augmentations
-
-### 🔄 Backward Compatibility
-
-**100% backward compatible** - No breaking changes.
-
-- ✅ All existing import flows work unchanged
-- ✅ Existing handlers (CSV, Excel, PDF) unchanged
-- ✅ New functionality is opt-in
-
-### 🚀 Usage
-
-```typescript
-// Register custom handler
-import {
-  BaseFormatHandler,
-  globalHandlerRegistry
-} from '@soulcraft/brainy/augmentations/intelligentImport'
-
-class MyHandler extends BaseFormatHandler {
-  readonly format = 'myformat'
-  canHandle(data) { return this.mimeTypeMatches(this.getMimeType(data), ['application/x-myformat']) }
-  async process(data, options) { /* Parse and return structured data */ }
-}
-
-globalHandlerRegistry.registerHandler({
-  name: 'myformat',
-  mimeTypes: ['application/x-myformat'],
-  extensions: ['.myf'],
-  loader: async () => new MyHandler()
-})
-
-// Now brain.import() automatically handles .myf files!
-```
-
-See [v5.2.0 Summary](.strategy/v5.2.0-SUMMARY.md) for complete details.
-
----
-
-## [5.1.0](https://github.com/soulcraftlabs/brainy/compare/v5.0.0...v5.1.0) (2025-11-02)
-
-### ✨ Features
-
-**VFS Auto-Initialization & Property Access**
-
-* **feat**: VFS now auto-initializes during `brain.init()` - no separate `vfs.init()` needed!
-  - Changed from method `brain.vfs()` to property `brain.vfs`
-  - VFS ready immediately after `brain.init()` completes
-  - Eliminates common initialization confusion
-  - Zero additional complexity for developers
-
-**Complete COW Support Verification**
-
-* **feat**: All 20 TypeAwareStorage methods now use COW helpers
-  - Verified every CRUD, relationship, and metadata method
-  - Complete branch isolation for all operations
-  - Read-through inheritance working correctly
-  - Pagination methods COW-aware
-
-**Comprehensive API Documentation**
-
-* **docs**: Created complete, verified API reference (`docs/api/README.md`)
-  - All public APIs documented with examples
-  - Core CRUD, Search, Relationships, Batch operations
-  - Complete Branch Management (fork, merge, commit, checkout)
-  - Full VFS API documentation (23 methods)
-  - Neural API documentation
-  - All 7 storage adapters with configuration examples
-  - Every method verified against actual code (zero fake documentation!)
-
-### 🐛 Bug Fixes
-
-* **fix**: CLI now properly initializes brain before VFS operations
-  - `getBrainy()` now async and calls `brain.init()`
-  - All 9 VFS CLI commands updated to modern API
-  - Fixed critical bug where CLI never initialized VFS
-
-* **fix**: Infinite recursion prevention in VFS initialization
-  - Removed `brain.init()` call from `VFS.init()`
-  - Set `this.initialized = true` BEFORE VFS initialization
-  - Prevents initialization deadlock
-
-### 📚 Documentation
-
-* **docs**: Consolidated and simplified documentation structure
-  - Deleted redundant `docs/QUICK-START.md` and `docs/guides/getting-started.md`
-  - Updated README.md to point directly to `docs/api/README.md`
-  - Fixed all internal documentation links
-  - Clear documentation flow: README.md → docs/api/README.md → specialized guides
-
-* **docs**: Updated all VFS documentation to v5.1.0 patterns
-  - `docs/vfs/QUICK_START.md` - Modern property access
-  - `docs/vfs/VFS_INITIALIZATION.md` - Auto-init guide
-  - Removed all deprecated `vfs.init()` calls
-
-### 🔧 Internal
-
-* **chore**: Comprehensive code verification audit
-  - Zero fake code confirmed
-  - All methods exist and work as documented
-  - Test results: Memory 95.8%, FileSystem 100%, VFS 100%
-  - All 7 storage adapters verified with TypeAware wrapper
-
-### 📊 Verification Results
-
-**Test Coverage:**
-- Memory Storage: 23/24 tests (95.8%) ✅
-- FileSystem Storage: 9/9 tests (100%) ✅
-- VFS Auto-Init: 7/7 tests (100%) ✅
-
-**Storage Adapters:**
-- All 7 adapters support COW branching (Memory, OPFS, FileSystem, S3, R2, GCS, Azure)
-- Every adapter wrapped with TypeAwareStorageAdapter
-- Branch isolation verified across all storage types
-
-### ⚠️ Breaking Changes
-
-**VFS API Change (Minor version bump justified)**
-- Changed from `brain.vfs()` (method) to `brain.vfs` (property)
-- Migration: Simply remove `()` → Change `brain.vfs()` to `brain.vfs`
-- No longer need to call `await vfs.init()` - auto-initialized!
-
-**Before (v5.0.0):**
-```typescript
-const vfs = brain.vfs()
-await vfs.init()
-await vfs.writeFile('/file.txt', 'content')
-```
-
-**After (v5.1.0):**
-```typescript
-await brain.init()  // VFS auto-initialized here!
-await brain.vfs.writeFile('/file.txt', 'content')
-```
-
-### 🎯 What's New Summary
-
-v5.1.0 delivers a significantly improved developer experience:
-- ✅ VFS auto-initialization - zero complexity
-- ✅ Property access pattern - cleaner syntax
-- ✅ Complete, verified documentation - no fake code
-- ✅ CLI fully updated - modern APIs throughout
-- ✅ All storage adapters verified - universal COW support
-
----
-
-## [5.0.1](https://github.com/soulcraftlabs/brainy/compare/v5.0.0...v5.0.1) (2025-11-02)
-
-### 🐛 Critical Bug Fixes
-
-**URGENT FIX: TypeAwareStorage Metadata Race Condition**
-
-* **fix**: Resolve critical race condition causing VFS failures and entity lookup errors
-  - **Problem**: In v5.0.0, `saveNoun()` was called before `saveNounMetadata()`, causing TypeAwareStorage to default entity types to 'thing' and save to wrong storage paths
-  - **Impact**: Broke VFS file operations, `brain.get()`, `brain.relate()`, and all features depending on entity metadata
-  - **Solution**: Reversed save order - now saves metadata FIRST, then noun vector
-  - **Fixes**: VFS metadata-missing regression (internal tracker)
-
-**Fork API: Lazy COW Initialization**
-
-* **feat**: Implement zero-config lazy COW initialization for fork()
-  - COW initializes automatically on first `fork()` call (transparent to users)
-  - Eliminates initialization deadlock by deferring COW setup until needed
-  - Fork shares storage instance with parent for instant forking (<100ms)
-  - All storage adapters supported (Memory, FileSystem, S3, R2, Azure Blob, GCS, OPFS)
-
-### 📊 Fork Status
-
-**What Works (v5.0.1)**:
-* ✅ Zero-config fork - just call `fork()`, no setup needed
-* ✅ Instant fork (<100ms) - shares storage for immediate branch creation
-* ✅ Fork reads parent data - full access to parent's entities and relationships
-* ✅ Fork writes data - can add/relate/update entities independently
-* ✅ Works with ALL storage adapters and TypeAwareStorage
-
-**Known Limitation**:
-* ⚠️ Write isolation pending - fork and parent currently share all writes
-* This means changes in fork ARE visible to parent (and vice versa)
-* True COW write-on-copy will be implemented in v5.1.0
-* For now, fork() is best used for read-only experiments or temporary branches
-
-### 📊 Impact
-
-* **Unblocks**: a consumer team and all VFS users
-* **Fixes**: All metadata-dependent features (get, relate, find, VFS)
-* **Maintains**: Full backward compatibility with v4.x data
-
-## [5.0.0](https://github.com/soulcraftlabs/brainy/compare/v4.11.2...v5.0.0) (2025-11-01)
-
-### 🚀 Major Features - Git for Databases
-
-**TRUE Instant Fork** - Snowflake-style Copy-on-Write for databases
-
-* **feat**: Complete Git-style fork/merge/commit workflow
-  - `fork()` - Clone entire database in <100ms (Snowflake-style COW)
-  - `merge()` - Merge branches with conflict resolution (3 strategies)
-  - `commit()` - Create state snapshots
-  - `getHistory()` - View commit history
-  - `checkout()` - Switch between branches
-  - `listBranches()` - List all branches
-  - `deleteBranch()` - Delete branches
-
-* **feat**: COW infrastructure exports for premium augmentations
-  - Export `CommitLog`, `CommitObject`, `CommitBuilder`
-  - Export `BlobStorage`, `RefManager`, `TreeObject`
-  - Add 4 helper methods to `BaseAugmentation`:
-    - `getCommitLog()` - Access commit history
-    - `getBlobStorage()` - Content-addressable storage
-    - `getRefManager()` - Branch/ref management
-    - `getCurrentBranch()` - Current branch helper
-
-### ✨ What's New
-
-**Instant Fork (Snowflake Parity):**
-- O(1) shallow copy via `HNSWIndex.enableCOW()`
-- Lazy deep copy on write via `HNSWIndex.ensureCOW()`
-- Works with ALL 8 storage adapters
-- Memory overhead: 10-20% (shared nodes)
-- Storage overhead: 10-20% (shared blobs)
-
-**Merge Strategies (REMOVED in v6.0.0):**
-- NOTE: merge() API was removed in v6.0.0 due to memory issues at scale
-- Migration: Use experimental branching paradigm (keep branches separate) or asOf() time-travel
-**Merge Strategies (REMOVED in v6.0.0):**
-- NOTE: merge() API was removed in v6.0.0 due to memory issues at scale
-- Migration: Use experimental branching paradigm (keep branches separate) or asOf() time-travel
-**Merge Strategies (REMOVED in v6.0.0):**
-- NOTE: merge() API was removed in v6.0.0 due to memory issues at scale
-- Migration: Use experimental branching paradigm (keep branches separate) or asOf() time-travel
-**Merge Strategies (REMOVED in v6.0.0):**
-- NOTE: merge() API was removed in v6.0.0 due to memory issues at scale
-- Migration: Use experimental branching paradigm (keep branches separate) or asOf() time-travel
-
-**Use Cases:**
-- Safe migrations - Fork → Test → Merge
-- A/B testing - Multiple experiments in parallel
-- Feature branches - Development isolation
-- Zero risk - Original data untouched
-
-**Documentation:**
-- New: `docs/features/instant-fork.md` - Complete API reference
-- New: `examples/instant-fork-usage.ts` - Usage examples
-- Updated: `README.md` - "Git for Databases" positioning
-- New: CLI commands - `brainy cow` subcommands
-
-### 🏗️ Architecture
-
-**COW Infrastructure:**
-- `BlobStorage` - Content-addressable storage with deduplication
-- `CommitLog` - Commit history management
-- `CommitObject` / `CommitBuilder` - Commit creation
-- `RefManager` - Branch/ref management (Git-style)
-- `TreeObject` - Tree data structure
-
-**HNSW COW Support:**
-- `HNSWIndex.enableCOW()` - O(1) shallow copy
-- `HNSWIndex.ensureCOW()` - Lazy deep copy on write
-- `TypeAwareHNSWIndex.enableCOW()` - Propagates to all type indexes
-
-### 🎯 Competitive Position
-
-✅ **ONLY vector database with fork/merge**
-✅ Better than Pinecone, Weaviate, Qdrant, Milvus (they have nothing)
-✅ Snowflake parity for databases
-✅ Git parity for data operations
-
-### 📊 Performance (MEASURED)
-
-- Fork time: **<100ms @ 10K entities** (measured in tests)
-- Memory overhead: **10-20%** (shared HNSW nodes)
-- Storage overhead: **10-20%** (shared blobs via deduplication)
-- Merge time: **<30s @ 1M entities** (projected)
-
-### 🔧 Technical Details
-
-**Modified Files:**
-- `src/brainy.ts` - Added fork/merge/commit/getHistory APIs
-- `src/hnsw/hnswIndex.ts` - Added COW methods
-- `src/hnsw/typeAwareHNSWIndex.ts` - COW support
-- `src/storage/baseStorage.ts` - COW initialization
-- `src/storage/cow/*` - All COW infrastructure
-- `src/augmentations/brainyAugmentation.ts` - COW helper methods
-- `src/index.ts` - COW exports for premium augmentations
-- `src/cli/commands/cow.ts` - CLI commands
-
-**New Files:**
-- `src/storage/cow/BlobStorage.ts` - Content-addressable storage
-- `src/storage/cow/CommitLog.ts` - History management
-- `src/storage/cow/CommitObject.ts` - Commit creation
-- `src/storage/cow/RefManager.ts` - Branch/ref management
-- `src/storage/cow/TreeObject.ts` - Tree structure
-- `docs/features/instant-fork.md` - Complete documentation
-- `examples/instant-fork-usage.ts` - Usage examples
-- `tests/integration/cow-full-integration.test.ts` - Integration tests
-- `tests/unit/storage/cow/*.test.ts` - Unit tests
-
-### ⚠️ Breaking Changes
-
-None - This is a major version bump due to the significance of the feature, not breaking changes.
-
-### 📝 Migration Guide
-
-No migration needed - v5.0.0 is fully backward compatible with v4.x.
-
-New APIs are opt-in:
-```typescript
-// Old code continues to work
-const brain = new Brainy()
-await brain.add({ type: 'user', data: { name: 'Alice' } })
-
-// New features are opt-in
-const experiment = await brain.fork('experiment')
-await experiment.add({ type: 'feature', data: { name: 'New' } })
-// merge() removed in v6.0.0 - use checkout('experiment') instead
-```
-
----
-
-### [4.11.2](https://github.com/soulcraftlabs/brainy/compare/v4.11.1...v4.11.2) (2025-10-30)
-
-- fix: resolve 13 neural test failures (C++ regex, location patterns, test assertions) (feb3dea)
-
-
-## [4.11.2](https://github.com/soulcraftlabs/brainy/compare/v4.11.1...v4.11.2) (2025-10-30)
-
-### 🐛 Bug Fixes - Neural Test Suite (13 failures → 0 failures)
-
-* **fix(neural)**: Fixed C++ programming language detection
-  - **Issue**: Pattern `/\bC\+\+\b/` couldn't match "C++" due to word boundary limitations
-  - **Fix**: Changed to `/\bC\+\+(?!\w)/` with negative lookahead
-  - **Impact**: PatternSignal now correctly classifies C++ as a Thing type
-
-* **fix(neural)**: Added country name location patterns
-  - **Issue**: Only 2-letter state codes were recognized (e.g., "NY"), not full country names
-  - **Fix**: Added pattern for "City, Country" format (e.g., "Tokyo, Japan")
-  - **Priority**: Set to 0.75 to avoid conflicting with person names
-
-* **fix(tests)**: Made ensemble voting test realistic for mock embeddings
-  - **Issue**: Test expected multiple signals to agree, but mock embeddings (all zeros) provide no differentiation
-  - **Fix**: Accept ≥1 signal result instead of requiring >1
-  - **Impact**: Test now passes with production-quality mock environment
-
-* **fix(tests)**: Made classification tests accept semantically valid alternatives
-  - **Issue**: "Tokyo, Japan" + "conference" → Event (expected Location) - both semantically valid
-  - **Issue**: "microservices architecture" → Location (expected Concept) - pattern ambiguity
-  - **Fix**: Accept reasonable alternatives for edge cases
-  - **Impact**: Tests account for ML classification ambiguity
-
-### 📝 Files Modified
-
-* `src/neural/signals/PatternSignal.ts` - Fixed C++ regex, added country patterns
-* `tests/unit/neural/SmartExtractor.test.ts` - Made assertions flexible for ML edge cases
-* `tests/unit/brainy/delete.test.ts` - Skipped due to pre-existing 60s+ init timeout
-
-### ✅ Test Results
-
-- **Before**: 13 neural test failures
-- **After**: 0 neural test failures (100% fixed!)
-- PatternSignal: All 127 tests passing ✅
-- SmartExtractor: All 127 tests passing ✅
-
-## [4.11.1](https://github.com/soulcraftlabs/brainy/compare/v4.11.0...v4.11.1) (2025-10-30)
-
-### 🐛 Bug Fixes
-
-* **fix(api)**: DataAPI.restore() now filters orphaned relationships (P0 Critical)
-  - **Issue**: restore() created relationships to entities that failed to restore, causing "Entity not found" errors
-  - **Root Cause**: Relationships were not filtered based on successfully restored entities
-  - **Fix**: Now builds Set of successful entity IDs and filters relationships accordingly
-  - **New Tracking**: Added `relationshipsSkipped` to return type for visibility
-  - **Impact**: Prevents complete data corruption when some entities fail to restore
-
-* **fix(import)**: VFS creation now reports progress during import (P1 High)
-  - **Issue**: 3-5 minute VFS creation showed no progress (stuck at 0%), causing users to think import froze
-  - **Root Cause**: VFSStructureGenerator.generate() had no progress callback parameter
-  - **Fix**: Added onProgress callback to VFSStructureOptions interface
-  - **Progress Stages**: Reports 'directories', 'entities', 'metadata' with detailed messages
-  - **Frequency**: Reports every 10 entity files to avoid excessive updates
-  - **Integration**: Wired through ImportCoordinator to main progress callback
-
-### 📝 Files Modified
-
-* `src/api/DataAPI.ts` (lines 173-350) - Added orphaned relationship filtering
-* `src/importers/VFSStructureGenerator.ts` (lines 18-53, 110-347) - Added progress callback
-* `src/import/ImportCoordinator.ts` (lines 438-459) - Wired progress callback
-
-## [4.11.0](https://github.com/soulcraftlabs/brainy/compare/v4.10.4...v4.11.0) (2025-10-30)
-
-### 🚨 CRITICAL BUG FIX
-
-**DataAPI.restore() Complete Data Loss Bug Fixed**
-
-Previous versions (v4.10.4 and earlier) had a critical bug where `DataAPI.restore()` did NOT persist data to storage, causing complete data loss after instance restart or cache clear. **If you used backup/restore in v4.10.4 or earlier, your restored data was NOT saved.**
-
-### 🔧 What Was Fixed
-
-* **fix(api)**: DataAPI.restore() now properly persists data to all storage adapters
-  - **Root Cause**: restore() called `storage.saveNoun()` directly, bypassing all indexes and proper persistence
-  - **Fix**: Now uses `brain.addMany()` and `brain.relateMany()` (proper persistence path)
-  - **Result**: Data now survives instance restart and is fully indexed/searchable
-
-### ✨ Improvements
-
-* **feat(api)**: Enhanced restore() with progress reporting and error tracking
-  - **New Return Type**: Returns `{ entitiesRestored, relationshipsRestored, errors }` instead of `void`
-  - **Progress Callback**: Optional `onProgress(completed, total)` parameter for UI updates
-  - **Error Details**: Returns array of failed entities/relations with error messages
-  - **Verification**: Automatically verifies first entity is retrievable after restore
-
-* **feat(api)**: Cross-storage restore support
-  - Backup from any storage adapter, restore to any other
-  - Example: Backup from GCS → Restore to Filesystem
-  - Automatically uses target storage's optimal batch configuration
-
-* **perf(api)**: Storage-aware batching for restore operations
-  - Leverages v4.10.4's storage-aware batching (10-100x faster on cloud storage)
-  - Automatic backpressure management prevents circuit breaker activation
-  - Separate read/write circuit breakers (backup can run during restore throttling)
-
-### 📊 What's Now Guaranteed
-
-| Feature | v4.10.4 | v4.11.0 |
-|---------|---------|---------|
-| Data Persists to Storage | ❌ No | ✅ Yes |
-| Data Survives Restart | ❌ No | ✅ Yes |
-| HNSW Index Updated | ❌ No | ✅ Yes |
-| Metadata Index Updated | ❌ No | ✅ Yes |
-| Searchable After Restore | ❌ No | ✅ Yes |
-| Progress Reporting | ❌ No | ✅ Yes |
-| Error Tracking | ❌ Silent | ✅ Detailed |
-| Cross-Storage Support | ❌ No | ✅ Yes |
-
-### 🔄 Migration Guide
-
-**No code changes required!** The fix is backward compatible:
-
-```typescript
-// Old code (still works)
-await brain.data().restore({ backup, overwrite: true })
-
-// New code (with progress tracking)
-const result = await brain.data().restore({
-  backup,
-  overwrite: true,
-  onProgress: (done, total) => {
-    console.log(`Restoring... ${done}/${total}`)
-  }
-})
-
-console.log(`✅ Restored ${result.entitiesRestored} entities`)
-if (result.errors.length > 0) {
-  console.warn(`⚠️ ${result.errors.length} failures`)
-}
-```
-
-### ⚠️ Breaking Changes (Minor API Change)
-
-* **DataAPI.restore()** return type changed from `Promise` to `Promise<{ entitiesRestored, relationshipsRestored, errors }>`
-  - Impact: Minimal - most code doesn't use the return value
-  - Fix: Remove explicit `Promise` type annotations if present
-
-### 📝 Files Modified
-
-* `src/api/DataAPI.ts` - Complete rewrite of restore() method (lines 161-338)
-
-### [4.10.4](https://github.com/soulcraftlabs/brainy/compare/v4.10.3...v4.10.4) (2025-10-30)
-
-* fix: prevent circuit breaker activation and data loss during bulk imports
-  - Storage-aware batching system prevents rate limiting on cloud storage (GCS, S3, R2, Azure)
-  - Separate read/write circuit breakers prevent read lockouts during write throttling
-  - ImportCoordinator uses addMany()/relateMany() for 10-100x performance improvement
-  - Fixes silent data loss and 30+ second lockouts on 1000+ row imports
-
-### [4.10.3](https://github.com/soulcraftlabs/brainy/compare/v4.10.2...v4.10.3) (2025-10-29)
-
-* fix: add atomic writes to ALL file operations to prevent concurrent write corruption
-
-### [4.10.2](https://github.com/soulcraftlabs/brainy/compare/v4.10.1...v4.10.2) (2025-10-29)
-
-* fix: VFS not initialized during Excel import, causing 0 files accessible
-
-### [4.10.1](https://github.com/soulcraftlabs/brainy/compare/v4.10.0...v4.10.1) (2025-10-29)
-
-- fix: add mutex locks to FileSystemStorage for HNSW concurrency (CRITICAL) (ff86e88)
-
-
-### [4.10.0](https://github.com/soulcraftlabs/brainy/compare/v4.9.2...v4.10.0) (2025-10-29)
-
-- perf: 48-64× faster HNSW bulk imports via concurrent neighbor updates (4038afd)
-
-
-### [4.9.2](https://github.com/soulcraftlabs/brainy/compare/v4.9.1...v4.9.2) (2025-10-29)
-
-- fix: resolve HNSW concurrency race condition across all storage adapters (0bcf50a)
-
-
-## [4.9.1](https://github.com/soulcraftlabs/brainy/compare/v4.9.0...v4.9.1) (2025-10-29)
-
-### 📚 Documentation
-
-* **vfs**: Fix NO FAKE CODE policy violations in VFS documentation
-  - **Removed**: 9 undocumented feature sections (~242 lines) from VFS docs
-    - Version History, Distributed Filesystem, AI Auto-Organization
-    - Security & Permissions, Smart Collections, Express.js middleware
-    - VSCode extension, Production Metrics, Backup & Recovery
-  - **Added**: Status labels (✅ Production, ⚠️ Beta, 🧪 Experimental) to all VFS features
-  - **Updated**: Performance claims with MEASURED vs PROJECTED labels
-  - **Created**: `docs/vfs/ROADMAP.md` for planned features (preserves vision without misleading)
-  - **Fixed**: Storage adapter list to show only 8 built-in adapters (removed Redis, PostgreSQL, ChromaDB)
-  - **Impact**: VFS documentation now 100% compliant with NO FAKE CODE policy
-
-### Files Modified
-- `docs/vfs/README.md`: Removed 9 fake feature sections, updated performance claims
-- `docs/vfs/SEMANTIC_VFS.md`: Added status labels, updated scale testing tables
-- `docs/vfs/VFS_API_GUIDE.md`: Fixed storage adapter compatibility list
-- `docs/vfs/ROADMAP.md`: New file organizing planned features by version
-
-## [4.9.0](https://github.com/soulcraftlabs/brainy/compare/v4.8.6...v4.9.0) (2025-10-28)
-
-**UNIVERSAL RELATIONSHIP EXTRACTION - Knowledge Graph Builder**
-
-This release transforms Brainy imports from entity extractors into true knowledge graph builders with full provenance tracking and semantic relationship enhancement.
-
-### ✨ Features
-
-* **import**: Universal relationship extraction with provenance tracking
-  - **Document Entity Creation**: Every import now creates a `document` entity representing the source file
-  - **Provenance Relationships**: Full data lineage with `document → entity` relationships for every imported entity
-  - **Relationship Type Metadata**: All relationships tagged as `vfs`, `semantic`, or `provenance` for filtering
-  - **Enhanced Column Detection**: 7 relationship types (vs 1 previously) - Location, Owner, Creator, Uses, Member, Friend, Related
-  - **Type-Based Inference**: Smart relationship classification based on entity types and context analysis
-  - **Impact**: A consumer import now creates ~3,900 relationships (vs 581), with 5-20+ connections per entity
-
-* **import**: New configuration option `createProvenanceLinks` (defaults to `true`)
-  - Enables/disables provenance relationship creation
-  - Backward compatible - all features opt-in
-
-### 📊 Impact
-
-**Before v4.9.0:**
-```
-Import: glossary.xlsx (1,149 rows)
-Result: 1,149 entities, 581 relationships (VFS only)
-Graph: Isolated nodes, 0 semantic connections
-```
-
-**After v4.9.0:**
-```
-Import: glossary.xlsx (1,149 rows)
-Result: 1,150 entities (+ document), ~3,900 relationships
-  - 1,149 provenance (document → entity)
-  - ~1,500 semantic (entity ↔ entity, diverse types)
-  - 581 VFS (directory structure, marked separately)
-Graph: Rich network, 5-20+ connections per entity
-```
-
-### 🔧 Technical Details
-
-* **Files Modified**: 3 files, 257 insertions(+), 11 deletions(-)
-  - `ImportCoordinator.ts`: +175 lines (document entity, provenance, inference)
-  - `SmartExcelImporter.ts`: +65 lines (enhanced column patterns)
-  - `VirtualFileSystem.ts`: +2 lines (relationship type metadata)
-
-* **Universal Support**: Works across ALL 7 import formats (Excel, PDF, CSV, JSON, Markdown, YAML, DOCX)
-* **Backward Compatible**: 100% - all features opt-in, existing imports unchanged
-
-### [4.8.6](https://github.com/soulcraftlabs/brainy/compare/v4.8.5...v4.8.6) (2025-10-28)
-
-- fix: per-sheet column detection in Excel importer (401443a)
-
-
-### [4.7.4](https://github.com/soulcraftlabs/brainy/compare/v4.7.3...v4.7.4) (2025-10-27)
-
-**CRITICAL SYSTEMIC VFS BUG FIX - A consumer team Unblocked!**
-
-This hotfix resolves a systemic bug affecting ALL storage adapters that caused VFS queries to return empty results even when data existed.
-
-#### 🐛 Critical Bug Fixes
-
-* **storage**: Fix systemic metadata skip bug across ALL 7 storage adapters
-  - **Impact**: VFS queries returned empty arrays despite 577 "Contains" relationships existing
-  - **Root Cause**: All storage adapters skipped entities if metadata file read returned null
-  - **Bug Pattern**: `if (!metadata) continue` in getNouns()/getVerbs() methods
-  - **Fixed Locations**: 12 bug sites across 7 adapters (TypeAware, Memory, FileSystem, GCS, S3, R2, OPFS, Azure)
-  - **Solution**: Allow optional metadata with `metadata: (metadata || {}) as NounMetadata`
-  - **Result**: a consumer team UNBLOCKED - VFS entities now queryable
-
-* **neural**: Fix SmartExtractor weighted score threshold bug (28 test failures → 4)
-  - **Root Cause**: Single signal with 0.8 confidence × 0.2 weight = 0.16 < 0.60 threshold
-  - **Solution**: Use original confidence when only one signal matches
-  - **Impact**: Entity type extraction now works correctly
-
-* **neural**: Fix PatternSignal priority ordering
-  - Specific patterns (organization "Inc", location "City, ST") now ranked higher than generic patterns
-  - Prevents person full-name pattern from overriding organization/location indicators
-
-* **api**: Fix Brainy.relate() weight parameter not returned in getRelations()
-  - **Root Cause**: Weight stored in metadata but read from wrong location
-  - **Solution**: Extract weight from metadata: `v.metadata?.weight ?? 1.0`
-
-#### 📊 Test Results
-
-- TypeAwareStorageAdapter: 17/17 tests passing (was 7 failures)
-- SmartExtractor: 42/46 tests passing (was 28 failures)
-- Neural domain clustering: 3/3 tests passing
-- Brainy.relate() weight: 1/1 test passing
-
-#### 🏗️ Architecture Notes
-
-**Two-Phase Fix**:
-1. Storage Layer (NOW FIXED): Returns ALL entities, even with empty metadata
-2. VFS Layer (ALREADY SAFE): PathResolver uses optional chaining `entity.metadata?.vfsType`
-
-**Result**: Valid VFS entities pass through, invalid entities safely filtered out.
-
-### [4.7.3](https://github.com/soulcraftlabs/brainy/compare/v4.7.2...v4.7.3) (2025-10-27)
-
-- fix(storage): CRITICAL - preserve vectors when updating HNSW connections (v4.7.3) (46e7482)
-
-
-### [4.4.0](https://github.com/soulcraftlabs/brainy/compare/v4.3.2...v4.4.0) (2025-10-24)
-
-- docs: update CHANGELOG for v4.4.0 release (a3c8a28)
-- docs: add VFS filtering examples to brain.find() JSDoc (d435593)
-- test: comprehensive tests for remaining APIs (17/17 passing) (f9e1bad)
-- fix: add includeVFS to initializeRoot() - prevents duplicate root creation (fbf2605)
-- fix: vfs.search() and vfs.findSimilar() now filter for VFS files only (0dda9dc)
-- test: add comprehensive API verification tests (21/25 passing) (ce8530b)
-- fix: wire up includeVFS parameter to ALL VFS-related APIs (6 critical bugs) (7582e3f)
-- test: fix brain.add() return type usage in VFS tests (970f243)
-- feat: brain.find() excludes VFS by default (Option 3C) (014b810)
-- test: update VFS where clause tests for correct field names (86f5956)
-- fix: VFS where clause field names + isVFS flag (f8d2d37)
-
-
-## [4.4.0](https://github.com/soulcraftlabs/brainy/compare/v4.3.2...v4.4.0) (2025-10-24)
-
-
-### 🎯 VFS Filtering Architecture (Option 3C)
-
-Clean separation between VFS (Virtual File System) entities and knowledge graph entities with opt-in inclusion.
-
-### ✨ Features
-
-* **brain.similar()**: add includeVFS parameter for VFS filtering consistency
-  - New `includeVFS` parameter in `SimilarParams` interface
-  - Passes through to `brain.find()` for consistent VFS filtering
-  - Excludes VFS entities by default, opt-in with `includeVFS: true`
-  - Enables clean knowledge similarity queries without VFS pollution
-
-### 🐛 Critical Bug Fixes
-
-* **vfs.initializeRoot()**: add includeVFS to prevent duplicate root creation
-  - **Critical Fix**: VFS init was creating ~10 duplicate root entities (a consumer team issue)
-  - **Root Cause**: `initializeRoot()` called `brain.find()` without `includeVFS: true`, never found existing VFS root
-  - **Impact**: Every `vfs.init()` created a new root, causing empty `readdir('/')` results
-  - **Solution**: Added `includeVFS: true` to root entity lookup (line 171)
-
-* **vfs.search()**: wire up includeVFS and add vfsType filter
-  - **Critical Fix**: `vfs.search()` returned 0 results after v4.3.3 VFS filtering
-  - **Root Cause**: Called `brain.find()` without `includeVFS: true`, excluded all VFS entities
-  - **Impact**: VFS semantic search completely broken
-  - **Solution**: Added `includeVFS: true` + `vfsType: 'file'` filter to return only VFS files
-
-* **vfs.findSimilar()**: wire up includeVFS and add vfsType filter
-  - **Critical Fix**: `vfs.findSimilar()` returned 0 results or mixed knowledge entities
-  - **Root Cause**: Called `brain.similar()` without `includeVFS: true` or vfsType filter
-  - **Impact**: VFS similarity search broken, could return knowledge docs without .path property
-  - **Solution**: Added `includeVFS: true` + `vfsType: 'file'` filter
-
-* **vfs.searchEntities()**: add includeVFS parameter
-  - Added `includeVFS: true` to ensure VFS entity search works correctly
-
-* **VFS semantic projections**: fix all 3 projection classes
-  - **TagProjection**: Fixed 3 `brain.find()` calls with `includeVFS: true`
-  - **AuthorProjection**: Fixed 2 `brain.find()` calls with `includeVFS: true`
-  - **TemporalProjection**: Fixed 2 `brain.find()` calls with `includeVFS: true`
-  - **Impact**: VFS semantic views (/by-tag, /by-author, /by-date) were empty
-
-### 📝 Documentation
-
-* **JSDoc**: Added VFS filtering examples to `brain.find()` with 3 usage patterns
-* **Inline comments**: Documented VFS filtering architecture at all usage sites
-* **Code comments**: Explained critical bug fixes inline for maintainability
-
-### ✅ Testing
-
-* **45/49 APIs tested** (92% coverage) with 46 new integration tests
-* **952/1005 tests passing** (95% pass rate) - all v4.4.0 changes verified
-* Comprehensive tests for:
-  - brain.updateMany() - Batch metadata updates with merging
-  - brain.import() - CSV import with VFS integration
-  - vfs file operations (unlink, rmdir, rename, copy, move)
-  - neural.clusters() - Semantic clustering with VFS filtering
-  - Production scale verified (100 entities, 50 batch updates, 20 VFS files)
-
-### 🏗️ Architecture
-
-* **Option 3C**: VFS entities in graph with `isVFS` flag for clean separation
-* **Default behavior**: `brain.find()` and `brain.similar()` exclude VFS by default
-* **Opt-in inclusion**: Use `includeVFS: true` parameter to include VFS entities
-* **VFS APIs**: Automatically filter for VFS-only (never return knowledge entities)
-* **Cross-boundary relationships**: Link VFS files to knowledge entities with `brain.relate()`
-
-### 🔍 API Behavior
-
-**Before v4.4.0:**
-```javascript
-const results = await brain.find({ query: 'documentation' })
-// Returned mixed knowledge + VFS files (confusing, polluted results)
-```
-
-**After v4.4.0:**
-```javascript
-// Clean knowledge queries (VFS excluded by default)
-const knowledge = await brain.find({ query: 'documentation' })
-// Returns only knowledge entities
-
-// Opt-in to include VFS
-const everything = await brain.find({
-  query: 'documentation',
-  includeVFS: true
-})
-// Returns knowledge + VFS files
-
-// VFS-only search
-const files = await vfs.search('documentation')
-// Returns only VFS files (automatic filtering)
-```
-
-### 🎓 Migration Notes
-
-**No breaking changes** - All existing code continues to work:
-- Existing `brain.find()` queries get cleaner results (VFS excluded)
-- VFS APIs now work correctly (bugs fixed)
-- Add `includeVFS: true` only if you need VFS entities in knowledge queries
-
-### [4.2.4](https://github.com/soulcraftlabs/brainy/compare/v4.2.3...v4.2.4) (2025-10-23)
-
-
-### ⚡ Performance Improvements
-
-* **all-indexes**: extend adaptive loading to HNSW and Graph indexes for complete cold start optimization
-  - **Issue**: v4.2.3 only optimized MetadataIndex - HNSW and Graph indexes still used fixed pagination (1000 items/batch)
-  - **Root Cause**: HNSW `rebuild()` and Graph `rebuild()` methods still called `getNounsWithPagination()`/`getVerbsWithPagination()` repeatedly
-    - Each pagination call triggered `getAllShardedFiles()` reading all 256 shard directories
-    - For 1,157 entities: MetadataIndex (2-3s) + HNSW (~20s) + Graph (~10s) = **30-35 seconds total**
-    - a consumer team reported: "v4.2.3 is at batch 7 after ~60 seconds" - still far from claimed 100x improvement
-  - **Solution**: Apply v4.2.3 adaptive loading pattern to ALL 3 indexes
-    - **FileSystemStorage/MemoryStorage/OPFSStorage**: Load all entities at once (limit: 10000000)
-    - **Cloud storage (GCS/S3/R2/Azure)**: Keep pagination (native APIs are efficient)
-    - Detection: Auto-detect storage type via `constructor.name`
-  - **Performance Impact**:
-    - **FileSystem Cold Start**: 30-35 seconds → **6-9 seconds** (5x faster than v4.2.3)
-    - **Complete Fix**: MetadataIndex (2-3s) + HNSW (2-3s) + Graph (2-3s) = 6-9 seconds total
-    - **From v4.2.0**: 8-9 minutes → 6-9 seconds (**60-90x faster overall**)
-    - Directory scans: 3 indexes × multiple batches → 3 indexes × 1 scan each
-    - Cloud storage: No regression (pagination still efficient with native APIs)
-  - **Benefits**:
-    - Eliminates pagination overhead for local storage completely
-    - One `getAllShardedFiles()` call per index instead of multiple
-    - FileSystem/Memory/OPFS can handle thousands of entities in single load
-    - Cloud storage unaffected (already efficient with continuation tokens)
-  - **Technical Details**:
-    - HNSW Index: Loads all nodes at once for local, paginated for cloud (lines 858-1010)
-    - Graph Index: Loads all verbs at once for local, paginated for cloud (lines 300-361)
-    - Pattern matches v4.2.3 MetadataIndex implementation exactly
-    - Zero config: Completely automatic based on storage adapter type
-  - **Resolution**: Fully resolves a consumer team's v4.2.x performance regression
-  - **Files Changed**:
-    - `src/hnsw/hnswIndex.ts` (updated rebuild() with adaptive loading)
-    - `src/graph/graphAdjacencyIndex.ts` (updated rebuild() with adaptive loading)
-
-### [4.2.3](https://github.com/soulcraftlabs/brainy/compare/v4.2.2...v4.2.3) (2025-10-23)
-
-
-### 🐛 Bug Fixes
-
-* **metadata-index**: fix rebuild stalling after first batch on FileSystemStorage
-  - **Critical Fix**: v4.2.2 rebuild stalled after processing first batch (500/1,157 entities)
-  - **Root Cause**: `getAllShardedFiles()` was called on EVERY batch, re-reading all 256 shard directories each time
-  - **Performance Impact**: Second batch call to `getAllShardedFiles()` took 3+ minutes, appearing to hang
-  - **Solution**: Load all entities at once for local storage (FileSystem/Memory/OPFS)
-    - FileSystem/Memory/OPFS: Load all nouns/verbs in single batch (no pagination overhead)
-    - Cloud (GCS/S3/R2): Keep conservative pagination (25 items/batch for socket safety)
-  - **Benefits**:
-    - FileSystem: 1,157 entities load in **2-3 seconds** (one `getAllShardedFiles()` call)
-    - Cloud: Unchanged behavior (still uses safe batching)
-    - Zero config: Auto-detects storage type via `constructor.name`
-  - **Technical Details**:
-    - Pagination was designed for cloud storage socket exhaustion
-    - FileSystem doesn't need pagination - can handle loading thousands of entities at once
-    - Eliminates repeated directory scans: 3 batches × 256 dirs → 1 batch × 256 dirs
-  - **A consumer team**: This resolves the v4.2.2 stalling issue - rebuild will now complete in seconds
-  - **Files Changed**: `src/utils/metadataIndex.ts` (rebuilt() method with adaptive loading strategy)
-
-### [4.2.2](https://github.com/soulcraftlabs/brainy/compare/v4.2.1...v4.2.2) (2025-10-23)
-
-
-### ⚡ Performance Improvements
-
-* **metadata-index**: implement adaptive batch sizing for first-run rebuilds
-  - **Issue**: v4.2.1 field registry only helps on 2nd+ runs - first run still slow (8-9 min for 1,157 entities)
-  - **Root Cause**: Batch size of 25 was designed for cloud storage socket exhaustion, too conservative for local storage
-  - **Solution**: Adaptive batch sizing based on storage adapter type
-    - **FileSystemStorage/MemoryStorage/OPFSStorage**: 500 items/batch (fast local I/O, no socket limits)
-    - **GCS/S3/R2 (cloud storage)**: 25 items/batch (prevent socket exhaustion)
-  - **Performance Impact**:
-    - FileSystem first-run rebuild: 8-9 min → **30-60 seconds** (10-15x faster)
-    - 1,157 entities: 46 batches @ 25 → 3 batches @ 500 (15x fewer I/O operations)
-    - Cloud storage: No change (still 25/batch for safety)
-  - **Detection**: Auto-detects storage type via `constructor.name`
-  - **Zero Config**: Completely automatic, no configuration needed
-  - **Combined with v4.2.1**: First run fast, subsequent runs instant (2-3 sec)
-  - **Files Changed**: `src/utils/metadataIndex.ts` (updated rebuild() with adaptive batch sizing)
-
-### [4.2.1](https://github.com/soulcraftlabs/brainy/compare/v4.2.0...v4.2.1) (2025-10-23)
-
-
-### 🐛 Bug Fixes
-
-* **performance**: persist metadata field registry for instant cold starts
-  - **Critical Fix**: Metadata index rebuild now takes 2-3 seconds instead of 8-9 minutes for 1,157 entities
-  - **Root Cause**: `fieldIndexes` Map not persisted - caused unnecessary rebuilds even when sparse indices existed on disk
-  - **Discovery Problem**: `getStats()` checked empty in-memory Map → returned `totalEntries = 0` → triggered full rebuild
-  - **Solution**: Persist field directory as `__metadata_field_registry__` (same pattern as HNSW system metadata)
-    - Save registry during flush (automatic, ~4-8KB file)
-    - Load registry on init (O(1) discovery of persisted fields)
-    - Populate fieldIndexes Map → getStats() finds indices → skips rebuild
-  - **Performance**:
-    - Cold start: 8-9 min → 2-3 sec (100x faster)
-    - Works for 100 to 1B entities (field count grows logarithmically)
-    - Universal: All storage adapters (FileSystem, GCS, S3, R2, Memory, OPFS)
-  - **Zero Config**: Completely automatic, no configuration needed
-  - **Self-Healing**: Gracefully handles missing/corrupt registry (rebuilds once)
-  - **Impact**: Fixes a consumer team bug report - production-ready at billion scale
-  - **Files Changed**: `src/utils/metadataIndex.ts` (added saveFieldRegistry/loadFieldRegistry methods, updated init/flush)
-
-### [4.2.0](https://github.com/soulcraftlabs/brainy/compare/v4.1.4...v4.2.0) (2025-10-23)
-
-
-### ✨ Features
-
-* **import**: implement progressive flush intervals for streaming imports
-  - Dynamically adjusts flush frequency based on current entity count (not total)
-  - Starts at 100 entities for frequent early updates, scales to 5000 for large imports
-  - Works for both known totals (files) and unknown totals (streaming APIs)
-  - Provides live query access during imports and crash resilience
-  - Zero configuration required - always-on streaming architecture
-  - Updated documentation with engineering insights and usage examples
-
-### [4.1.4](https://github.com/soulcraftlabs/brainy/compare/v4.1.3...v4.1.4) (2025-10-21)
-
-- feat: add import API validation and v4.x migration guide (a1a0576)
-
-
-### [4.1.3](https://github.com/soulcraftlabs/brainy/compare/v4.1.2...v4.1.3) (2025-10-21)
-
-- perf: make getRelations() pagination consistent and efficient (54d819c)
-- fix: resolve getRelations() empty array bug and add string ID shorthand (8d217f3)
-
-
-### [4.1.3](https://github.com/soulcraftlabs/brainy/compare/v4.1.2...v4.1.3) (2025-10-21)
-
-
-### 🐛 Bug Fixes
-
-* **api**: fix getRelations() returning empty array when called without parameters
-  - Fixed critical bug where `brain.getRelations()` returned `[]` instead of all relationships
-  - Added support for retrieving all relationships with pagination (default limit: 100)
-  - Added string ID shorthand syntax: `brain.getRelations(entityId)` as alias for `brain.getRelations({ from: entityId })`
-  - **Performance**: Made pagination consistent - now ALL query patterns paginate at storage layer
-  - **Efficiency**: `getRelations({ from: id, limit: 10 })` now fetches only 10 instead of fetching ALL then slicing
-  - Fixed storage.getVerbs() offset handling - now properly converts offset to cursor for adapters
-  - Production safety: Warns when fetching >10k relationships without filters
-  - Fixed broken method calls in improvedNeuralAPI.ts (replaced non-existent `getVerbsForNoun` with `getRelations`)
-  - Fixed property access bugs: `verb.target` → `verb.to`, `verb.verb` → `verb.type`
-  - Added comprehensive integration tests (14 tests covering all query patterns)
-  - Updated JSDoc documentation with usage examples
-  - **Impact**: Resolves a consumer team bug where 524 imported relationships were inaccessible
-  - **Breaking**: None - fully backward compatible
-
-### [4.1.2](https://github.com/soulcraftlabs/brainy/compare/v4.1.1...v4.1.2) (2025-10-21)
-
-
-### 🐛 Bug Fixes
-
-* **storage**: resolve count synchronization race condition across all storage adapters ([798a694](https://github.com/soulcraftlabs/brainy/commit/798a694))
-  - Fixed critical bug where entity and relationship counts were not tracked correctly during add(), relate(), and import()
-  - Root cause: Race condition where count increment tried to read metadata before it was saved
-  - Fixed in baseStorage for all storage adapters (FileSystem, GCS, R2, Azure, Memory, OPFS, S3, TypeAware)
-  - Added verb type to VerbMetadata for proper count tracking
-  - Refactored verb count methods to prevent mutex deadlocks
-  - Added rebuildCounts utility to repair corrupted counts from actual storage data
-  - Added comprehensive integration tests (11 tests covering all operations)
-
-### [4.1.1](https://github.com/soulcraftlabs/brainy/compare/v4.1.0...v4.1.1) (2025-10-20)
-
-
-### 🐛 Bug Fixes
-
-* correct Node.js version references from 24 to 22 in comments and code ([22513ff](https://github.com/soulcraftlabs/brainy/commit/22513ffcb40cc6498898400ac5d1bae19c5d02ed))
-
-## [4.1.0](https://github.com/soulcraftlabs/brainy/compare/v4.0.1...v4.1.0) (2025-10-20)
-
-
-### 📚 Documentation
-
-* restructure README for clarity and engagement ([26c5c78](https://github.com/soulcraftlabs/brainy/commit/26c5c784293293e2d922e0822b553b860262af1c))
-
-
-### ✨ Features
-
-* simplify GCS storage naming and add Cloud Run deployment options ([38343c0](https://github.com/soulcraftlabs/brainy/commit/38343c012846f0bdf70dc7402be0ef7ad93d7179))
-
-## [4.0.0](https://github.com/soulcraftlabs/brainy/compare/v3.50.2...v4.0.0) (2025-10-17)
-
-### 🎉 Major Release - Cost Optimization & Enterprise Features
-
-**v4.0.0 focuses on production cost optimization and enterprise-scale features**
-
-### ✨ Features
-
-#### 💰 Cloud Storage Cost Optimization (Up to 96% Savings)
-
-**Lifecycle Management** (GCS, S3, Azure):
-- Automatic tier transitions based on age or access patterns
-- Delete policies for aged data
-- GCS Autoclass for fully automatic optimization (94% savings!)
-- AWS S3 Intelligent-Tiering for automatic cost reduction
-- Interactive CLI policy builder with provider-specific guides
-- Cost savings estimation tool
-
-**Cost Impact @ Scale**:
-```
-Small (5TB):   $1,380/year → $59/year    (96% savings = $1,321/year)
-Medium (50TB): $13,800/year → $594/year  (96% savings = $13,206/year)
-Large (500TB): $138,000/year → $5,940/year (96% savings = $132,060/year)
-```
-
-**CLI Commands**:
-```bash
-# Interactive lifecycle policy builder
-$ brainy storage lifecycle set
-? Choose optimization strategy:
-  🎯 Intelligent-Tiering (Recommended - Automatic)
-  📅 Lifecycle Policies (Manual tier transitions)
-  🚀 Aggressive Archival (Maximum savings)
-
-# Cost estimation tool
-$ brainy storage cost-estimate
-💰 Estimated Annual Savings: $132,060/year (96%)
-```
-
-#### ⚡ High-Performance Batch Operations
-
-**Batch Delete**:
-- S3: Uses DeleteObjects API (1000 objects/request)
-- Azure: Uses Batch API
-- GCS: Batch operations support
-- **1000x faster** than serial deletion
-- Performance: **533 entities/sec** (was 0.5/sec)
-- Automatic retry with exponential backoff
-- CLI integration with progress tracking
-
-**Example**:
-```bash
-$ brainy storage batch-delete entities.txt
-✓ Deleted 5000 entities in 9.4s (533/sec)
-```
-
-#### 📦 FileSystem Compression
-
-**Gzip Compression**:
-- 60-80% space savings
-- Transparent compression/decompression
-- CLI commands: `enable`, `disable`, `status`
-- Only for FileSystem storage (not cloud)
-
-**Example**:
-```bash
-$ brainy storage compression enable
-✓ Compression enabled!
-  Expected space savings: 60-80%
-```
-
-#### 📊 Quota Monitoring
-
-**Storage Status**:
-- Health checks for all providers
-- Quota tracking (OPFS, all providers)
-- Usage percentage with color-coded warnings
-- Provider-specific details (bucket, region, path)
-
-**Example**:
-```bash
-$ brainy storage status --quota
-📊 Quota Information
-
-Metric  Value
-Usage   45.2 GB
-Quota   100 GB
-Used    45.2%
-```
-
-#### 🎨 Enhanced CLI System (47 Commands)
-
-**Storage Management** (9 commands):
-- `brainy storage status` - Health and quota monitoring
-- `brainy storage lifecycle set/get/remove` - Lifecycle policy management
-- `brainy storage compression enable/disable/status` - Compression management
-- `brainy storage batch-delete` - High-performance batch deletion
-- `brainy storage cost-estimate` - Interactive cost calculator
-
-**Enhanced Import** (2 commands):
-- `brainy import` - Universal neural import
-  - Supports files, directories, URLs
-  - All formats: JSON, CSV, JSONL, YAML, Markdown, HTML, XML, text
-  - Neural features: concept extraction, entity extraction, relationship detection
-  - Progress tracking for large imports
-- `brainy vfs import` - VFS directory import
-  - Recursive directory imports
-  - Automatic embedding generation
-  - Metadata extraction
-  - Batch processing (100 files/batch)
-
-**Example**:
-```bash
-$ brainy import ./research-papers --extract-concepts --progress
-✓ Found 150 files
-✓ Extracted 237 concepts
-✓ Extracted 89 named entities
-✓ Neural import complete with AI type matching
-```
-
-### 🏗️ Implementation
-
-**Storage Adapters**:
-- `src/storage/adapters/gcsStorage.ts` (lines 1892-2175) - Lifecycle + Autoclass
-- `src/storage/adapters/s3CompatibleStorage.ts` (lines 4058-4237) - Lifecycle + Batch
-- `src/storage/adapters/azureBlobStorage.ts` (lines 2038-2292) - Lifecycle + Batch
-- All adapters: `getStorageStatus()` for quota monitoring
-
-**CLI**:
-- `src/cli/commands/storage.ts` (842 lines) - 9 storage commands
-- `src/cli/commands/import.ts` (592 lines) - 2 enhanced import commands
-
-### 📚 Documentation
-
-- `docs/MIGRATION-V3-TO-V4.md` - Complete migration guide
-- `.strategy/V4_READINESS_REPORT.md` - Implementation summary
-- `.strategy/ENHANCED_IMPORT_COMPLETE.md` - Import system documentation
-- `.strategy/PRODUCTION_CLI_COMPLETE.md` - CLI documentation
-- All CLI commands have interactive help
-
-### 🎯 Enterprise Ready
-
-**Cost Savings**:
-- Up to 96% storage cost reduction with lifecycle policies
-- Automatic optimization with GCS Autoclass
-- Provider-specific optimization strategies
-- Interactive cost estimation tool
-
-**Performance**:
-- 1000x faster batch deletions (533 entities/sec)
-- Optimized for billions of entities
-- Production-tested at scale
-
-**Developer Experience**:
-- Interactive CLI for all operations
-- Beautiful terminal UI with tables, spinners, colors
-- JSON output for automation (`--json`, `--pretty`)
-- Comprehensive error handling with helpful messages
-- Provider-specific guides (AWS/GCS/Azure/R2)
-
-### ⚠️ Breaking Changes
-
-#### 💥 Import API Redesign
-
-The import API has been redesigned for clarity and better feature control. **Old v3.x option names are no longer recognized** and will throw errors.
-
-**What Changed:**
-
-| v3.x Option | v4.x Option | Action Required |
-|-------------|-------------|-----------------|
-| `extractRelationships` | `enableRelationshipInference` | **Rename option** |
-| `autoDetect` | *(removed)* | **Delete option** (always enabled) |
-| `createFileStructure` | `vfsPath` | **Replace** with VFS path |
-| `excelSheets` | *(removed)* | **Delete option** (all sheets processed) |
-| `pdfExtractTables` | *(removed)* | **Delete option** (always enabled) |
-| - | `enableNeuralExtraction` | **Add option** (new in v4.x) |
-| - | `enableConceptExtraction` | **Add option** (new in v4.x) |
-| - | `preserveSource` | **Add option** (new in v4.x) |
-
-**Why These Changes?**
-
-1. **Clearer option names**: `enableRelationshipInference` explicitly indicates AI-powered relationship inference
-2. **Separation of concerns**: Neural extraction, relationship inference, and VFS are now separate, explicit options
-3. **Better defaults**: Auto-detection and AI features are enabled by default
-4. **Reduced confusion**: Removed redundant options like `autoDetect` and format-specific options
-
-**Migration Examples:**
-
-
-Example 1: Basic Excel Import - -```typescript -// v3.x (OLD - Will throw error) -await brain.import('./glossary.xlsx', { - extractRelationships: true, - createFileStructure: true -}) - -// v4.x (NEW - Use this) -await brain.import('./glossary.xlsx', { - enableRelationshipInference: true, - vfsPath: '/imports/glossary' -}) -``` -
- -
-Example 2: Full-Featured Import - -```typescript -// v3.x (OLD - Will throw error) -await brain.import('./data.xlsx', { - extractRelationships: true, - autoDetect: true, - createFileStructure: true -}) - -// v4.x (NEW - Use this) -await brain.import('./data.xlsx', { - enableNeuralExtraction: true, // Extract entity names - enableRelationshipInference: true, // Infer semantic relationships - enableConceptExtraction: true, // Extract entity types - vfsPath: '/imports/data', // VFS directory - preserveSource: true // Save original file -}) -``` -
- -**Error Messages:** - -If you use old v3.x options, you'll get a clear error message: - -``` -❌ Invalid import options detected (Brainy v4.x breaking changes) - -The following v3.x options are no longer supported: - - ❌ extractRelationships - → Use: enableRelationshipInference - → Why: Option renamed for clarity in v4.x - -📖 Migration Guide: https://brainy.dev/docs/guides/migrating-to-v4 -``` - -**Other v4.0.0 Features (Non-Breaking):** - -All other v4.0.0 features are: -- ✅ Opt-in (lifecycle, compression, batch operations) -- ✅ Additive (new CLI commands, new methods) -- ✅ Non-breaking (existing code continues to work) - -### 📝 Migration - -**Import API migration required** if you use `brain.import()` with the old v3.x option names. - -#### Required Changes: -1. Update to v4.0.0: `npm install @soulcraft/brainy@4.0.0` -2. Update import calls to use new option names (see table above) -3. Test your imports - you'll get clear error messages if you use old options - -#### Optional Enhancements: -- Enable lifecycle policies: `brainy storage lifecycle set` -- Use batch operations: `brainy storage batch-delete entities.txt` -- See full migration guide: `docs/guides/migrating-to-v4.md` - -**Complete Migration Guide:** [docs/guides/migrating-to-v4.md](./docs/guides/migrating-to-v4.md) - -### 🎓 What This Means - -**For Users**: -- Massive cost savings (up to 96%) with automatic tier management -- 1000x faster batch operations for large-scale cleanups -- Complete CLI tooling for all enterprise operations -- Neural import system with AI-powered type matching - -**For Developers**: -- Production-ready code with zero fake implementations -- Complete TypeScript type safety -- Comprehensive error handling -- Beautiful interactive UX - -**For Brainy**: -- Enterprise-grade cost optimization -- World-class CLI experience -- Production-ready at billion-scale -- Sets standard for database tooling - ---- +All notable changes to this project will be documented in this file. See [standard-version](https://github.com/soulcraftlabs/standard-version) for commit guidelines. ### [3.50.2](https://github.com/soulcraftlabs/brainy/compare/v3.50.1...v3.50.2) (2025-10-16) @@ -3496,11 +85,11 @@ After upgrading to v3.50.2: ### ✨ Features -**Phase 2: Type-Aware HNSW - PROJECTED 87% Memory Reduction @ Billion Scale** +**Phase 2: Type-Aware HNSW - 87% Memory Reduction @ Billion Scale** - **feat**: TypeAwareHNSWIndex with separate HNSW graphs per entity type - - **PROJECTED 87% HNSW memory reduction**: 384GB → 50GB (-334GB) @ 1B scale (calculated from architectural analysis, not yet benchmarked at billion scale) - - **PROJECTED 10x faster single-type queries**: search 100M nodes instead of 1B (not yet benchmarked) + - **87% HNSW memory reduction**: 384GB → 50GB (-334GB) @ 1B scale + - **10x faster single-type queries**: search 100M nodes instead of 1B - **5-8x faster multi-type queries**: search subset of types - **~3x faster all-types queries**: 31 smaller graphs vs 1 large graph - Lazy initialization - only creates indexes for types with entities @@ -3518,11 +107,11 @@ After upgrading to v3.50.2: - Maintains O(log n) performance guarantees - Zero API changes for existing code -### 📊 Impact @ Billion Scale (PROJECTED) +### 📊 Impact @ Billion Scale -**Memory Reduction (Phase 2) - PROJECTED:** +**Memory Reduction (Phase 2):** ``` -HNSW memory: 384GB → 50GB (-87% / -334GB) - PROJECTED from architectural analysis, not benchmarked at 1B scale +HNSW memory: 384GB → 50GB (-87% / -334GB) ``` **Query Performance:** @@ -3559,18 +148,17 @@ Background mode: 0 seconds perceived startup Part of the billion-scale optimization roadmap: - **Phase 0**: Type system foundation (v3.45.0) ✅ - **Phase 1a**: TypeAwareStorageAdapter (v3.45.0) ✅ -- **Phase 1b**: MetadataIndex Uint32Array tracking (v3.46.0) ✅ +- **Phase 1b**: TypeFirstMetadataIndex (v3.46.0) ✅ - **Phase 1c**: Enhanced Brainy API (v3.46.0) ✅ - **Phase 2**: Type-Aware HNSW (v3.47.0) ✅ **← COMPLETED** -- **Phase 3**: Type-First Query Optimization (planned - PROJECTED 40% latency reduction) +- **Phase 3**: Type-First Query Optimization (planned - 40% latency reduction) -**Cumulative Impact (Phases 0-2) - MEASURED up to 1M entities:** -- Memory: MEASURED -87% for HNSW (Phase 2 tests), -99.2% for type count tracking (Phase 1b) -- Query Speed: MEASURED 10x faster for type-specific queries (typeAwareHNSW.integration.test.ts) -- Rebuild Speed: MEASURED 31x faster with type filtering (test results) -- Cache Performance: MEASURED +25% hit rate improvement +**Cumulative Impact (Phases 0-2):** +- Memory: -87% for HNSW, -99.2% for type tracking +- Query Speed: 10x faster for type-specific queries +- Rebuild Speed: 31x faster with type filtering +- Cache Performance: +25% hit rate improvement - Backward Compatibility: 100% (zero breaking changes) -- Note: Billion-scale claims are PROJECTIONS (not tested at 1B scale) ### 📝 Files Changed @@ -3588,7 +176,7 @@ Part of the billion-scale optimization roadmap: ### 🎯 Next Steps **Phase 3** (planned): Type-First Query Optimization -- Query: PROJECTED 40% latency reduction via type-aware planning (not yet benchmarked) +- Query: 40% latency reduction via type-aware planning - Index: Smart query routing based on type cardinality - Estimated: 2 weeks implementation @@ -3598,11 +186,11 @@ Part of the billion-scale optimization roadmap: ### ✨ Features -**Phase 1b: MetadataIndexManager - 99.2% Memory Reduction for Type Count Tracking** +**Phase 1b: TypeFirstMetadataIndex - 99.2% Memory Reduction for Type Tracking** - **feat**: Enhanced MetadataIndexManager with Uint32Array type tracking (ddb9f04) - - Fixed-size type tracking: 31 noun types + 40 verb types = 284 bytes (was ~35KB Map) - - **99.2% memory reduction** for type count tracking ONLY (not total index memory) + - Fixed-size type tracking: 31 noun types + 40 verb types = 284 bytes (was ~35KB) + - **99.2% memory reduction** for type count tracking - 6 new O(1) type enum methods for faster type-specific queries - Bidirectional sync between Maps ↔ Uint32Arrays for backward compatibility - Type-aware cache warming: preloads top 3 types + their top 5 fields on init @@ -3654,10 +242,10 @@ Top types query: O(31 × 1B) → O(31) iteration (1B x faster) Part of the billion-scale optimization roadmap: - **Phase 0**: Type system foundation (v3.45.0) ✅ - **Phase 1a**: TypeAwareStorageAdapter (v3.45.0) ✅ -- **Phase 1b**: MetadataIndex Uint32Array tracking (v3.46.0) ✅ +- **Phase 1b**: TypeFirstMetadataIndex (v3.46.0) ✅ - **Phase 1c**: Enhanced Brainy API (v3.46.0) ✅ -- **Phase 2**: Type-Aware HNSW (planned - PROJECTED 87% HNSW memory reduction) -- **Phase 3**: Type-First Query Optimization (planned - PROJECTED 40% latency reduction) +- **Phase 2**: Type-Aware HNSW (planned - 87% HNSW memory reduction) +- **Phase 3**: Type-First Query Optimization (planned - 40% latency reduction) **Cumulative Impact (Phases 0-1c):** - Memory: -99.2% for type tracking diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 568c10db..00000000 --- a/CLAUDE.md +++ /dev/null @@ -1,208 +0,0 @@ -# Brainy - Claude Code Project Guide - -This file provides guidance for Claude Code (and human contributors) when working on the Brainy codebase. - -## Cross-Project Coordination - -Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md` - -**At session START:** Read the handoff. Find rows where Owner = Brainy. Act on those first. - -**At session END:** Mark completed actions ✅, delete rows you finished, delete threads with zero remaining actions. File must not grow. **If you shipped anything consumers need to know about, update `RELEASES.md` before closing.** - -**Brainy's current open actions:** None. MIT open-source — no platform-specific actions. - -**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`) - ---- - -## Project Overview - -Brainy is a Universal Knowledge Protocol -- a Triple Intelligence database that combines vector similarity search, graph traversal, and metadata filtering into a single TypeScript library. Published as `@soulcraft/brainy` on npm under the MIT license. - -## Getting Started - -```bash -npm install # Install dependencies -npm run build # Build the project -npm test # Run test suite (Vitest) -``` - -## Architecture - -Full architecture reference: `.claude/skills/architecture.md` - -### Core Systems -- **Storage** (`src/storage/`): Pluggable storage backends via StorageAdapter interface (`src/coreTypes.ts`) -- **Vector Search** (`src/hnsw/`): HNSW approximate nearest neighbor search -- **Graph Engine** (`src/graph/`): Relationship traversal with adjacency index and pathfinding -- **Metadata Index** (`src/utils/metadataIndex.ts`): O(1) exact match, O(log n) range queries -- **Triple Intelligence** (`src/triple/`): Unified query combining all three intelligence types -- **Aggregation Engine** (`src/aggregation/`): Write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows -- **Virtual Filesystem** (`src/vfs/`): Full VFS with semantic search - -### Type System -- **NounType** (42 types): Entity classification -- Person, Concept, Collection, Document, Task, etc. -- **VerbType** (127 types): Relationship types -- Contains, RelatedTo, PartOf, Creates, DependsOn, etc. -- Defined in `src/types/graphTypes.ts` - -## Code Standards - -### TypeScript -- Strict mode enabled -- Target: ES2020, NodeNext module resolution -- All new code must be TypeScript -- Follow existing patterns -- read related code before writing - -### Quality -- All code must compile without errors -- All code must have working tests that exercise real behavior -- No stub returns (`return {} as any`) -- No incomplete implementations with TODO comments -- If something can't be fully implemented, throw an explicit error rather than faking it - -### Verification Before Code Changes -1. Check that interfaces and methods actually exist before using them -2. Check that type properties are in the type definitions -3. Run `npm test` -- tests must pass -4. Run `npm run build` -- build must succeed - -### Testing -- Framework: Vitest -- Tests in `tests/` (unit, integration, benchmarks, comprehensive) -- Use in-memory storage for speed where possible -- Tests must exercise real behavior, not mock it -- Benchmarks are in `tests/benchmarks/` (not tests/performance/) - -## Commit Conventions - -Use [Conventional Commits](https://www.conventionalcommits.org/): - -``` -feat: add new feature (minor version bump) -fix: resolve bug (patch version bump) -docs: update documentation (patch version bump) -perf: improve performance (patch version bump) -refactor: restructure code (patch version bump) -test: add/update tests (patch version bump) -``` - -**Important:** Never use `BREAKING CHANGE` in commit messages. Major version bumps are manual decisions only (`npm run release:major`). - -## Docs Pipeline — soulcraft.com/docs - -Docs in `docs/**/*.md` are published with the npm package (included in `files`) and synced to soulcraft.com/docs on every portal deploy. Frontmatter controls what appears publicly. - -### Docs check triggers - -Run the docs check whenever the user says ANY of: -- "commit, publish, release" / "release" / "publish" -- "update the docs" / "make sure docs are accurate" / "check the docs" -- "review docs" / "clean up docs" - -### Pre-release docs check (MANDATORY before every release) - -When the user says "commit, publish, release" or any variation, **before committing**: - -1. **Scan all files changed in this session** (and any recently added `docs/*.md` files) -2. For each changed/new doc, decide: is this useful to external users? - - **Yes** → ensure it has complete frontmatter (add or update it) - - **No** (internal, migration, dev-only) → ensure it has no frontmatter or `public: false` -3. For docs that already have frontmatter, verify: - - `description` still matches the actual content - - `next` links still exist and are still the right follow-up pages - - `title` matches the doc's h1 -4. Include frontmatter changes in the commit - -### Frontmatter format - -```yaml ---- -title: Human-readable title -slug: category/page-name # URL: soulcraft.com/docs/category/page-name -public: true # false or absent = not published -category: getting-started | concepts | guides | api -template: guide | concept | api # controls layout on soulcraft.com -order: 1 # sidebar position within category (lower = first) -description: One sentence. What this doc covers and why it matters. -next: # "Next steps" links shown at bottom of page - - category/other-slug ---- -``` - -### Category guide - -| category | use for | -|----------|---------| -| `getting-started` | installation, quick start, first steps | -| `concepts` | how the system works, mental models | -| `guides` | how to do specific things, recipes | -| `api` | method reference, signatures, parameters | - -### What stays internal (no frontmatter / `public: false`) - -- Release guides, developer learning paths -- Migration guides for old versions (v3→v4, v5.11) -- Architecture analysis docs (clustering algorithms, etc.) -- Anything in `docs/internal/` -- Deployment/ops/cost docs (cloud-run, kubernetes, cost-optimization) - -## Release Process - -Fully automated via `scripts/release.sh`: - -```bash -npm run release:dry # Preview (no changes) -npm run release:patch # Bug fixes -npm run release:minor # New features -npm run release:major # Breaking changes (rare, manual decision) -``` - -The script: verifies clean git state, builds, tests, bumps version, updates CHANGELOG.md, commits, tags, pushes, publishes to npm, and creates a GitHub release. - -After a successful release, remind the user: -> "Published. Deploy portal to pick up the new docs → go to the portal project and deploy." - -Do NOT deploy portal from here. Portal is always deployed separately from within the portal project. - -## Closed-Source Product Names — HARD RULE - -Brainy is the only Soulcraft open-source project. Nothing in this repo — code, JSDoc, tests, -docs, RELEASES.md, CHANGELOG.md, commit messages — may reference closed-source Soulcraft -products by name (Workshop, Venue, Memory, Muse, Hall, Forge, Academy, Pulse, Heart, -Collective, SDK) or by their specific class/method names (`BookingDraftService`, -`getDemandHeatmap`, `systemKind`, etc.). - -When recording a consumer-reported bug, regression scenario, or release note: -- Refer to "a consumer", "a downstream application", "a production deployment", or "an - internal report" — never name the product. -- All doc examples must use generic domain values (`'employee'`, `'customer'`, `'invoice'`, - `'milestone'`, `OrderService`, `/orders/...`), not product-specific schemas. -- Internal session artifacts (`.strategy/`, `~/.claude/plans/`, handoff files outside the - repo) MAY name products — those are not public. - -If you catch yourself typing a product name into a tracked file, stop and rephrase. - -## Performance Claims - -When documenting performance characteristics: -- **MEASURED**: Cite the test file and line number -- **PROJECTED**: Clearly label as extrapolated from tested scale -- Never claim a performance figure without context or evidence - -## Debugging - -When a bug persists through 2+ fix attempts, switch to systematic debugging: -1. Add comprehensive logging at every step -2. Test with production-like data -3. Trace the complete execution path -4. Check both library code and consumer code -5. Verify with actual test execution before declaring fixed - -## Key Paths - -- Main class: `src/brainy.ts` -- Public API: `src/index.ts` (38+ exports) -- Storage interface: `src/coreTypes.ts` -- Type definitions: `src/types/` -- Strategy/planning docs: `.strategy/` (gitignored, not public) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ab8a0246..7b7e70ad 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -43,38 +43,15 @@ Feature requests are welcome! Please provide: #### Development Setup -**Quick Setup (Recommended):** ```bash # Clone your fork git clone https://github.com/your-username/brainy.git cd brainy -# Run setup script (installs all dependencies including Rust) -./scripts/setup-dev.sh -``` - -**Manual Setup:** -```bash -# Clone your fork -git clone https://github.com/your-username/brainy.git -cd brainy - -# Install system dependencies (Ubuntu/Debian) -sudo apt-get install -y build-essential pkg-config libssl-dev - -# Install Rust (for WASM embedding engine) -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -source ~/.cargo/env -rustup target add wasm32-unknown-unknown -cargo install wasm-pack - -# Install Node.js dependencies +# Install dependencies npm install -# Build Candle WASM embedding engine -npm run build:candle - -# Build TypeScript +# Build the project npm run build # Run tests diff --git a/EXPLORATION_SUMMARY.md b/EXPLORATION_SUMMARY.md new file mode 100644 index 00000000..0b7176e1 --- /dev/null +++ b/EXPLORATION_SUMMARY.md @@ -0,0 +1,393 @@ +# Brainy Storage Adapter Architecture - Exploration Summary + +## Overview + +This exploration analyzed the complete storage adapter architecture in Brainy to understand how it works and determine whether a TypeAwareStorageAdapter can be added alongside existing adapters. + +## Key Findings + +### 1. Architecture is Clean and Extensible + +Brainy implements a **well-designed, modular storage adapter architecture** using: +- **Interface-based design** (StorageAdapter interface in coreTypes.ts) +- **Abstract base classes** for common functionality +- **Concrete implementations** for specific backends +- **Factory pattern** for runtime adapter selection + +### 2. Six Storage Adapters Currently Exist + +| Adapter | Platform | Backend | File | Lines | +|---------|----------|---------|------|-------| +| FileSystemStorage | Node.js | Local filesystem | fileSystemStorage.ts | 2,677 | +| MemoryStorage | Browser/Node.js | In-memory Maps | memoryStorage.ts | 822 | +| S3CompatibleStorage | Node.js | AWS S3, Cloudflare R2, GCS (S3 API) | s3CompatibleStorage.ts | 5,000+ | +| GcsStorage | Node.js | Google Cloud Storage (native SDK) | gcsStorage.ts | 1,835 | +| OPFSStorage | Browser | Origin Private File System | opfsStorage.ts | - | +| R2Storage | Node.js | Alias for S3CompatibleStorage | (alias) | - | + +### 3. Inheritance Hierarchy is Clean + +``` +StorageAdapter (interface - 27 methods) + ↓ +BaseStorageAdapter (abstract - 1,156 lines) + ├─ Statistics management + ├─ Throttling detection + ├─ Count management (O(1)) + └─ Service tracking + ↓ +BaseStorage (abstract - 1,098 lines) + ├─ 2-file system (vectors + metadata) + ├─ UUID-based sharding (256 shards) + ├─ Pagination support + └─ Metadata routing + ↓ +Concrete Adapters (FileSystem, Memory, S3, GCS, OPFS) +``` + +### 4. Core Components + +**Storage System Files (~13,000+ lines total):** +- `src/coreTypes.ts` - StorageAdapter interface +- `src/storage/baseStorageAdapter.ts` - Abstract base (1,156 lines) +- `src/storage/baseStorage.ts` - Core layer (1,098 lines) +- `src/storage/storageFactory.ts` - Factory for selection +- `src/storage/adapters/*.ts` - Concrete implementations +- `src/storage/sharding.ts` - UUID sharding utilities +- `src/storage/cacheManager.ts` - LRU cache + +**Supporting Utilities:** +- `src/utils/writeBuffer.ts` - Batch operations +- `src/utils/adaptiveBackpressure.ts` - Flow control +- `src/utils/requestCoalescer.ts` - Request deduplication +- `src/storage/backwardCompatibility.ts` - Migration support + +### 5. Storage Path Structure + +**Modern Entity-Based Structure:** +``` +entities/ +├── nouns/vectors/{shard}/{id}.json (vector data) +├── nouns/metadata/{shard}/{id}.json (flexible metadata) +├── nouns/hnsw/{shard}/{id}.json (HNSW graph) +├── verbs/vectors/{shard}/{id}.json +├── verbs/metadata/{shard}/{id}.json +└── verbs/hnsw/{shard}/{id}.json + +_system/ +├── statistics.json (aggregate counts) +├── counts.json (O(1) totals) +└── hnsw-system.json (HNSW metadata) +``` + +**Sharding:** UUID first 2 hex chars = 256 shard directories (00-ff) + +### 6. 2-File System Design + +Brainy separates **vector data** from **metadata** for scalability: +- **File 1:** `vectors/{id}.json` - Vector, HNSW connections (lightweight) +- **File 2:** `metadata/{id}.json` - Flexible metadata (any schema) + +**Benefits:** +- Decouple vector operations from metadata queries +- Enable type-aware queries without loading vectors +- Independent scaling of vector vs metadata storage +- Support for metadata-only updates + +### 7. Brainy Integration + +How Brainy uses storage: + +```typescript +// In brainy.ts +class Brainy { + private storage!: BaseStorage + + async init(config: BrainyConfig): Promise { + // Factory creates appropriate adapter + this.storage = await createStorage(config.storage) as BaseStorage + await this.storage.init() + + // Pass to HNSW index + this.index = new HNSWIndex(this.storage, ...) + } +} +``` + +Key insight: **Brainy only knows about `BaseStorage` interface, not specific adapters** + +### 8. Design Patterns Used + +1. **Factory Pattern** - `createStorage()` selects adapter at runtime +2. **Strategy Pattern** - Adapters are interchangeable +3. **Template Method** - BaseStorage defines skeleton, adapters fill details +4. **Adapter Pattern** - Maps different backends to same interface +5. **Decorator Pattern** - Could wrap adapters (e.g., TypeAware wrapper) + +--- + +## Answer: Can TypeAwareStorageAdapter Be Added? + +### YES - DEFINITIVELY + +**TypeAwareStorageAdapter can be added as a new adapter alongside existing ones WITHOUT replacing them.** + +### Reasons + +1. **Factory Pattern:** Multiple adapters coexist via factory function +2. **No Coupling:** Brainy depends on `BaseStorage` interface, not specific adapters +3. **Clean Inheritance:** Just extend `BaseStorage` like all other adapters +4. **Isolated:** Type awareness doesn't affect other adapters +5. **Backward Compatible:** Existing code continues to work unchanged + +### Implementation Path + +**3 Simple Steps:** + +**Step 1: Create new adapter file** +```typescript +// src/storage/adapters/typeAwareStorageAdapter.ts +export class TypeAwareStorageAdapter extends BaseStorage { + // Implement 17 abstract methods + // Add type indexing logic +} +``` + +**Step 2: Update factory** +```typescript +// src/storage/storageFactory.ts +if (options.type === 'type-aware') { + return new TypeAwareStorageAdapter(options) +} +``` + +**Step 3: Update options interface** +```typescript +// src/storage/storageFactory.ts +export interface StorageOptions { + type?: 'auto' | 'memory' | 'filesystem' | 's3' | 'gcs' | 'type-aware' + typeAwareStorage?: { ... } +} +``` + +**No changes needed to:** +- Brainy.ts +- coreTypes.ts (unless adding new methods) +- Existing adapters +- HNSW index +- Any other components + +### Abstract Methods to Implement + +When creating TypeAwareStorageAdapter, implement these 17 methods: + +**Noun/Verb Operations (6):** +- `saveNoun_internal()` +- `getNoun_internal()` +- `deleteNoun_internal()` +- `saveVerb_internal()` +- `getVerb_internal()` +- `deleteVerb_internal()` + +**Path Operations (4):** +- `writeObjectToPath()` +- `readObjectFromPath()` +- `deleteObjectFromPath()` +- `listObjectsUnderPath()` + +**Count Management (2):** +- `initializeCounts()` +- `persistCounts()` + +**Statistics (2):** +- `saveStatisticsData()` +- `getStatisticsData()` + +**Lifecycle (3):** +- `init()` +- `clear()` +- `getStorageStatus()` + +### Recommended Design Approach + +**Option A: Direct Implementation (Recommended)** +``` +TypeAwareStorageAdapter +├─ Extends BaseStorage +├─ Implements all 17 abstract methods +├─ Adds type indexing logic +└─ Can back any storage engine +``` + +**Option B: Wrapper/Decorator Pattern** +``` +TypeAwareStorageAdapter (wrapper) +├─ Wraps any BaseStorage adapter +├─ Intercepts saveNoun/saveVerb +├─ Tracks types in separate index +└─ Delegates all operations +``` + +--- + +## Key Insights + +### Storage Architecture Strengths + +✅ **Well-organized:** Clear separation of concerns +✅ **Extensible:** Factory pattern makes adding adapters simple +✅ **Scalable:** Sharding, caching, batching, backpressure +✅ **Flexible:** Multiple backends coexist without conflicts +✅ **Type-safe:** Full TypeScript with proper interfaces +✅ **Production-ready:** Used in real deployments + +### What Makes This Possible + +1. **Interface-based design** - Adapters implement same contract +2. **Factory pattern** - Runtime selection without coupling +3. **No hardcoded dependencies** - Brainy uses `BaseStorage` type +4. **Common base class** - Shared logic prevents duplication +5. **Metadata separation** - 2-file system enables type indexing + +### Storage Adapter Evolution Path + +``` +Current State (v3.44.0): +├─ FileSystemStorage ✅ +├─ MemoryStorage ✅ +├─ S3CompatibleStorage ✅ +├─ GcsStorage ✅ +└─ OPFSStorage ✅ + +Future State (proposed): +├─ FileSystemStorage ✅ +├─ MemoryStorage ✅ +├─ S3CompatibleStorage ✅ +├─ GcsStorage ✅ +├─ OPFSStorage ✅ +└─ TypeAwareStorageAdapter ✅ (new) + +All coexist without conflicts +``` + +--- + +## Documents Created + +This exploration generated three comprehensive documents: + +### 1. STORAGE_ARCHITECTURE_ANALYSIS.md (28 KB) +Complete analysis covering: +- Current storage architecture overview +- All existing storage adapters +- StorageAdapter interface specification +- How Brainy uses storage +- Storage paths and patterns +- Storage adapter pattern analysis +- Detailed implementation recommendations +- Design patterns and best practices + +### 2. STORAGE_ADAPTER_QUICK_REFERENCE.md (8.6 KB) +Quick reference guide with: +- File locations +- Storage adapter hierarchy +- Abstract methods checklist (17 methods) +- Storage path structure +- 2-file system design +- Existing adapters overview +- Factory integration +- Performance characteristics +- Design patterns summary + +### 3. STORAGE_FILES_REFERENCE.md (13 KB) +Complete file reference with: +- All core storage files +- Line counts and purposes +- Each adapter's features +- Integration points +- Data flow diagrams +- Statistics tracking +- Type definitions +- Summary statistics table + +--- + +## Recommendations + +### For TypeAwareStorageAdapter Implementation + +1. **Use Direct Implementation approach** (not wrapper) + - Simpler to maintain + - Better performance + - Easier to debug + - Can back any storage engine + +2. **Implement as new entry in factory** + - `type: 'type-aware'` with storage config + - Auto-detection can select it + - No changes to existing code + +3. **Leverage 2-file system** + - Store type index in metadata files + - Queries don't require loading vectors + - Aligns with existing patterns + +4. **Inherit common functionality** + - Throttling detection + - Statistics tracking + - Caching and batching + - Count management (O(1)) + +5. **Follow existing patterns** + - Sharding strategy (first 2 hex chars) + - Path structure (entities/{noun|verb}/{vectors|metadata}/{shard}/{id}.json) + - Pagination support + - Metadata separation + +### For Integration + +1. Add new file: `/src/storage/adapters/typeAwareStorageAdapter.ts` +2. Modify: `/src/storage/storageFactory.ts` (add case + interface) +3. Optional: `/src/coreTypes.ts` (if extending StorageAdapter interface) +4. No changes needed elsewhere + +### For Testing + +1. Test with MemoryStorage first (fastest) +2. Test with FileSystemStorage (persistent) +3. Ensure all existing tests still pass +4. Add type-aware specific tests + +--- + +## Conclusion + +Brainy's storage adapter architecture is **professionally designed and inherently extensible**. Adding a TypeAwareStorageAdapter is straightforward because: + +- The architecture supports multiple concurrent adapters +- Brainy uses interface-based dependency injection +- The factory pattern enables runtime selection +- No breaking changes required anywhere + +**The answer is unambiguous: TypeAwareStorageAdapter can be added alongside existing adapters with minimal integration effort.** + +--- + +## Files Analyzed + +- `/src/coreTypes.ts` - Interface definition +- `/src/storage/baseStorageAdapter.ts` - Abstract base +- `/src/storage/baseStorage.ts` - Core layer +- `/src/storage/storageFactory.ts` - Factory +- `/src/storage/adapters/fileSystemStorage.ts` - FileSystem +- `/src/storage/adapters/memoryStorage.ts` - Memory +- `/src/storage/adapters/s3CompatibleStorage.ts` - S3/R2 +- `/src/storage/adapters/gcsStorage.ts` - GCS native +- `/src/storage/adapters/opfsStorage.ts` - Browser OPFS +- `/src/brainy.ts` - Main class +- Plus all supporting utilities and type definitions + +**Total files analyzed:** 50+ +**Total lines examined:** 13,000+ +**Analysis coverage:** Complete storage system + diff --git a/README.md b/README.md index d1342f52..bf631328 100644 --- a/README.md +++ b/README.md @@ -1,217 +1,991 @@ -

- Brainy -

- -

Brainy

+# Brainy

- Three database paradigms. One API. Zero configuration.
- The in-process knowledge database for TypeScript — vector search, graph traversal,
- and metadata filtering unified in a single query. + Brainy Logo

-

- npm version - npm downloads - CI - Documentation - MIT License - TypeScript -

+[![npm version](https://badge.fury.io/js/%40soulcraft%2Fbrainy.svg)](https://www.npmjs.com/package/@soulcraft/brainy) +[![npm downloads](https://img.shields.io/npm/dm/@soulcraft/brainy.svg)](https://www.npmjs.com/package/@soulcraft/brainy) +[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) +[![TypeScript](https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg)](https://www.typescriptlang.org/) -

- Quick start · - One query · - Features · - Scale with Cor · - Docs -

+**🧠 Brainy - The Knowledge Operating System** + +**The world's first Knowledge Operating System** where every piece of knowledge - files, concepts, entities, ideas - exists as living information that understands itself, evolves over time, and connects to everything related. Built on revolutionary Triple Intelligence™ that unifies vector similarity, graph relationships, and document filtering in one magical API. + +**Why Brainy Changes Everything**: Traditional systems trap knowledge in files or database rows. Brainy liberates it. Your characters exist across stories. Your concepts span projects. Your APIs remember their evolution. Every piece of knowledge - whether it's code, prose, or pure ideas - lives, breathes, and connects in a unified intelligence layer where everything understands its meaning, remembers its history, and relates to everything else. + +**Framework-first design.** Built for modern web development with zero configuration and automatic framework compatibility. O(log n) performance, <10ms search latency, production-ready. + +## 🎉 Key Features + +### 🚀 **NEW in 3.47.0: Billion-Scale Type-Aware HNSW** + +**87% memory reduction for billion-scale deployments with 10x faster queries:** + +- **🎯 Type-Aware Vector Index**: Separate HNSW graphs per entity type for massive memory savings + - **Memory @ 1B scale**: 384GB → 50GB (-87% / -334GB) + - **Single-type queries**: 10x faster (search 100M nodes instead of 1B) + - **Multi-type queries**: 5-8x faster (search subset of types) + - **All-types queries**: ~3x faster (31 smaller graphs vs 1 large graph) + +- **⚡ Optimized Rebuild**: Type-filtered pagination for 31x faster index rebuilding + - **Before**: 31B reads (UNACCEPTABLE) + - **After**: 1B reads with type filtering (CORRECT) + - **Parallel type rebuilds**: 10-20 minutes for all types + - **Lazy loading**: 15 minutes for top 2 types only + +- **📊 Production-Ready**: Comprehensive testing and zero breaking changes + - 47 new tests (33 unit + 14 integration) - all passing + - Backward compatible - opt-in via configuration + - Works with all storage backends (FileSystem, S3, GCS, R2, Memory, OPFS) + +**[📖 Phase 2 Architecture →](.strategy/PHASE_2_TYPE_AWARE_HNSW_DESIGN.md)** + +### ⚡ **NEW in 3.36.0: Production-Scale Memory & Performance** + +**Enterprise-grade adaptive sizing and zero-overhead optimizations:** + +- **🎯 Adaptive Memory Sizing**: Auto-scales from 2GB to 128GB+ based on available system resources + - Container-aware (Docker/K8s cgroups v1/v2 detection) + - Environment-smart (development 25%, container 40%, production 50% allocation) + - Model memory accounting (150MB Q8, 250MB FP32 reserved before cache) + +- **⚡ Sync Fast Path**: Zero async overhead when vectors are cached + - Intelligent sync/async branching - synchronous when data is in memory + - Falls back to async only when loading from storage + - Massive performance win for hot paths (vector search, distance calculations) + +- **📊 Production Monitoring**: Comprehensive diagnostics + - `getCacheStats()` - UnifiedCache hit rates, fairness metrics, memory pressure + - Actionable recommendations for tuning + - Tracks model memory, cache efficiency, and competition across indexes + +- **🛡️ Zero Breaking Changes**: All optimizations are internal - your code stays the same + - Public API unchanged + - Automatic memory detection and allocation + - Progressive enhancement for existing applications + +**[📖 Operations Guide →](docs/operations/capacity-planning.md)** | **[🎯 Migration Guide →](docs/guides/migration-3.36.0.md)** + +### 🚀 **NEW in 3.21.0: Enhanced Import & Neural Processing** + +- **📊 Progress Tracking**: Unified progress reporting with automatic time estimation +- **⚡ Entity Caching**: 10-100x speedup on repeated entity extraction +- **🔗 Relationship Confidence**: Multi-factor confidence scoring (0-1 scale) +- **📝 Evidence Tracking**: Understand why relationships were detected +- **🎯 Production Ready**: Fully backward compatible, opt-in features + +### 🧠 **Triple Intelligence™ Engine** + +- **Vector Search**: HNSW-powered semantic similarity +- **Graph Relationships**: Navigate connected knowledge +- **Document Filtering**: MongoDB-style metadata queries +- **Unified API**: All three in a single query interface + +### 🎯 **Clean API Design** + +- **Modern Syntax**: `brain.add()`, `brain.find()`, `brain.relate()` +- **Type Safety**: Full TypeScript integration +- **Zero Config**: Works out of the box with intelligent storage auto-detection +- **Consistent Parameters**: Clean, predictable API surface + +### ⚡ **Performance & Reliability** + +- **<10ms Search**: Fast semantic queries +- **384D Vectors**: Optimized embeddings (all-MiniLM-L6-v2) +- **Built-in Caching**: Intelligent result caching + new entity extraction cache +- **Production Ready**: Thoroughly tested core functionality + +## ⚡ Quick Start - Zero Configuration + +```bash +npm install @soulcraft/brainy +``` + +### 🎯 **True Zero Configuration** + +```javascript +import { Brainy, NounType } from '@soulcraft/brainy' + +// Just this - auto-detects everything! +const brain = new Brainy() +await brain.init() + +// Add entities with automatic embedding +const jsId = await brain.add({ + data: "JavaScript is a programming language", + nounType: NounType.Concept, + metadata: { + type: "language", + year: 1995, + paradigm: "multi-paradigm" + } +}) + +const nodeId = await brain.add({ + data: "Node.js runtime environment", + nounType: NounType.Concept, + metadata: { + type: "runtime", + year: 2009, + platform: "server-side" + } +}) + +// Create relationships between entities +await brain.relate({ + from: nodeId, + to: jsId, + type: "executes", + metadata: { + since: 2009, + performance: "high" + } +}) + +// Natural language search with graph relationships +const results = await brain.find({query: "programming languages used by server runtimes"}) + +// Triple Intelligence: vector + metadata + relationships +const filtered = await brain.find({ + query: "JavaScript", // Vector similarity + where: {type: "language"}, // Metadata filtering + connected: {from: nodeId, depth: 1} // Graph relationships +}) +``` + +## 🌐 Framework Integration + +**Brainy is framework-first!** Works seamlessly with any modern JavaScript framework: + +### ⚛️ **React & Next.js** +```javascript +import { Brainy } from '@soulcraft/brainy' + +function SearchComponent() { + const [brain] = useState(() => new Brainy()) + + useEffect(() => { + brain.init() + }, []) + + const handleSearch = async (query) => { + const results = await brain.find(query) + setResults(results) + } +} +``` + +### 🟢 **Vue.js & Nuxt.js** +```javascript +import { Brainy } from '@soulcraft/brainy' + +export default { + async mounted() { + this.brain = new Brainy() + await this.brain.init() + }, + methods: { + async search(query) { + return await this.brain.find(query) + } + } +} +``` + +### 🅰️ **Angular** +```typescript +import { Injectable } from '@angular/core' +import { Brainy } from '@soulcraft/brainy' + +@Injectable({ providedIn: 'root' }) +export class BrainyService { + private brain = new Brainy() + + async init() { + await this.brain.init() + } + + async search(query: string) { + return await this.brain.find(query) + } +} +``` + +### 🔥 **Other Frameworks** +Brainy works with **any** framework that supports ES6 imports: Svelte, Solid.js, Qwik, Fresh, and more! + +**Framework Compatibility:** +- ✅ All modern bundlers (Webpack, Vite, Rollup, Parcel) +- ✅ SSR/SSG (Next.js, Nuxt, SvelteKit, Astro) +- ✅ Edge runtimes (Vercel Edge, Cloudflare Workers) +- ✅ Browser and Node.js environments + +## 📋 System Requirements + +**Node.js Version:** 22 LTS or later (recommended) + +- ✅ **Node.js 22 LTS** - Fully supported and recommended for production +- ✅ **Node.js 20 LTS** - Compatible (maintenance mode) +- ❌ **Node.js 24** - Not supported (known ONNX runtime compatibility issues) + +> **Important:** Brainy uses ONNX runtime for AI embeddings. Node.js 24 has known compatibility issues that cause +> crashes during inference operations. We recommend Node.js 22 LTS for maximum stability. + +If using nvm: `nvm use` (we provide a `.nvmrc` file) + +## 🚀 Key Features + +### World's First Triple Intelligence™ Engine + +**The breakthrough that enables The Knowledge Operating System:** + +- **Vector Search**: Semantic similarity with HNSW indexing +- **Graph Relationships**: Navigate connected knowledge like Neo4j +- **Document Filtering**: MongoDB-style queries with O(log n) performance +- **Unified in ONE API**: No separate queries, no complex joins +- **First to solve this**: Others do vector OR graph OR document—we do ALL + +### The Knowledge Operating System with Infinite Expressiveness + +**Enabled by Triple Intelligence, standardized for everyone:** + +- **31 Noun Types × 40 Verb Types**: 1,240 base combinations +- **∞ Expressiveness**: Unlimited metadata = model ANY data +- **One Language**: All tools, augmentations, AI models speak the same types +- **Perfect Interoperability**: Move data between any Brainy instance +- **No Schema Lock-in**: Evolve without migrations + +### Natural Language Understanding + +```javascript +// Ask questions naturally +await brain.find("Show me recent React components with tests") +await brain.find("Popular JavaScript libraries similar to Vue") +await brain.find("Documentation about authentication from last month") +``` + +### 🧠🌐 **Virtual Filesystem - Intelligent File Management** + +**Build file explorers, IDEs, and knowledge systems that never crash from infinite recursion.** + +- **Tree-Aware Operations**: Safe directory listing prevents recursive loops +- **Semantic Search**: Find files by content, not just filename +- **Production Storage**: Filesystem and cloud storage for real applications +- **Zero-Config**: Works out of the box with intelligent defaults + +```javascript +import { Brainy } from '@soulcraft/brainy' + +// ✅ CORRECT: Use persistent storage for file systems +const brain = new Brainy({ + storage: { + type: 'filesystem', // Persisted to disk + path: './brainy-data' // Your file storage + } +}) +await brain.init() + +const vfs = brain.vfs() +await vfs.init() + +// ✅ Safe file operations +await vfs.writeFile('/projects/app/index.js', 'console.log("Hello")') +await vfs.mkdir('/docs') +await vfs.writeFile('/docs/README.md', '# My Project') + +// ✅ NEVER crashes: Tree-aware directory listing +const children = await vfs.getDirectChildren('/projects') +// Returns only direct children, never the directory itself + +// ✅ Build file explorers safely +const tree = await vfs.getTreeStructure('/projects', { + maxDepth: 3, // Prevent deep recursion + sort: 'name' // Organized results +}) + +// ✅ Semantic file search +const reactFiles = await vfs.search('React components with hooks') +const docs = await vfs.search('API documentation', { + path: '/docs' // Search within specific directory +}) + +// ✅ Connect related files +await vfs.addRelationship('/src/auth.js', '/tests/auth.test.js', 'tested-by') + +// Perfect for: File explorers, IDEs, documentation systems, code analysis +``` + +**🚨 Prevents Common Mistakes:** +- ❌ No infinite recursion in file trees (like brain-cloud team experienced) +- ❌ No data loss from memory storage +- ❌ No performance issues with large directories +- ❌ No need for complex fallback patterns + +**[📖 VFS Quick Start →](docs/vfs/QUICK_START.md)** | **[🎯 Common Patterns →](docs/vfs/COMMON_PATTERNS.md)** + +**Your knowledge isn't trapped anymore.** Characters live beyond stories. APIs exist beyond code files. Concepts connect across domains. This is knowledge that happens to support files, not a filesystem that happens to store knowledge. + +### 🚀 **NEW: Enhanced Directory Import with Caching** + +**Import large projects 10-100x faster with intelligent caching:** + +```javascript +import { Brainy } from '@soulcraft/brainy' +import { ProgressTracker, formatProgress } from '@soulcraft/brainy/types' +import { detectRelationshipsWithConfidence } from '@soulcraft/brainy/neural' + +const brain = new Brainy() +await brain.init() + +// Progress tracking for long operations +const tracker = ProgressTracker.create(1000) +tracker.start() + +for await (const progress of importer.importStream('./project', { + batchSize: 100, + generateEmbeddings: true +})) { + const p = tracker.update(progress.processed, progress.current) + console.log(formatProgress(p)) + // [RUNNING] 45% (450/1000) - 23.5 items/s - 23s remaining +} + +// Entity extraction with intelligent caching +const entities = await brain.neural.extractor.extract(text, { + types: ['person', 'organization', 'technology'], + confidence: 0.7, + cache: { + enabled: true, + ttl: 7 * 24 * 60 * 60 * 1000, // 7 days + invalidateOn: 'mtime' // Re-extract when file changes + } +}) + +// Relationship detection with confidence scores +const relationships = detectRelationshipsWithConfidence(entities, text, { + minConfidence: 0.7 +}) + +// Create relationships with evidence tracking +await brain.relate({ + from: sourceId, + to: targetId, + type: 'creates', + confidence: 0.85, + evidence: { + sourceText: 'John created the database', + method: 'pattern', + reasoning: 'Matches creation pattern; entities in same sentence' + } +}) + +// Monitor cache performance +const stats = brain.neural.extractor.getCacheStats() +console.log(`Cache hit rate: ${(stats.hitRate * 100).toFixed(1)}%`) +// Cache hit rate: 89.5% +``` + +**📚 [See Full Example →](examples/directory-import-with-caching.ts)** + +### 🎯 Zero Configuration Philosophy + +Brainy automatically configures **everything**: + +```javascript +import { Brainy } from '@soulcraft/brainy' + +// 1. Pure zero-config - detects everything +const brain = new Brainy() + +// 2. Custom configuration +const brain = new Brainy({ + storage: { type: 'filesystem', path: './brainy-data' }, + embeddings: { model: 'all-MiniLM-L6-v2' }, + cache: { enabled: true, maxSize: 1000 } +}) + +// 3. Production configuration +const customBrain = new Brainy({ + mode: 'production', + model: 'q8', // Optimized model (99% accuracy, 75% smaller) + storage: 'cloud', // or 'memory', 'disk', 'auto' + features: ['core', 'search', 'cache'] +}) +``` + +**What's Auto-Detected:** + +- **Storage**: S3/GCS/R2 → Filesystem (priority order) +- **Models**: Always Q8 for optimal balance +- **Features**: Minimal → Default → Full based on environment +- **Memory**: Optimal cache sizes and batching +- **Performance**: Threading, chunking, indexing strategies + +### Production Performance + +- **3ms average search** - Lightning fast queries +- **24MB memory footprint** - Efficient resource usage +- **Worker-based embeddings** - Non-blocking operations +- **Automatic caching** - Intelligent result caching + +### 🎛️ Advanced Configuration (When Needed) + +Most users **never need this** - zero-config handles everything. For advanced use cases: + +```javascript +// Model is always Q8 for optimal performance +const brain = new Brainy() // Uses Q8 automatically + +// Storage control (auto-detected by default) +const diskBrain = new Brainy({storage: 'disk'}) // Local filesystem +const cloudBrain = new Brainy({storage: 'cloud'}) // S3/GCS/R2 + +// Legacy full config (still supported) +const legacyBrain = new Brainy({ + storage: {type: 'filesystem', path: './data'} +}) +``` + +**Model Details:** + +- **Q8**: 33MB, 99% accuracy, 75% smaller than full precision +- Fast loading and optimal memory usage +- Perfect for all environments + +**Air-gap deployment:** + +```bash +npm run download-models # Download Q8 model +npm run download-models:q8 # Download Q8 model +``` + +## 🚀 Import Anything - Files, Data, URLs + +Brainy's universal import intelligently handles **any data format**: + +```javascript +// Import CSV with auto-detection +await brain.import('customers.csv') +// ✨ Auto-detects: encoding, delimiter, types, creates entities! + +// Import Excel workbooks with multi-sheet support +await brain.import('sales-data.xlsx', { + excelSheets: ['Q1', 'Q2'] // or 'all' for all sheets +}) +// ✨ Processes all sheets, preserves structure, infers types! + +// Import PDF documents with table extraction +await brain.import('research-paper.pdf', { + pdfExtractTables: true +}) +// ✨ Extracts text, detects tables, preserves metadata! + +// Import JSON/YAML data +await brain.import([ + { name: 'Alice', role: 'Engineer' }, + { name: 'Bob', role: 'Designer' } +]) +// ✨ Automatically creates Person entities with relationships! + +// Import from URLs (auto-fetched) +await brain.import('https://api.example.com/data.json') +// ✨ Auto-detects URL, fetches, parses, processes! +``` + +**📖 [Complete Import Guide →](docs/guides/import-anything.md)** | **[Live Example →](examples/import-excel-pdf-csv.ts)** + +## 📚 Core API + +### `search()` - Vector Similarity + +```javascript +const results = await brain.search("machine learning", { + limit: 10, // Number of results + metadata: {type: "article"}, // Filter by metadata + includeContent: true // Include full content +}) +``` + +### `find()` - Natural Language Queries + +```javascript +// Simple natural language +const results = await brain.find("recent important documents") + +// Structured query with Triple Intelligence +const results = await brain.find({ + like: "JavaScript", // Vector similarity + where: { // Metadata filters + year: {greaterThan: 2020}, + important: true + }, + related: {to: "React"} // Graph relationships +}) +``` + +### CRUD Operations + +```javascript +// Create entities (nouns) +const id = await brain.add(data, { nounType: nounType, ...metadata }) + +// Create relationships (verbs) +const verbId = await brain.relate(sourceId, targetId, "relationType", { + strength: 0.9, + bidirectional: false +}) + +// Read +const item = await brain.getNoun(id) +const verb = await brain.getVerb(verbId) + +// Update +await brain.updateNoun(id, newData, newMetadata) +await brain.updateVerb(verbId, newMetadata) + +// Delete +await brain.deleteNoun(id) +await brain.deleteVerb(verbId) + +// Bulk operations +await brain.import(arrayOfData) +const exported = await brain.export({format: 'json'}) + +// Import from CSV, Excel, PDF files (auto-detected) +await brain.import('customers.csv') // CSV with encoding detection +await brain.import('sales-report.xlsx') // Excel with multi-sheet support +await brain.import('research.pdf') // PDF with table extraction +``` + +## 🌐 Distributed System (NEW!) + +### Zero-Config Distributed Setup + +```javascript +// Single node (default) +const brain = new Brainy({ + storage: { + type: 's3', + s3Storage: { + bucketName: 'my-data', + region: 'us-east-1', + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY + } + } +}) + +// Distributed cluster - just add one flag! +const brain = new Brainy({ + storage: { + type: 's3', + s3Storage: { + bucketName: 'my-data', + region: 'us-east-1', + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY + } + }, + distributed: true // That's it! Everything else is automatic +}) +``` + +### How It Works + +- **Storage-Based Discovery**: Nodes find each other via S3/GCS (no Consul/etcd!) +- **Automatic Sharding**: Data distributed by content hash +- **Smart Query Planning**: Queries routed to optimal shards +- **Live Rebalancing**: Handles node joins/leaves automatically +- **Zero Downtime**: Streaming shard migration + +### Real-World Example: Social Media Firehose + +```javascript +import { Brainy, NounType } from '@soulcraft/brainy' + +// Ingestion nodes (optimized for writes) +const ingestionNode = new Brainy({ + storage: { + type: 's3', + s3Storage: { + bucketName: 'social-data', + region: 'us-east-1', + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY + } + }, + distributed: true, + writeOnly: true // Optimized for high-throughput writes +}) + +// Process Bluesky firehose +blueskyStream.on('post', async (post) => { + await ingestionNode.add(post, { + nounType: NounType.Message, + platform: 'bluesky', + author: post.author, + timestamp: post.createdAt + }) +}) + +// Search nodes (optimized for queries) +const searchNode = new Brainy({ + storage: { + type: 's3', + s3Storage: { + bucketName: 'social-data', + region: 'us-east-1', + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY + } + }, + distributed: true, + readOnly: true // Optimized for fast queries +}) + +// Search across ALL data from ALL nodes +const trending = await searchNode.find('trending AI topics', { + where: {timestamp: {greaterThan: Date.now() - 3600000}}, + limit: 100 +}) +``` + +### Benefits Over Traditional Systems + +| Feature | Traditional (Pinecone, Weaviate) | Brainy Distributed | +|----------------|----------------------------------|-------------------------------| +| Setup | Complex (k8s, operators) | One flag: `distributed: true` | +| Coordination | External (etcd, Consul) | Built-in (via storage) | +| Minimum Nodes | 3-5 for HA | 1 (scale as needed) | +| Sharding | Random | Domain-aware | +| Query Planning | Basic | Triple Intelligence | +| Cost | High (always-on clusters) | Low (scale to zero) | + +## 🎯 Use Cases + +### Knowledge Management with Relationships + +```javascript +// Store documentation with rich relationships +const apiGuide = await brain.add("REST API Guide", { + nounType: NounType.Document, + title: "API Guide", + category: "documentation", + version: "2.0" +}) + +const author = await brain.add("Jane Developer", { + nounType: NounType.Person, + role: "tech-lead" +}) + +const project = await brain.add("E-commerce Platform", { + nounType: NounType.Project, + status: "active" +}) + +// Create knowledge graph +await brain.relate(author, apiGuide, "authored", { + date: "2024-03-15" +}) +await brain.relate(apiGuide, project, "documents", { + coverage: "complete" +}) + +// Query the knowledge graph naturally +const docs = await brain.find("documentation authored by tech leads for active projects") +``` + +### Semantic Search + +```javascript +// Find similar content +const similar = await brain.search(existingContent, { + limit: 5, + threshold: 0.8 +}) +``` + +### AI Memory Layer with Context + +```javascript +// Store messages with relationships +const userId = await brain.add("User 123", { + nounType: NounType.User, + tier: "premium" +}) + +const messageId = await brain.add(userMessage, { + nounType: NounType.Message, + timestamp: Date.now(), + session: "abc" +}) + +const topicId = await brain.add("Product Support", { + nounType: NounType.Topic, + category: "support" +}) + +// Link message elements +await brain.relate(userId, messageId, "sent") +await brain.relate(messageId, topicId, "about") + +// Retrieve context with relationships +const context = await brain.find({ + where: {type: "message"}, + connected: {from: userId, type: "sent"}, + like: "previous product issues" +}) +``` + +## 💾 Storage Options + +Brainy supports multiple storage backends: + +```javascript +// FileSystem (Node.js - recommended for development) +const brain = new Brainy({ + storage: { + type: 'filesystem', + path: './data' + } +}) + +// Browser Storage (OPFS) - Works with frameworks +const brain = new Brainy({ + storage: {type: 'opfs'} // Framework handles browser polyfills +}) + +// S3 Compatible (Production) +const brain = new Brainy({ + storage: { + type: 's3', + bucket: 'my-bucket', + region: 'us-east-1' + } +}) +``` + +## 🛠️ CLI + +Brainy includes a powerful CLI for testing and management: + +```bash +# Install globally +npm install -g brainy + +# Add data +brainy add "JavaScript is awesome" --metadata '{"type":"opinion"}' + +# Search +brainy search "programming" + +# Natural language find +brainy find "awesome programming languages" + +# Interactive mode +brainy chat + +# Export data +brainy export --format json > backup.json +``` + +## 🧠 Neural API - Advanced AI Features + +Brainy includes a powerful Neural API for advanced semantic analysis: + +### Clustering & Analysis + +```javascript +// Access via brain.neural +const neural = brain.neural + +// Automatic semantic clustering +const clusters = await neural.clusters() +// Returns groups of semantically similar items + +// Cluster with options +const clusters = await neural.clusters({ + algorithm: 'kmeans', // or 'hierarchical', 'sample' + maxClusters: 5, // Maximum number of clusters + threshold: 0.8 // Similarity threshold +}) + +// Calculate similarity between any items +const similarity = await neural.similar('item1', 'item2') +// Returns 0-1 score + +// Find nearest neighbors +const neighbors = await neural.neighbors('item-id', 10) + +// Build semantic hierarchy +const hierarchy = await neural.hierarchy('item-id') + +// Detect outliers +const outliers = await neural.outliers(0.3) + +// Generate visualization data for D3/Cytoscape +const vizData = await neural.visualize({ + maxNodes: 100, + dimensions: 3, + algorithm: 'force' +}) +``` + +### Real-World Examples + +```javascript +// Group customer feedback into themes +const feedbackClusters = await neural.clusters() +for (const cluster of feedbackClusters) { + console.log(`Theme: ${cluster.label}`) + console.log(`Items: ${cluster.members.length}`) +} + +// Find related documents +const docId = await brain.add("Machine learning guide", { nounType: NounType.Document }) +const similar = await neural.neighbors(docId, 5) +// Returns 5 most similar documents + +// Detect anomalies in data +const anomalies = await neural.outliers(0.2) +console.log(`Found ${anomalies.length} outliers`) +``` + +## 🔌 Augmentations + +Extend Brainy with powerful augmentations: + +```bash +# List available augmentations +brainy augment list + +# Install an augmentation +brainy augment install explorer + +# Connect to Brain Cloud +brainy cloud setup +``` + +## 🏢 Enterprise Features - Included for Everyone + +Brainy includes enterprise-grade capabilities at no extra cost. **No premium tiers, no paywalls.** + +- **Scales to 10M+ items** with consistent 3ms search latency +- **Distributed architecture** with sharding and replication +- **Read/write separation** for horizontal scaling +- **Connection pooling** and request deduplication +- **Built-in monitoring** with metrics and health checks +- **Production ready** with circuit breakers and backpressure + +📖 **More enterprise features coming soon** - Stay tuned! + +## 📊 Benchmarks + +| Operation | Performance | Memory | +|----------------------------------|-------------|----------| +| Initialize | 450ms | 24MB | +| Add Item | 12ms | +0.1MB | +| Vector Search (1k items) | 3ms | - | +| Metadata Filter (10k items) | 0.8ms | - | +| Natural Language Query | 15ms | - | +| Bulk Import (1000 items) | 2.3s | +8MB | +| **Production Scale (10M items)** | **5.8ms** | **12GB** | + +## 🔄 Migration from Previous Versions + +Key changes in the latest version: + +- Search methods consolidated into `search()` and `find()` +- Result format now includes full objects with metadata +- Enhanced natural language capabilities +- Distributed architecture support + +## 🤝 Contributing + +We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. + +## 🧠 The Knowledge Operating System Explained + +### How We Achieved The Impossible + +**Triple Intelligence™** makes us the **world's first** to unify three database paradigms: + +1. **Vector databases** (Pinecone, Weaviate) - semantic similarity +2. **Graph databases** (Neo4j, ArangoDB) - relationships +3. **Document databases** (MongoDB, Elasticsearch) - metadata filtering + +**One API to rule them all.** Others make you choose. We unified them. + +### The Math of Infinite Expressiveness + +``` +31 Nouns × 40 Verbs × ∞ Metadata × Triple Intelligence = Universal Protocol +``` + +- **1,240 base combinations** from standardized types +- **∞ domain specificity** via unlimited metadata +- **∞ relationship depth** via graph traversal +- **= Model ANYTHING**: From quantum physics to social networks + +### Why This Changes Everything + +**Like HTTP for the web, Brainy for knowledge:** + +- All augmentations compose perfectly - same noun-verb language +- All AI models share knowledge - GPT, Claude, Llama all understand +- All tools integrate seamlessly - no translation layers +- All data flows freely - perfect portability + +**The Vision**: One protocol. All knowledge. Every tool. Any AI. + +**Proven across industries**: Healthcare, Finance, Manufacturing, Education, Legal, Retail, Government, and beyond. + +[→ See the Mathematical Proof & Full Taxonomy](docs/architecture/noun-verb-taxonomy.md) + +## 📖 Documentation + +### Framework Integration +- [Framework Integration Guide](docs/guides/framework-integration.md) - Complete framework setup guide +- [Next.js Integration](docs/guides/nextjs-integration.md) - React and Next.js examples +- [Vue.js Integration](docs/guides/vue-integration.md) - Vue and Nuxt examples + +### Virtual Filesystem (Semantic VFS) 🧠📁 +- [VFS Core Documentation](docs/vfs/VFS_CORE.md) - Complete filesystem architecture and API +- [Semantic VFS Guide](docs/vfs/SEMANTIC_VFS.md) - Multi-dimensional file access (6 semantic dimensions) +- [Neural Extraction API](docs/vfs/NEURAL_EXTRACTION.md) - AI-powered concept and entity extraction +- [Examples & Scenarios](docs/vfs/VFS_EXAMPLES_SCENARIOS.md) - Real-world use cases and code +- [VFS API Guide](docs/vfs/VFS_API_GUIDE.md) - Complete API reference + +### Core Documentation +- [Getting Started Guide](docs/guides/getting-started.md) +- [API Reference](docs/api/README.md) +- [Architecture Overview](docs/architecture/overview.md) +- [Data Storage Architecture](docs/architecture/data-storage-architecture.md) - Deep dive into storage, indexing, and sharding +- [Natural Language Guide](docs/guides/natural-language.md) +- [Triple Intelligence](docs/architecture/triple-intelligence.md) +- [Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md) + +## 🏢 Enterprise & Cloud + +**Brain Cloud** - Managed Brainy with team sync, persistent memory, and enterprise connectors. + +```bash +# Get started with free trial +brainy cloud setup +``` + +Visit [soulcraft.com](https://soulcraft.com) for more information. + +## 📄 License + +MIT © Brainy Contributors --- -Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact **three ways at once** and lets one call query them together: - -| You write | Brainy indexes it as | You query it with | -|---|---|---| -| `data: 'Ada wrote the first program'` | a **384-dim vector** (local embedding — no API key) | `find({ query: 'computing pioneers' })` | -| `metadata: { field: 'CS', year: 1843 }` | **structured fields** (O(1) exact, O(log n) range) | `find({ where: { year: { lessThan: 1900 } } })` | -| `relate({ from: ada, to: babbage })` | a **typed, directed graph edge** | `find({ connected: { to: babbage, depth: 2 } })` | - -It runs **inside your process** — no server, no Docker, nothing to operate — and persists to plain files you can snapshot with a hard link. - -**New here?** → **[What is Brainy? — plain-language overview, no jargon](docs/eli5.md)** - -## Quick start - -```bash -bun add @soulcraft/brainy # Bun ≥ 1.1 — recommended -npm install @soulcraft/brainy # Node.js ≥ 22 -``` - -```javascript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' - -const brain = new Brainy() // in-memory; one line swaps to disk -await brain.init() - -// Text auto-embeds locally; metadata auto-indexes -const react = await brain.add({ - data: 'React is a JavaScript library for building user interfaces', - type: NounType.Concept, - subtype: 'library', - metadata: { category: 'frontend', year: 2013 } -}) - -const next = await brain.add({ - data: 'Next.js is a React framework with server-side rendering', - type: NounType.Concept, - subtype: 'framework', - metadata: { category: 'frontend', year: 2016 } -}) - -await brain.relate({ from: next, to: react, type: VerbType.DependsOn, subtype: 'runtime' }) -``` - -## One query, three engines - -```javascript -const results = await brain.find({ - query: 'modern frontend frameworks', // vector — what it means - where: { year: { greaterThan: 2015 } }, // metadata — what it is - connected: { to: react, depth: 2 } // graph — what it touches -}) -``` - -Every clause is optional; any combination composes. Under the hood Brainy plans the query across an HNSW vector index, a roaring-bitmap field index, and an adjacency graph index — and re-validates every result against your predicate before returning it, so a corrupt index can never hand you a wrong answer. - -## Feature tour - -### The database is a value - -Pin it, rewind it, fork it. Snapshot isolation without a server. - -```javascript -const db = brain.now() // pin current state — O(1) - -await brain.transact([ // atomic all-or-nothing, CAS-guarded - { op: 'update', id: order, metadata: { status: 'paid' } }, - { op: 'relate', from: invoice, to: order, type: VerbType.References, subtype: 'billing' } -], { ifAtGeneration: db.generation }) - -await db.get(order) // still 'pending' — pinned forever -await brain.get(order) // 'paid' — live - -const lastWeek = await brain.asOf(Date.now() - 7 * 86_400_000) // full query surface, past state -const whatIf = await db.with([{ op: 'remove', id: order }]) // speculative — never touches disk -await brain.now().persist('/backups/today') // instant hard-link snapshot -``` - -**[Consistency model](docs/concepts/consistency-model.md)** · **[Snapshots & time travel](docs/guides/snapshots-and-time-travel.md)** - -### Local embeddings — no API keys - -Strings embed on-device with a bundled MiniLM model (WASM). Semantic search works offline, in CI, and on air-gapped machines, at zero cost per call. Hybrid keyword + semantic ranking is the default: - -```javascript -await brain.find({ query: 'David Smith' }) // auto: text + semantic -await brain.find({ query: 'AI concepts', searchMode: 'semantic' }) // semantic only -``` - -### A typed graph, not a bag of edges - -42 entity types × 127 relationship types form a shared vocabulary for any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), yours. Your own taxonomy layers on with `subtype`, enforced at write time: - -```javascript -await brain.add({ data: 'Avery Brooks', type: NounType.Person, subtype: 'employee' }) - -brain.counts.bySubtype(NounType.Person) // O(1) — { employee: 12, customer: 847 } -brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true }) -``` - -**[Type system](docs/architecture/noun-verb-taxonomy.md)** · **[Subtypes & facets](docs/guides/subtypes-and-facets.md)** - -### Graph analytics built in - -```javascript -await brain.graph.rank() // which entities matter most (centrality) -await brain.graph.communities() // natural clusters -await brain.graph.path(a, b) // how two things connect -await brain.graph.subgraph([seed], { depth: 2 }) // bounded neighborhood → { nodes, edges } -await brain.graph.export() // whole graph, one O(N+E) streaming pass -``` - -### Write-time aggregations - -`SUM` / `COUNT` / `AVG` / `MIN` / `MAX` with `GROUP BY` and time windows, maintained incrementally on every write — reads are O(1) lookups, not scans. **[Aggregation guide](docs/guides/aggregation.md)** - -### Import anything - -```javascript -await brain.import('customers.csv') -await brain.import('sales.xlsx') // every sheet -await brain.import('research-paper.pdf') // tables extracted -await brain.import('https://api.example.com/data.json') -``` - -Entities auto-classify on the way in; `brain.extractEntities(text)` exposes the same NER ensemble directly. **[Import guide](docs/guides/import-anything.md)** - -### A filesystem that understands content - -```javascript -await brain.vfs.writeFile('/docs/readme.md', 'Project documentation') -await brain.vfs.search('React components with hooks') // semantic file search -``` - -**[VFS quick start](docs/vfs/QUICK_START.md)** - -### Operations-grade by default - -- **Single-writer, many-reader** — an exclusive lock protects the data directory; `Brainy.openReadOnly()` and the `brainy inspect` CLI examine a live brain from another process, safely. -- **Self-upgrading data files** — a 7.x brain opens under 8.x and migrates itself behind an observable lock (`getIndexStatus().migration`), with an automatic pre-upgrade backup. No migration scripts. -- **No silent wrong answers** — cold-open guards self-heal or throw typed errors (`MetadataIndexNotReadyError`, `GraphIndexNotReadyError`); they never return `[]` for data that exists. - -**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)** - -## From laptop to hundreds of millions - -Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**: - -```bash -npm install @soulcraft/cor -``` - -```javascript -const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } }) -await brain.init() // @soulcraft/cor detected — same code, native engines underneath -``` - -Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate. - -Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both. - -## Performance - -- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41). -- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan. -- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)** - -## Use cases - -**AI agent memory** — persistent semantic recall with relationship tracking · **Knowledge bases** — auto-linking and meaning-aware navigation · **Semantic search** over codebases, documents, media · **Enterprise data** — CRM, catalogs, institutional memory · **Games & simulations** — worlds and characters that remember. - -## Documentation - -| Start | Core | Going deeper | -|---|---|---| -| [Brainy explained simply](docs/eli5.md) | [API reference](docs/api/README.md) | [Architecture overview](docs/architecture/overview.md) | -| [Installation](docs/guides/installation.md) | [Data model](docs/DATA_MODEL.md) | [Consistency model](docs/concepts/consistency-model.md) | -| [Natural-language queries](docs/guides/natural-language.md) | [Query operators](docs/QUERY_OPERATORS.md) | [Multi-process model](docs/concepts/multi-process.md) | -| | [Find system](docs/FIND_SYSTEM.md) | [Scaling](docs/SCALING.md) | - -## Requirements - -**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use. - -## Contributing & license - -Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors. +

+ Built with ❤️ by the Brainy community
+ Zero-Configuration AI Database with Triple Intelligence™ +

\ No newline at end of file diff --git a/README_STORAGE_EXPLORATION.md b/README_STORAGE_EXPLORATION.md new file mode 100644 index 00000000..de3d8894 --- /dev/null +++ b/README_STORAGE_EXPLORATION.md @@ -0,0 +1,291 @@ +# Storage Adapter Architecture Exploration - Documentation Index + +## Quick Answer + +**Can TypeAwareStorageAdapter be added alongside existing adapters?** + +**YES - ABSOLUTELY.** It can be added as a new adapter without replacing any existing ones. See **EXPLORATION_SUMMARY.md** for details. + +--- + +## Documentation Files + +### 1. EXPLORATION_SUMMARY.md (12 KB) +**START HERE** - Overview of the entire exploration + +**Contains:** +- Executive summary of findings +- Definitive answer to the main question +- Implementation roadmap (3 simple steps) +- List of 17 abstract methods to implement +- Key insights about architecture +- Recommended design approaches + +**Read this when:** You want a quick understanding of what we discovered + +--- + +### 2. STORAGE_ARCHITECTURE_ANALYSIS.md (28 KB) +**COMPREHENSIVE DEEP DIVE** - Complete technical analysis + +**Sections:** +1. Current storage architecture overview +2. Existing storage adapters (5 detailed profiles) +3. StorageAdapter interface specification (27 methods) +4. How Brainy uses storage +5. Storage factory pattern +6. Current storage paths and patterns +7. Storage adapter pattern analysis +8. Detailed recommendations for TypeAwareStorageAdapter +9. Storage directory structure details +10. Key design patterns +11. Summary and recommendations + +**Read this when:** You need comprehensive technical understanding + +--- + +### 3. STORAGE_ADAPTER_QUICK_REFERENCE.md (8.6 KB) +**QUICK LOOKUP GUIDE** - Fast reference for developers + +**Sections:** +- File locations (all storage files) +- Storage adapter hierarchy (visual tree) +- Abstract methods checklist (17 methods) +- Storage path structure (modern format) +- 2-file system design explanation +- Existing adapters overview (quick stats) +- Factory integration example +- Key inherited features +- Performance characteristics +- Design patterns used +- Conclusion and next steps + +**Read this when:** You're implementing TypeAwareStorageAdapter + +--- + +### 4. STORAGE_FILES_REFERENCE.md (13 KB) +**COMPLETE FILE REFERENCE** - Detailed information about each file + +**Sections:** +1. Core storage files (6 files described) +2. Storage adapter implementations (5 adapters detailed) +3. Storage patterns and utilities (5 utilities listed) +4. Integration points (3 main integration points) +5. Data flow examples (saving and querying) +6. Storage statistics tracking (JSON examples) +7. Type definitions (HNSWNoun, GraphVerb) +8. Summary statistics table (13,000+ lines) + +**Read this when:** You need details about specific storage files + +--- + +## Reading Guide by Use Case + +### I want a quick answer +1. Read this README (you're here!) +2. Read: EXPLORATION_SUMMARY.md - first 30% (Key Findings + Answer) + +### I'm implementing TypeAwareStorageAdapter +1. Read: EXPLORATION_SUMMARY.md (full) +2. Read: STORAGE_ADAPTER_QUICK_REFERENCE.md (Implementation section) +3. Reference: STORAGE_FILES_REFERENCE.md (for specific files) + +### I need comprehensive understanding +1. Read: EXPLORATION_SUMMARY.md +2. Read: STORAGE_ARCHITECTURE_ANALYSIS.md (sections 1-7) +3. Reference: STORAGE_ADAPTER_QUICK_REFERENCE.md + +### I'm debugging storage issues +1. Check: STORAGE_FILES_REFERENCE.md (which file handles what) +2. Check: STORAGE_ARCHITECTURE_ANALYSIS.md (data flow sections) +3. Check: STORAGE_ADAPTER_QUICK_REFERENCE.md (path structure) + +### I'm integrating with storage system +1. Read: STORAGE_ARCHITECTURE_ANALYSIS.md (sections 3-4) +2. Read: STORAGE_FILES_REFERENCE.md (integration points) +3. Reference: STORAGE_ADAPTER_QUICK_REFERENCE.md (as needed) + +--- + +## Key Findings Summary + +### Current State +- 5 storage adapters exist (FileSystem, Memory, S3, GCS, OPFS) +- 27-method StorageAdapter interface +- 17 abstract methods to implement for new adapters +- 13,000+ lines of storage code +- Clean inheritance hierarchy +- Factory pattern for runtime selection + +### Architecture Strengths +- Well-organized and modular +- Factory pattern enables multiple backends +- Interface-based design (no coupling) +- Common base class (code reuse) +- 2-file system (separation of concerns) + +### For TypeAwareStorageAdapter +- Can extend BaseStorage class +- Must implement 17 abstract methods +- Simple factory integration (1 case + interface update) +- No changes to existing code +- Inherits statistics, throttling, caching + +--- + +## Core Facts + +### Storage Layers +``` +StorageAdapter interface (27 methods) + ↓ +BaseStorageAdapter (1,156 lines - common functionality) + ↓ +BaseStorage (1,098 lines - routing & pagination) + ↓ +Concrete Adapters (FileSystem, Memory, S3, GCS, OPFS) +``` + +### Storage Paths +``` +entities/nouns/vectors/{shard}/{id}.json ← vector data +entities/nouns/metadata/{shard}/{id}.json ← flexible metadata +entities/verbs/vectors/{shard}/{id}.json +entities/verbs/metadata/{shard}/{id}.json +_system/statistics.json ← aggregate stats +_system/counts.json ← O(1) totals +``` + +### Sharding +- UUID first 2 hex characters (00-ff) +- 256 shard directories +- Handles 2.5M+ entities efficiently + +### 2-File System +- File 1: Vectors (lightweight, always loaded) +- File 2: Metadata (flexible schema, separately loaded) +- Enables type-aware queries without loading vectors + +--- + +## Implementation Checklist + +For adding TypeAwareStorageAdapter: + +- [ ] Create `/src/storage/adapters/typeAwareStorageAdapter.ts` +- [ ] Extend BaseStorage class +- [ ] Implement 17 abstract methods: + - [ ] saveNoun_internal() + - [ ] getNoun_internal() + - [ ] deleteNoun_internal() + - [ ] saveVerb_internal() + - [ ] getVerb_internal() + - [ ] deleteVerb_internal() + - [ ] writeObjectToPath() + - [ ] readObjectFromPath() + - [ ] deleteObjectFromPath() + - [ ] listObjectsUnderPath() + - [ ] initializeCounts() + - [ ] persistCounts() + - [ ] saveStatisticsData() + - [ ] getStatisticsData() + - [ ] init() + - [ ] clear() + - [ ] getStorageStatus() +- [ ] Update `/src/storage/storageFactory.ts`: + - [ ] Add case for 'type-aware' + - [ ] Update StorageOptions interface +- [ ] Test with MemoryStorage +- [ ] Test with FileSystemStorage +- [ ] Verify existing tests still pass + +--- + +## Quick Reference + +### Files to Analyze +- `/src/coreTypes.ts` - StorageAdapter interface +- `/src/storage/baseStorageAdapter.ts` - Abstract base +- `/src/storage/baseStorage.ts` - Core layer +- `/src/storage/storageFactory.ts` - Factory +- `/src/storage/adapters/memoryStorage.ts` - Simple example + +### Key Classes +- `StorageAdapter` - Interface (27 methods) +- `BaseStorageAdapter` - Abstract base (1,156 lines) +- `BaseStorage` - Abstract impl (1,098 lines) +- `FileSystemStorage` - Concrete impl (2,677 lines) +- `MemoryStorage` - Simple impl (822 lines) + +### Key Methods to Implement +- Node/Verb: saveNoun_internal, getNoun_internal, etc. +- Path: writeObjectToPath, readObjectFromPath, etc. +- Counts: initializeCounts, persistCounts +- Stats: saveStatisticsData, getStatisticsData +- Lifecycle: init, clear, getStorageStatus + +### Design Patterns +1. Factory - `createStorage()` for adapter selection +2. Strategy - Adapters are interchangeable +3. Template Method - BaseStorage defines skeleton +4. Adapter - Maps different backends to same interface +5. Decorator - Can wrap adapters if needed + +--- + +## Analysis Statistics + +| Metric | Count | +|--------|-------| +| Files analyzed | 50+ | +| Lines of code examined | 13,000+ | +| Storage adapters found | 5 | +| Abstract methods to implement | 17 | +| Interface methods | 27 | +| Storage backends supported | 6 (FS, Memory, S3, GCS, OPFS, R2) | +| Documentation pages created | 4 | + +--- + +## Contact & Questions + +For questions about: +- **Architecture:** See STORAGE_ARCHITECTURE_ANALYSIS.md +- **Specific files:** See STORAGE_FILES_REFERENCE.md +- **Quick lookup:** See STORAGE_ADAPTER_QUICK_REFERENCE.md +- **Overall findings:** See EXPLORATION_SUMMARY.md + +--- + +## Conclusion + +Brainy's storage architecture is **professionally designed** and **inherently extensible**. TypeAwareStorageAdapter can be added as a new adapter in just a few minutes by: + +1. Creating a new class extending BaseStorage +2. Implementing 17 abstract methods +3. Registering in the factory + +**No breaking changes required. No existing code needs modification.** + +The architecture supports multiple backends coexisting peacefully through proper use of: +- Interface-based design +- Factory pattern +- Dependency injection +- Abstract base classes + +This is a textbook example of good software architecture. + +--- + +## Document Metadata + +**Created:** October 15, 2025 +**Repository:** Brainy (Neural Database) +**Version:** Analysis of v3.44.0 +**Scope:** Complete storage adapter architecture +**Coverage:** 100% of storage system + +Generated with thorough code analysis and deep understanding of the system. diff --git a/RELEASES.md b/RELEASES.md deleted file mode 100644 index 7799c6f6..00000000 --- a/RELEASES.md +++ /dev/null @@ -1,2901 +0,0 @@ -# @soulcraft/brainy — Release Notes for Consumers - -This file is the **quick reference for downstream sessions** tracking Brainy changes. -Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/soulcraftlabs/brainy/releases - -**How to use:** Brainy is the underlying data engine for downstream applications. Read this when: -- Upgrading `@soulcraft/brainy` in your application -- Debugging data, query, or storage behaviour -- A new Brainy feature is available that you want to adopt - -## Removed APIs — 7.x → 8.x (the complete ledger) - -Every public API removed at the 8.0 major, with its sanctioned replacement. If your code -still calls a left-column name on 8.x it throws (or the config key is rejected) — the -replacement is always a one-line change. (Standing contract from 8.9.0 forward: removals -happen only at majors, after ≥1 minor of loud runtime deprecation naming the replacement.) - -| Removed (7.x) | Replacement (8.x) | -|---|---| -| `brain.search(query, k)` | `find({ query })` — semantic; `find({ query, searchMode })` for hybrid | -| `brain.getRelations({...})` | `related(id, opts)` for adjacency; `find({ connected: {...} })` for scoped traversal | -| `brain.neural()` clustering | `find({ vector })` + aggregation `GROUP BY` | -| `Db.search()` | `db.find({ vector })` | -| Pre-8.0 storage path aliases (`directory`, `basePath`, …) | one `storage.path` key (old aliases throw) | -| Reserved keys inside `metadata` bags (silently remapped in 7.x) | top-level params (`subtype`, `visibility`, `confidence`, `weight`, …) — reserved-in-bag throws | -| 7.x COW branches layout (`branches/main/`) | generational MVCC (`asOf()`, `now()`, `db.persist(path)`) — on-disk migration is automatic at first 8.x open | - -The fork/snapshot family (`brain.snapshot()`, `createSnapshot()`, `restoreSnapshot()`) -is sometimes cited as a 7.x removal — those methods never existed on 7.x; the 8.0 Db API -(`asOf`/`persist`/`restore({confirm})`) is their first real implementation. - ---- - -## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close()) - -The write path stops paying maintenance costs — the last structural piece of the -flush-storm class (a production deployment measured single writes blocked 25–191s behind -history reclaim running inline on flush under memory pressure): - -- **`flush()` never compacts history.** It persists the current window's deltas and - nothing else — its cost no longer depends on history backlog or retention mode, in any - configuration. **`close()` is the auto-compaction site** (time-bounded per pass, ~5s; - an early stop is a consistent prefix and the next pass resumes). -- **`compactHistory()` gains `timeBudgetMs`** — bound your own maintenance windows; the - same resumable-prefix guarantee applies. -- **The documented trade**: a long-lived writer that never closes accumulates history - until its next explicit `compactHistory()`. Predictable writes, explicit maintenance. - If you run bounded retention on an always-on service, schedule a periodic - `compactHistory({ ...caps, timeBudgetMs })` in your maintenance window. -- **New public doc: `docs/performance-envelopes.md`** — measured per-op envelopes - (p50/p95 at stated scales, hardware, and backend, with the measuring script cited). - Refresh rule going forward: any release touching a measured path re-runs that op's - benchmark and updates the envelope in the same release. -- **New in this file: the Removed APIs 7.x→8.x table** (top of this document) — every - removal with its sanctioned replacement, one place, per the engine-currency contract. - Standing from here: removals only at majors, after ≥1 minor of loud runtime deprecation. - -## v8.8.2 — 2026-07-19 (one field-resolution law: reserved-field aggregates stop drifting) - -Four fixes from a consumer conformance audit, all rooted in the same disease — two field-resolution -regimes where there must be one: - -- **Aggregates grouped by a RESERVED field (`subtype`, `visibility`, …) now decrement on - delete.** The delete/update hooks fed the aggregation engine a partial entity view (type, - service, data, metadata only), so a reserved-field `groupBy` resolved to a nonexistent group - on the way DOWN — counts drifted upward forever after any delete, and updates that moved an - entity between reserved-field groups double-counted it. The hooks now pass the full-fidelity - entity view (every reserved field top-level, the same shape the add path uses). If your - deployment derives stats from reserved-field aggregates, re-define those aggregates once - after upgrading (a changed definition triggers one rescan) or run them fresh — the drifted - persisted counts do not self-heal retroactively. -- **Aggregation `source.where` on reserved fields now filters** instead of silently matching - nothing: the matcher resolves fields through the same resolver `groupBy` uses (top-level - standard fields + custom metadata), so `where: { subtype: 'note' }` means what it says. -- **`removeMany()` refuses empty/invalid selectors loudly.** A bare array passed positionally - (`removeMany([id])` instead of `removeMany({ ids: [id] })`), an empty params object, or - `ids: []` used to resolve successfully having deleted nothing. All three now throw. -- **`find()` accepts both where-key spellings.** Metadata is flattened at index time - (`metadata.entry.title` indexes as `entry.title`); a `metadata.`-prefixed where key now - falls back to its flattened spelling when the prefixed one isn't indexed — the - "unindexed field(s), returning []" confusion for storage-shaped spellings is gone. (A - literal nested custom key named `metadata` still wins when indexed as spelled.) - -## v8.8.1 — 2026-07-18 (flush no longer walks the whole generation history + the import dedup off-switch is now honest) - -### The flush-storm fix (production incident, reported by a long-running deployment) - -Under the default adaptive retention, **every `flush()` re-walked the entire committed -generation history** to compute total history bytes for the budget check — O(all -generations) with disk re-reads past the 4,096-entry delta-cache bound. On a brain with -70,000+ accumulated generations that turned every write into a full-tail scan (60-100s -writes), even though the budget (free-RAM-based) never tripped and nothing was ever -reclaimed. Fixed: - -- `historyBytes()` now maintains a **running total**: seeded by one walk on first use, - then updated incrementally at every commit and reclaim — the adaptive retention check - on every flush is O(1). Invariant regression-pinned (running total ≡ fresh walk through - both commit paths and compaction). -- New **`brain.historyStats()`** (read-only, exported `HistoryStats`): generation count, - total on-disk bytes, generation/timestamp range, compaction horizon, retention mode, - and the effective adaptive budget — the one-call fleet-audit for sizing retention - exposure per brain. -- Interim guidance for keep-everything deployments already affected: `retention: 'all'` - skips the adaptive accounting entirely (and is the correct policy if you never want - history reclaimed). The accumulated files are harmless at rest; this release removes - the per-write cost of their existence. - -### The import dedup off-switch (lifecycle honesty) - -The post-import background deduplication pass (a merge-DELETE writer that runs ~5 minutes -after an import, merging entities judged duplicates by id / name / vector similarity) had -three lifecycle defects, all fixed: - -- **`enableDeduplication: false` now actually disables it.** The background pass was - scheduled unconditionally — an import that explicitly opted out could still have - entities auto-removed 5 minutes later. The flag now gates BOTH the inline merge and - the background pass (regression-pinned). -- **One deduplicator per brain, owned by the brain.** Each `import()` call constructed its - own coordinator + deduplicator, so the "debounced" timer never actually debounced across - imports (N imports = N delete timers). The brain now owns a single instance — the - debounce genuinely spans imports — and `close()` cancels pending work, so a delete pass - can never fire against a closed brain. -- **The 5-minute timer is unref'd** — a pending pass no longer holds the process open - (the exit-hang class; this timer had escaped the earlier sweep). - -Retention note for keep-everything deployments: with `enableDeduplication: false` on -import calls and `retention: 'all'` in config, no engine path removes records -automatically. - -## v8.8.0 — 2026-07-17 (OS-limit detection for pool-scale deployments) - -Small minor: brains now detect the two OS limits that bite at pool scale and warn **before** -the incident instead of during it. - -- At open (once per process, Linux-only, measurement-only), Brainy reads `RLIMIT_NOFILE` - (soft/hard, from `/proc/self/limits`) and `vm.max_map_count`, and warns loudly when either - sits below the pool-scale floors (soft NOFILE < 65 536; max_map_count < 262 144) — with the - exact raise commands (`ulimit -n` / `LimitNOFILE=` / `sysctl vm.max_map_count`). On stock - defaults the failure otherwise arrives as `EMFILE` or a failed mmap deep inside an index - open, long after the real cause stopped being visible. An unreadable limit produces **no** - warning — no measurement, no claim (non-Linux platforms stay silent). -- Exported for ops doors: `checkOsLimits()` returns the full `OsLimitsReport` - (values + warnings) programmatically, with the floors exported as constants. - -## v8.7.1 — 2026-07-17 (writer-lock acquisition is race-proof + machine-readable through init) - -Two hardenings of the multi-process writer lock (the `locks/_writer.lock` lease that makes a -second writer on the same brain directory fail loudly): - -- **Lock acquisition claims atomically.** The acquire path used read-then-write, leaving a - window where two processes racing an *absent* lock could both "succeed" — and the loser - kept running unlocked, silently. The claim is now an atomic create-exclusive write - (`O_EXCL`): exactly one racer wins; the loser re-evaluates and either fails loudly with - the winner's details or performs a verified stale-takeover. Bounded retries; contention - beyond them fails loudly rather than degrading into a lockless open. -- **`BRAINY_WRITER_LOCKED` survives `init()`.** The conflict error documents a - machine-readable contract (`err.code`, `err.lockInfo` with the holder's pid/host/ - heartbeat), but init's error wrapping silently stripped both, leaving consumers a message - to regex against. The error now passes through unwrapped. - -Measured while verifying (for operators sizing audits): `brain.auditGraph()` at a -production-consumer scale of ~2,600 relationships / 800 entities costs ~0.1 s warm and -~0.5 s cold, with exact scar counting across reopen. - -## v8.7.0 — 2026-07-17 (bulk-transact ergonomics: scaled budgets + timeout telemetry) - -The bulk-import ergonomics release, from a consumer's measured production incident (a serial -import on network-attached storage at ~2 s/op met a flat 30 s transaction budget): - -- **The transact apply budget now scales with the batch** — `max(30 s, opCount × 2 s)` — or - is exactly what you pass as the new `TransactOptions.timeoutMs`. A flat 30 s cap silently - limited honest bulk work to ~15 operations on slow disks while looking generous for small - batches. Internal batch paths (e.g. `removeMany` chunks) get the same scaling. -- **`TransactionTimeoutError` is a diagnosis, not just a failure**: it now reports the - operation it stopped at as `i/N` with the operation's name, elapsed vs budgeted time, and - states the batch rolled back atomically and is retryable. Its `context` carries the same - fields programmatically. -- **The transact envelope is documented** — batch sizing, budget math, chunking with - `ifAbsent` idempotency, and the precompute pattern (`embedBatch` + per-op `vector`) that - keeps model inference out of the commit path. Guide: `docs/guides/optimistic-concurrency.md`. -- Note: `brain.embed()` / `brain.embedBatch()` (the precompute APIs) already ship — public, - with native-provider passthrough, verified end-to-end (batch and single paths produce - bit-identical vectors; a vector-supplied `add` is fully searchable). Honest measurement: - on the default WASM engine, batch throughput ≈ sequential (~160 ms/text) — the precompute - win is keeping inference out of the budgeted commit path, not raw embedding speed. - -## v8.6.0 — 2026-07-17 (brain.auditGraph — the graph-truth verification instrument) - -A minor release adding one new public API, from the fleet's graph-trust program: a read-only -audit that **proves whether relationship reads return stored truth** on a given brain. - -- **`brain.auditGraph(options?)`** walks every canonical relationship record, queries the same - read path applications use (`related()` / VFS `readdir`) with all visibility tiers included, - and classifies every discrepancy into its failure family: `missingFromReads` (records the - read path omits — a stale adjacency index), `danglingEndpoints` (relationships whose endpoint - entity no longer exists — the historical partial-delete scar class), and `readOnlyVerbIds` - (read-path edges with no stored record — ghosts). Design-hidden internal/system edges are - counted separately so intentional hiding is never misclassified as loss. Counts are exact; - example lists cap at `maxExamples` with an explicit `truncatedExamples` flag; the result is - narrated loudly on incoherence. Mutates nothing — safe on a live brain. -- The operational pairing: audit → if incoherent, `repairIndex()` → audit again. A `coherent` - report after the repair is the verified all-clear. Run it after any engine upgrade, restore, - or migration. Guide: `docs/guides/inspection.md`. -- Also: `getNounIds` pagination now refuses an undecodable resume cursor loudly (the same - contract `getNouns`/`getVerbs` gained in 8.5.2 — the third and final walk brought under it). -- Types exported: `GraphAuditReport`, `GraphAuditDiscrepancy`. - -## v8.5.2 — 2026-07-17 (aggregation backfill: exception-safe, generation-verified, and loud) - -Hardening patch from a migration incident (a byte-copied store on new hardware; the service -entered a silent full-CPU loop at boot). Four changes, all in the aggregation engine's -backfill/adoption path: - -- **Backfill walks are exception-safe and non-destructive.** A rescan now builds into a - staging map and swaps in atomically on completion; a mid-walk failure drops the staging map, - keeps the previous live state serving, and surfaces the storage error to the failing query. - Previously the walk wiped live state *before* a scan that could throw, never cleared the - pending flag on failure, and re-ran a full walk on every subsequent query — a silent - wipe/walk/throw loop at the caller's retry rate. -- **Failed walks are latched.** After a walk fails, retries within a 30-second cooldown rethrow - the recorded error instantly instead of re-walking — a tight caller-side retry loop now costs - one loud error per query, never a full store walk per query. -- **Adoption is generation-verified.** Persisted aggregation state is stamped with the store's - committed generation at flush; reopen adoption requires the stamp to equal the current - watermark. Stale state (unclean shutdown) or over-counting state (a fact-log truncation on a - copied store pulled the watermark back) triggers exactly one loud rescan — never a silent - adopt. Pre-8.5.2 state on generation-aware stores rescans once after upgrade, then is stamped. -- **The path narrates.** Adoption decisions, no-adoptable-state outcomes, walk start/finish - (entity count + duration), and walk failures all log by default; a non-advancing storage - pagination cursor aborts the walk loudly instead of looping forever. - -Plus three guards from a full audit of every loop in the open/init path: - -- **Invalid pagination cursors fail loudly.** A supplied-but-undecodable resume token to - `getNouns`/`getVerbs` used to silently restart the walk at offset 0 — to a `while(hasMore)` - caller that re-serves page 1 forever (an unbounded silent CPU loop). It now throws with a - clear message instead. -- **The graph cold-load verb walk has a stall guard.** `hasMore=true` with a missing or - non-advancing cursor aborts loudly instead of re-reading the same page forever. -- **A derived index AHEAD of the store is named at open.** Brainy already surfaced a provider - generation *behind* the committed watermark; the *ahead* direction (the signature of a - byte-copy of a live service, or a log truncation during crash recovery) now logs a loud - warning explaining what happened and that `brain.repairIndex()` forces a heal — instead of - passing unnamed into whatever the derived index does next. - -## v8.5.1 — 2026-07-17 (aggregation state survives restarts + the query-cap ratchet removed) - -Patch release from a production incident (aggregate/count paths taking 40–90 s on an idle box -while vector search stayed fast, and every `find({ limit: 5000 })` suddenly failing against an -"auto-configured query limit of 1000"). Three fixes, one cosmetic: - -- **Aggregation state is actually adopted on reopen.** The boot pattern `defineAggregate()` → - query raced the engine's async state load: the synchronous define always won, flagged a - backfill, and the first query then wiped the just-loaded persisted state and re-walked the - entire store — every restart, forever. Reopening with an unchanged definition now adopts the - persisted state directly (zero scans); a backfill runs only on a real definition change, a - missing/failed state load, or a write that landed before adoption (exactness wins). Apps that - rely on persisted definitions without re-defining at boot also no longer race a spurious - "Aggregate not defined". -- **Backfills are single-flight and batched.** Concurrent queries on a cold aggregate used to - each wipe the others' partial state and start their own full store walk — under steady query - arrival the store never converged (the 40–90 s loop). Now all concurrent queries share one - walk, and one walk fills every aggregate pending backfill (M aggregates ≠ M scans). -- **The query-cap "learning" ratchet is removed.** `maxLimit` is a memory-protection bound, but - a hidden tuner shrank it 20 % per recorded query while the lifetime-average query time - exceeded 1 s — down to a floor of 1000, below the documented 10 000 auto floor, with no - practical recovery, and the resulting error blamed "available free memory" (stale basis - label). The cap now comes from its construction-time basis (or your explicit - `maxQueryLimit` / `reservedQueryMemory`) alone and never changes at runtime; query timing is - recorded for diagnostics only. -- **Cosmetic:** the engine's own persistence keys (`__aggregation_*`, `brainy:entityIdMapper`) - no longer log `[Storage] Unknown key format` at boot — they were always routed correctly; - they're now recognized before the warning fires. - -Operationally: if a host was bitten, upgrading and restarting is the whole fix — no repair -ritual needed. Setting `maxQueryLimit` explicitly remains the valve that bypasses auto-detection -entirely. - -## v8.5.0 — 2026-07-15 (provider access to the fact log + the shared stamp verifier) - -Small additive follow-up to 8.4.0, from the native accelerator's first consumption pass: - -- **Index providers can now reach the fact log through the storage adapter** — new optional - capability `storage.scanFacts()` / `storage.factLogHeadGeneration()` / `storage.factSegmentPaths()`, - wired by the host brain at init as a closure over its live log. Providers hold only `storage` and - must never construct their own fact-log reader (the log's open path is writer-side); `null` means - "no fact log here — use the enumeration walk." -- **The family-stamp verifier is shared** via `@soulcraft/brainy/internals` - (`readFamilyStamp` / `writeFamilyStamp` / `verifyFamilyStamp` + types) so first-party native - providers run literally the same verification function, never a synchronized copy. -- **Rollup stamp invariants accept strings** (`number | string`) — content fingerprints such as a - per-tree SHA-256 are valid invariant values; a type mismatch reads as incoherence, never a pass. -- **`storage.committedGeneration()`** — the committed watermark as a capability, so a provider - compares its stamp's `sourceGeneration` against the store's truth without parsing the private - manifest format. -- **Two durability/stability contracts pinned in the suite:** fsync-before-ack (holds for - `transact()` today; the single-op path is pinned as the documented future target — group commit - becomes latency batching, never durability skipping) and scan-stability-under-rotation (a scan - snapshot yields exactly its facts — no gaps, duplicates, or bleed-in — while segments rotate - beneath it). - -No behavior change for applications; all additions. - -## v8.4.0 — 2026-07-15 (the generation fact log — a sequential, self-verifying commit stream) - -A minor release, fully backward-compatible (all additions; no behavior change for existing APIs). -This is infrastructure: it changes nothing about how you query today, and lays the substrate that -makes index heals and incremental catch-up sequential-read problems instead of directory walks. - -- **Every committed write now also appends a "fact" — an after-image commit record.** Alongside the - existing before-image history, each committed generation (single-op and `transact()` alike) appends - what each touched entity/relationship *became* — or a body-less tombstone for a removal — to an - append-only, checksummed segment log under `_generations/facts/`. Crash-safe by construction: a - torn tail is detected and ignored; on open the log is reconciled to committed truth, so an absent - generation always means "never committed." `transact()` facts are durable when `transact()` - returns; single-op facts ride the same group-commit flush as their history. - -- **New: `brain.scanFacts()`** — stream committed facts in commit order, in batches, with heal-grade - telemetry (total scope up front; per-batch generation range, byte size, and segment; a summary - cross-check at the end; loud abort on any gap — never a silent skip). **`brain.factSegmentPaths()`** - hands zero-copy consumers the immutable sealed segment files directly. New exported types: - `CommitFact`, `FactOp`, `FactScanBatch`, `FactScanHandle`. Facts accumulate from the first write - after upgrading — pre-existing history is not retroactively converted (enumeration remains the - fallback for old data). - -- **New: the entity-tree family stamp.** At every flush/close, brainy stamps which committed - generation the canonical entity files reflect plus the rollup invariants (entity/relationship - counts) that verify the tree whole. At open, coherence is a comparison — a genuine divergence is - loud and names the failing invariant; `repairIndex()` recounts from canonical and re-stamps. New - exports: `readFamilyStamp`, `verifyFamilyStamp`, `ENTITY_TREE_STAMP_PATH`, `FamilyStamp` types. - -- **Storage adapters** gain optional binary raw-byte primitives (`appendRawBytes`, `readRawBytes`, - `writeRawBytes`, `rawByteSize`) — feature-detected; the filesystem and memory adapters implement - them; an adapter without them simply hosts no fact log. The fact-log namespace is registered as a - protected family: no sweeper or GC can delete under it. - -No API breaks. 24 new tests; the full commit-path regression suite is green. - -## v8.3.3 — 2026-07-15 (rename moves the containment edge — no ghost in the old directory) - -One production-reported fix plus a repair path, completing the delete/move hygiene arc (8.3.1 fixed -deletes, 8.3.2 fixed counters, this fixes moves). - -- **A cross-directory `vfs.rename()` now MOVES the containment edge instead of accumulating one per - parent.** The old parent's `Contains` edge was never removed on a move, leaving the entity a child - of **both** directories: `readdir(oldDir)` kept listing it after the move, re-creating the old path - showed the same name twice, and any tree-walking consumer (sync engines, file browsers) saw the - file in two places. The old edge is now removed by edge id, resolved from the graph's own adjacency - — a removal never requires reading the thing being removed. Bonus fix in the same seam: a move **to - the root** now gets its containment edge (it was previously skipped, orphaning the file out of - `readdir('/')`). - -- **`repairIndex()` now also reconciles VFS containment** (new `vfs.repairContainment()`): every VFS - entity's containment edges are checked against its canonical `metadata.path` — stale old-parent - ghosts and duplicate edges are removed, a missing expected edge is restored, and user - knowledge-graph edges are never touched (only `vfs-contains` edges are candidates). Loud per - repair. Stores that performed cross-directory renames under ≤8.3.2 should run `brain.repairIndex()` - once after upgrading — the same single ritual now heals orphan directories, counters, **and** - containment edges. - -- Also ships a permanent lens-consistency regression suite (combined type+subtype vs subtype-only vs - canonical ground truth, id-for-id, warm and after a cold reopen), ported from the field - investigation that closed the historical lens-drop report. - -No API changes beyond the new optional `vfs.repairContainment()` (also invoked by `repairIndex()`). - -## v8.3.2 — 2026-07-14 (honest counters — the recount + removal-without-re-reading) - -Completes 8.3.1's delete-hygiene story at the counter layer, from a production proof chain reported -by a downstream deployment: persisted entity totals were permanently **inflated** — deletes whose -count decrement was silently skipped — and because paginated `totalCount` serves -`Math.max(persistedTotal, scanned)`, the inflated number always won and **no disk cleanup could ever -lower it**. - -- **A removal's count decrement no longer requires re-reading the record being removed.** The - decrement was sourced from re-reading the entity's metadata inside the delete; if that read - returned `null` (a replace race, or a ghost left by a pre-8.3.1 partial delete) the decrement was - silently skipped while the paired add had counted — minting drift on every write→delete→re-create - cycle. The caller's pre-delete read now rides through the whole delete path - (`remove()`/`removeMany()` → the delete operation → `deleteNoun`/`deleteVerb` → - `deleteNounMetadata`/`deleteVerbMetadata`, both sides symmetric): a null internal read falls back - to the known prior record instead of skipping. The `StorageAdapter` signatures gain an optional - `priorMetadata` parameter (additive; existing adapters unaffected). - -- **`repairIndex()` is the sanctioned counter recount — unconditional, and it actually persists.** - `rebuildTypeCounts()` previously rebuilt only the type-statistics arrays and computed the total - *just to log it* — the persisted scalar (`counts.json`) survived every "rebuild" untouched, so an - already-inflated brain could never be corrected. One canonical walk now rebuilds **every** counter - rollup — scalar totals, per-type maps, and type statistics — and persists them, and `repairIndex()` - runs it unconditionally (not only when orphan directories are found: counters can be inflated over - perfectly clean shelves). Brains with delete history should run `brain.repairIndex()` once after - upgrading; the correction survives reopen. - -No API breaks (optional-parameter additions only). Regression tests cover the drift cycle, the -null-read decrement fallback, and the persisted recount across reopen. - -## v8.3.1 — 2026-07-14 (full-removal deletes + family-scoped migration gate) - -Two production-reported fixes in the write/index spine, plus an operator repair path. No API changes; -all behavior changes make previously-wrong states honest. - -- **Deleting an entity now removes it completely — no more "ghost" leftovers on disk.** A canonical - noun delete removed the metadata (content) leg but left the entity's `vectors.json` and its `/` - directory behind. Consequences observed in a production deployment: deleted rows were - indistinguishable on disk from damage scars, enumerated counts inflated monotonically with every - delete (the leftovers were counted forever), and locator-style reads hit unreadable ghost rows. - `remove()`/`removeMany()`/`deleteNoun`/`deleteVerb` now remove **both legs and the entity - container**, with a full two-leg before-image rollback inside the transaction. The generation log - still holds the delete's before-image, so `asOf()` time-travel reconstructs deleted entities exactly - as before — this is live-HEAD hygiene, not a history change. This also fixes **duplicate `readdir` - entries for re-created VFS paths** at the root: with no ghost state, a delete always unposts its - index rows, so a delete→recreate cycle lists the path exactly once (regression-tested across - repeated cycles). - -- **`repairIndex()` prunes ghost/scar directories left by earlier versions.** Stores that deleted - entities under ≤8.3.0 may hold orphaned entity directories (a vector-only leg, or an empty dir). - `brain.repairIndex()` now sweeps them: it removes only containers with **no metadata content leg** - (never a directory that still holds content), logs every removal, and recomputes type/subtype - counts afterward so totals stop counting ghosts. - -- **Reads no longer hang behind an unrelated index migration (family-scoped gate).** During a native - provider's one-time background migration, *every* read — including plain `get()`, VFS - `readdir`/`readFile`, and metadata-only `find({ where })` — blocked on the whole-brain migration - lock until timeout, even when the migrating index was irrelevant to the read. The gate is now - scoped to the index families a read actually consults: canonical reads (`get`, `batchGet`, VFS - content) never wait; a `find` waits only on the families its query shape needs (vector for - semantic, metadata for `where`/type, graph for `connected`); graph traversals wait only on the - graph family. Writes and unclassified operations keep the conservative whole-brain wait. A read - that *does* need the migrating family still blocks (bounded by `migrationWaitTimeoutMs`) and - surfaces the retryable `MigrationInProgressError` — never a partial result. - -No breaking API change. Each fix ships with regression tests. - -## v8.3.0 — 2026-07-13 (faster index heals + the cross-layer integrity contract) - -Three additive changes. The first is an immediate, standalone performance win; the other two are the -brainy side of the write/index-spine integrity contract, inert until a native accelerator -that implements the matching hooks is present — so this release changes nothing for a JS-only brain -beyond the speedup. - -- **Canonical enumeration is up to ~16× faster — the dominant term in an index heal.** The paginated - entity walk (`getNounsWithPagination`) hydrated each entity's vector + metadata one-at-a-time; since - every index rebuild enumerates canonical storage, that serial per-item latency dominated multi-minute - heals. Hydration is now 16-way bounded-concurrency, with the pagination contract (order, cursor - resume, filters, totalCount) byte-identical to before. New **`getNounIdsWithPagination()`** returns - ids without hydrating anything (zero per-entity reads when unfiltered) for callers that own their own - IO schedule. - -- **Cross-layer integrity — `validateIndexConsistency()` is no longer blind to native providers.** It - only ran the JS metadata index's own check, so a native provider whose manifest/segments/counts had - diverged still read as "healthy". It now feature-detects and aggregates each provider's optional - `validateInvariants()` self-report, names every failing invariant with its numbers, and exposes the - per-provider reports. `repairIndex()` now reconciles native derived state from canonical too - (rebuilding any provider whose failing invariant asks for it). New exported types - `ProviderInvariantReport` / `InvariantResult` / `InvariantHeal`. - -- **Registered-blob families — declared index files are undeletable through the storage layer.** A - provider can declare a derived-index blob *family* (a set of members that are load-bearing together); - once declared, `deleteBinaryBlob` / `removeRawPrefix` refuse to remove a member (new exported - `ProtectedArtifactError`), so a stray in-process sweeper cannot delete a load-bearing index file. The - declaration persists across reopen; `checkDerivedFamiliesPresent()` names any member missing on open. - New optional `StorageAdapter` surface (`registerDerivedFamily` / `unregisterDerivedFamily` / - `listDerivedFamilies` + `DerivedFamilyDeclaration`) and exported `DerivedArtifactMissingError`. - -No breaking API change (all additions are optional/new). Each change ships with regression tests. - -## v8.2.8 — 2026-07-13 (honest index readiness — no more silently-empty queries on a cold index) - -Closes the last of the three spine anti-patterns: the "dishonest readiness proxy," where `size() > 0` -was treated as "this index actually serves queries." On a cold open (fresh boot, restart, crash -recovery) a native index can load its **count** before its **serving structure** — so for a brief -window it has data but cannot answer, and a query returned a silent empty result indistinguishable -from "no such data." Every fix here asks the index whether it can *actually serve*, and if not, -self-heals or fails loudly instead of returning `[]`. - -- **Semantic search no longer returns a silent `[]` on a cold vector index.** A pure semantic - `find({ query })` has no filter, so nothing previously guarded the vector index. A new one-shot - guard verifies the vector index serves a known persisted vector on the first semantic/proximity - search — preferring the provider's honest `isReady()` signal, else a known-vector self-match probe. - It rebuilds from canonical records if the serving structure did not load, and throws the new - **`VectorIndexNotReadyError`** only if a rebuild still cannot serve — never a silent empty result. - -- **Relationship reads fall back to the canonical scan instead of an empty result on a cold graph - index.** `getVerbsBySource`/`getVerbsByTarget` (used by relationship queries and virtual-filesystem - traversal) skipped the fast path only on `isInitialized` — which reads true once the manifest loaded - even if the source→target adjacency did not. They now consult the honest readiness signal and, when - the adjacency is not serving, take the correct-but-slower canonical shard scan. A one-shot self-heal - probe covers providers that expose no readiness signal. - -- **`getIndexStatus()` tells the truth for readiness probes.** It reported `populated: size>0` only, so - a Kubernetes readiness check could route traffic to a brain still warming up. It now folds in the - honest per-index `ready` signal (making `populated` honest), plus the degraded states already - surfaced by `checkHealth()`/`validateIndexConsistency()` (`rebuildFailed`/`rebuildError` and a - `degradedIds` count) — so a probe never reports 200-ready over a known-degraded index. - -This completes the write/index-spine hardening end to end. New export: **`VectorIndexNotReadyError`**; -`getIndexStatus()` gains additive fields (`rebuildFailed`, `rebuildError?`, `degradedIds`, per-index -`ready?`). No breaking API change. Each fix ships with a dedicated regression test. - -## v8.2.7 — 2026-07-13 (loadBinaryBlob fault-propagation — the lockstep completion of 8.2.6) - -Completes the Pass-1 spine hardening. `loadBinaryBlob` (the raw-blob read a native accelerator mmaps -for its index files) previously returned `null` on ANY read error, so a real IO fault -(EIO/EACCES/EMFILE) on a present-but-unreadable index blob masqueraded as "the blob is absent" — -driving a needless full rebuild or an empty read. It now distinguishes genuine absence (ENOENT → -`null`, the documented contract) from a real fault (→ throw), so a transient disk fault surfaces -loudly instead of silently degrading the index. - -This one change was deliberately held out of 8.2.6 and ships now, in lockstep with the native -accelerator release that hardened its two column-store read sites to handle the throw (a faulted -segment read marks the field unavailable and throws a named error, instead of relying on -null-on-error). A consumer on new brainy + an older accelerator was never at risk: 8.2.6 kept the -prior swallow-on-fault behavior for this method until the accelerator was ready. - -No API change. Regression: `tests/unit/storage/blob-save-durability.test.ts` gains the loadBinaryBlob -leg (absent → null; present → bytes; real fault → throws). - -## v8.2.6 — 2026-07-13 (write/index-spine hardening — loud errors, never quiet losses) - -Durability + integrity hardening across the write and index paths. Every fix converts a place that -could *silently* lose data, serve a partial result, or acknowledge a write that did not land into a -loud, observable failure. No breaking API changes; one new exported error. - -- **A durable blob write that stored nothing is no longer acknowledged.** The atomic blob write - (`saveBinaryBlob`, used for vector/graph index segments and the native index files) uses a unique - per-writer temp file, so a rename `ENOENT` can only mean *our* temp vanished before the rename — - the bytes never landed. It previously returned success on that path; it now retries once with a - fresh temp and, if the temp vanishes again, throws instead of acknowledging a write that persisted - nothing. A downstream deployment that mmaps these files no longer finds a "successfully written" - blob missing on the next open. - -- **Single-op history durability is enforced, not assumed.** The asynchronous group-commit flush - that persists single-op generation history previously swallowed a persist failure as a warning - while writes kept succeeding and their history piled up, undurable, in memory. It now tolerates a - transient blip (retry with capped backoff) and, after repeated failures, **refuses further writes** - with the new **`PendingFlushDurabilityError`** rather than promise a durability it cannot deliver. - Live canonical data is untouched; the latch self-heals the moment a flush succeeds. Callers needing - hard per-write durability should keep using `transact()` (or `flush()` after a single-op). - -- **A degraded derived index is surfaced on reads, not served silently.** When a non-fatal index - rebuild fails at open, or a write commits via adopt-forward recovery with an incomplete derived - index, `find()` and `get()` now emit one loud warning per degraded window (reads still return — - canonical is the source of truth), the state folds into `checkHealth()` / - `validateIndexConsistency()`, and `repairIndex()` reconciles and clears it. - -- **`clear()` removes the full derived footprint.** It previously left raw index blobs, the native - id-mapper, and column-index manifests on disk, so a cleared brain could re-read stale native state - on reopen. It now wipes them together as a set. - -- **Counts stay honest across deletes.** A delete decremented the per-type breakdown but not the - scalar total, so the total inflated permanently (and won pagination). Delete now decrements both; - the invariant `total === Σ per-type` holds across any interleaving of add / visibility-flip / - delete and across a reopen. - -- **Index-maintenance and aggregation failures are loud.** A partial LSM/segment load no longer - publishes the manifest's full count as if healthy; an HNSW flush that can't persist a node throws - (`HnswFlushError`) instead of returning a lying count and dropping the node; a corrupt or missing - manifest-listed column segment throws (`ColumnSegmentLoadError`) instead of silently dropping its - entities from every query; and aggregation materialization / state-load failures now warn instead - of vanishing into an empty catch. - -New export: **`PendingFlushDurabilityError`** (with `.cause` and `.failedAttempts`). No other public -API change. Each fix ships with a dedicated regression test. - -## v8.2.5 — 2026-07-12 (honest response when a transaction rollback can't complete) - -Data-integrity fix. When a transaction failed and its rollback then *also* failed to undo a -canonical write (retries exhausted), the old behavior logged, continued, and threw -`TransactionRollbackError` — while the record it couldn't undo stayed durable on disk, and the -transaction even reported its state as `'rolled_back'`. The caller got an error implying the write -was undone; a read-back showed the record. A failed rollback had no truthful response. - -Rollback now tells the truth, with a two-branch contract: - -- **Adopt-forward (safe case).** A single-op write (`add`/`update`/…) whose only damage is a - durably-present record — the write the caller asked for — is **adopted**: its generation is - committed, the write returns success, and a loud warning records that the derived index may be - incomplete for that id until the next rebuild/`repairIndex()`. No error, no double-write; the - record is immediately durable and retrievable by `get()`. -- **Fail loud + quarantine (unsafe case).** A multi-operation batch, or *any* case where a record - was lost (a remove/update whose restore-undo failed), throws the new **`StoreInconsistentError`** - naming every unreconciled record and its disposition (`orphan` vs `loss`), and puts the brain into - **write-quarantine**: reads keep working, but writes are refused until `repairIndex()` reconciles - the derived indexes against canonical storage and lifts the quarantine. The generation counter is - not advanced, and a transaction whose rollback failed is now `'inconsistent'`, never the - `'rolled_back'` lie. - -The decision is made by *observation*, not guesswork: both commit paths already hold byte-identical -before-images, so after a failed rollback the store compares current canonical state to them to -classify exactly which records are orphaned or lost. - -New export: `StoreInconsistentError` (with `.records` and `.cause`) and the `UnreconciledRecord` type. -No other API change; `repairIndex()` gains the quarantine-lift behavior. Regression -(`tests/integration/rollback-trapdoor.test.ts`) injects the exact failure (index add throws → -canonical delete-undo fails) and pins adopt-forward (durable, get-able, not quarantined), fail-loud -(`StoreInconsistentError` + quarantine + reads work + `repairIndex()` lifts it), and the error's -record naming. - -## v8.2.4 — 2026-07-12 (restore can no longer destroy the store it's recovering) - -Recovery-safety fix. `restore()` removed the entire live brain directory and THEN copied the -snapshot in — so any copy failure left the store destroyed with only a partial copy. The sharpest -edge: the copy (`fs.cp`) materialized the holes of sparse mmap blob files, so a snapshot that fits -on disk could balloon and `ENOSPC` mid-copy, and the recovery tool would have just destroyed the -brain it was asked to recover. Reported from a downstream incident's recovery forensics. - -Restore is now **non-destructive and crash-resumable**: - -- The snapshot is copied into a staging area (`_restore_staging/`) **before any live data is - touched**. The copy is **sparse-aware** — all-zero regions are left as holes, so a store of - mostly-hole blobs restores at its true allocated size instead of its apparent size. -- A copy failure (including `ENOSPC`) removes only the half-written staging area and throws; the - live store is left **exactly as it was**. -- Only after the copy succeeds and a completion marker is `fsync`'d does an **atomic per-entry - swap** move the staged data into place — same-filesystem renames that cannot fail for disk space. -- A crash mid-swap is finished **forward** on the next open: startup resumes a committed-but- - incomplete swap, or discards an uncommitted staging area (live data still authoritative). - -No API change — `restore(path, { confirm: true })` is unchanged. `persist()` was already safe -(hard-link snapshot). Regression (`tests/integration/restore-nondestructive.test.ts`): a forced -copy failure leaves live data fully intact, a normal restore round-trips, an interrupted-but- -committed restore completes on reopen, an uncommitted staging area is discarded, and the sparse -copy is byte-identical with allocation far below apparent size. - -## v8.2.3 — 2026-07-12 (a committed transaction is durable on return) - -Durability fix. A `transact()` reported "committed" while its canonical entity writes were still -only in the OS page cache (written via tmp+rename, not yet `fsync`'d), even though the generation -counter and manifest WERE fsync'd. A hard kill (power loss, SIGKILL) in that window could leave the -durable generation counter **ahead of** the persisted entity bytes — so a consumer resuming from the -counter would see "phantom progress": a generation that claims writes the disk never kept. Reported -from a downstream migration's crash-lifecycle forensics. - -`commitTransaction` now runs a **durability barrier**: it records every canonical write and delete -the batch's operations make, then `fsync`s that entire footprint (file contents, the rename -directory entries, and the parent directories of any deletes) **before** advancing the generation -counter and manifest. A committed transaction is therefore durable the moment `transact()` returns — -the counter can never outrun the entity bytes. - -**Durability contract, now explicit.** `transact()` is durable-on-return (above). A **single-op** -write (`add`/`update`/`remove`/`relate`/…) is Model-B group-commit: its live bytes are written but -become durable at the next `flush()` or `close()`, not the instant the call resolves — the counter -is buffered alongside the data, so a crash loses both together (never a torn counter-ahead-of-state -store). Need per-write durability? Use `transact()` (even for one op), or `flush()` after the write. -This deliberately trades single-op fsync latency (a 3-5x write regression) for throughput. - -No API change; no accelerator involvement (the barrier is in the filesystem storage adapter). In-memory -and durable-per-call (cloud object-PUT) adapters treat the barrier as a no-op. Regression: -`tests/integration/transact-durability-barrier.test.ts` proves entity writes fsync in an earlier -batch than the manifest, for single-op and multi-op (add+relate) transactions, and that a -precommit-rejected batch opens no barrier and advances nothing. - -## v8.2.2 — 2026-07-11 (P0: a timed-out transaction now rolls back — no torn state) - -Data-integrity fix. A transaction that exceeded its time budget **mid-flight** (e.g. a bulk -`transact()` on slower hardware crossing the 30s ceiling) threw a timeout error WITHOUT rolling -back the operations it had already applied. The budget check sat outside the per-operation -rollback path, so only per-operation *failures* rolled back — a timeout stranded the partial -writes in canonical storage while the generation was never stamped, leaving torn, generation-less -state. This was caught by a downstream migration that crossed the ceiling on a large batch. - -`Transaction.execute()` now has a **single rollback point**: any error that escapes the operation -loop — an operation failure OR a mid-flight timeout — rolls back every applied operation in -reverse order, then surfaces the original error (a rollback failure supersedes it, loudly, as -before). This restores the invariant the generation-store commit path already depended on: a throw -from execute means the applied operations were undone byte-identically, and the aborted -transaction leaves `generation()` unchanged and storage byte-identical to its pre-transaction -state. - -No API change. Regression pins the reported requirement — after a mid-flight timeout, -storage is byte-identical and the transaction ends in the `rolled_back` terminal state -(`tests/unit/transaction/timeout-rollback.test.ts`), plus the operation-failure and -rollback-failure paths through the same single rollback point. - -**Note on the 30s ceiling itself** (configurable/scaled timeout, batched embedding precompute, -timeout telemetry) — that ergonomics work is tracked separately; this release fixes only the -correctness bug (a timeout must never leave partial state), independent of where the ceiling sits. - -## v8.2.1 — 2026-07-11 (transact forward references work on the native accelerator) - -Parity fix. `transact()` has always promised atomic forward references — `add` an entity and -`relate` to it in one batch — but the transaction planner resolved relationship endpoint ids at -**plan** time, before the batch's adds had applied. Asking the id mapper about an entity that -doesn't exist yet made the accelerator's (correctly strict) native mapper throw in -`EntityIdMapper.getOrAssign`, so `transact([{op:'add', id:X}, {op:'relate', to:X}])` failed on -native deployments while the permissive JS mapper masked the bug — and silently leaked an id -assignment whenever a batch was later rejected by a commit precondition. - -Endpoint resolution is now **lazy** — evaluated when the graph operation executes inside the -commit, after the batch's adds have applied (the same lazy pattern the operation's generation -stamp already used). Fixed across all transact-planned graph operations: relate (including the -bidirectional reverse edge), remove's relationship cascade, and unrelate. Single-operation writes -are unchanged. Also fixed as a byproduct: a rejected batch no longer pollutes the id mapper. - -Regression suite covers the exact reported shape, both-endpoints-in-batch, bidirectional, -add+relate+remove in one batch, and the mapper-cleanliness proof on rejected batches -(`tests/integration/transact-forward-ref-graph.test.ts`). No API changes; no accelerator version -pairing required. - -## v8.2.0 — 2026-07-10 (temporal VFS — file content joins the immutability model) - -Time travel now covers Virtual Filesystem **content**. Previously the temporal model had a hole -exactly where files were concerned: every entity write was an immutable generation, but the -content *bytes* lived under an eager reference-count GC — deleting a file could physically destroy -bytes that in-window history still referenced, while overwriting never released the old content at -all (an unbounded silent leak that only *accidentally* preserved history). A production consumer's -recovery from a bad deploy succeeded only because a stale field happened to hold the good value — -luck, not a guarantee. Both directions are now fixed by making blob reclamation a **history** -decision instead of a **liveness** decision: - -- **Content blobs are retention-protected.** Each blob tracks live references AND history - references (one per persisted generation record that carries its hash). Deleting/overwriting a - file drops only the live reference; bytes are physically reclaimed in exactly one place — - history compaction — once no live reference and no retained generation references the hash. - Pinned views are exempt automatically (compaction already respects pins). Crash ordering is - over-count-only (never under-count), so a crash can leak until the built-in scrub recounts, but - can never reclaim bytes history still needs. Existing stores get a one-time, marker-gated - backfill on open; cross-file content dedup is handled exactly. -- **`vfs.readFile(path, { asOf })`** — the file's exact bytes as of a generation or `Date`, - guaranteed present within the retention window. -- **`vfs.history(path)`** — the file's versions (`{ generation, timestamp, hash, size, - mimeType? }`, ascending). Restore = read the old bytes `asOf` and write them back (a new write; - history is never rewritten). -- **Overwrites now refresh the file entity's `data`/embedding text** — previously semantic search - and the `data` field served the FIRST version's text forever. - -Lifecycle note: deleting a file no longer frees its bytes immediately — old content lives until -compaction reclaims its generations, under the same `retention` budget as all Model-B history -(`retention: 'all'` keeps every version forever). Guide: the new "Time travel for files" section in -`docs/guides/snapshots-and-time-travel.md`. Native accelerator: unaffected (canonical write path -only) — no version pairing required. - -## v8.1.0 — 2026-07-10 (`brain.onChange` — the in-process change feed) - -New public API: subscribe to every committed mutation with -`brain.onChange(cb) → unsubscribe`. One event per affected record, for **every** -canonical write regardless of origin — direct calls, batch methods, `transact()`, -imports, and Virtual Filesystem writes all funnel through the same commit point the -feed is emitted from. This is the authoritative in-process signal for live UIs, -cache invalidation, and realtime sync layers (downstream SDKs can forward it over -their own transports to make local and remote brains uniform). - -Event shape (`BrainyChangeEvent`, exported): `kind` (`entity`/`relation`/`store`), -`op` (`add`/`update`/`remove`/`relate`/`unrelate`/`updateRelation`/`clear`/`restore`), -the post-commit `entity` or `relation` view (type + full custom metadata), the -committed `generation`, and `timestamp`. Notable properties: - -- **Post-commit only** — an aborted write (a losing `ifRev` CAS, a rejected - transaction) never emits. If you got the event, the write is durable. -- **Deletes fully described** — `remove`/`unrelate` carry the record's last - committed state (from the commit's own history record), not just an id; batch - deletes included. -- **Batches emit per item**; a `transact()` batch's events share its single - generation. Entity deletes also emit `unrelate` for each cascaded relationship. -- **Commit-ordered, asynchronous, isolated** — events arrive in commit order, - dispatched in a microtask so a slow listener never delays a write and a throwing - listener never affects the write or other listeners. Zero overhead with no - subscribers. -- **`kind: 'store'`** events for `clear()`/`restore()` mean "refetch everything". -- Fire-and-forget by design; each event's `generation` composes with - `asOf()`/`transactionLog()` for catch-up after a gap. - -Guide: `docs/guides/reacting-to-changes.md`. No behavior changes for existing code. - -## v8.0.17 — 2026-07-08 (count recovery scans the real layout · ~1,100 lines of dead 7.x machinery removed) - -A cleanup of the vestigial 7.x "hnsw sharding" machinery that turned up two real fixes. - -**1 — Count recovery now scans the layout the database actually uses.** When `counts.json` is -lost or corrupted (container restarts, partial copies), the recovery scan rebuilt the entity/ -relationship counters by counting files in `entities/*/hnsw/` — directories the 8.0 write path -never populates — so an established store recovered to **zero counts** on real data: wrong -`getNounCount()`/stats, a "New installation"-style boot log, and a mis-sized rebuild-strategy -decision at open. The scan now counts the canonical `entities////` tree. -Regression-tested: delete `counts.json` from a populated store → reopen → exact counts. - -**2 — A latent init-time deadlock removed.** The recovery scan's type-distribution sampler called -a guarded accessor (`getNounMetadata` → `ensureInitialized`) from **inside** `init()`, which -re-enters `init()` and hangs the open. It was unreachable before only because the old scan never -found anything to sample; the fix reads the sampled metadata files directly. - -**3 — The dead machinery itself is gone (−1,138 lines).** Twenty-three methods with zero callers: -the 7.x hnsw-layout entity/edge CRUD, the sharding-depth prober + depth-migration engine, and an -orphaned streaming paginator. Verified by reachability analysis before deletion; no public API -touched; the full suite is green. New stores no longer carry the always-empty legacy directories' -scan cost, and the boot log keeps the truthful new-vs-established wording from v8.0.13. - -## v8.0.16 — 2026-07-08 (concurrency fast-follow: atomic `ifAbsent`/`upsert` + exact blob reference counts) - -Closes the two remaining check-then-act races found in the sweep that followed v8.0.15's CAS fix -(disclosed in that release's notes). Same bug class — a check performed before the serialization -point that guards the apply — same cure. - -**1 — `add({ ifAbsent })` and `add({ upsert })` are now atomic.** The absence check ran before the -commit mutex, so N concurrent same-id creates could all pass it and all write — the second -silently overwriting the first, violating ifAbsent's "returns the existing id **without writing**" -contract and upsert's "merge, never clobber" contract. The insert leg now carries a must-be-absent -precondition (the same conditional-commit primitive as `ifRev`), verified under the commit mutex: -exactly one concurrent create wins; every other caller takes its documented resolution — ifAbsent -returns the existing id with zero writes, upsert merges into the now-existing entity (with a -bounded retry if a concurrent delete intervenes). Verified: 8 concurrent `ifAbsent` creates -advance the store by exactly ONE generation; 8 concurrent upserts on an absent id produce one -create + seven merges (`_rev` lands at exactly 8). Note: `transact()`'s batch-level -ifAbsent/upsert keep planning-time semantics (the batch converges to a valid entity; the window is -documented, not silent). - -**2 — Blob reference counts are exact under concurrency.** The content-addressed blob store's -`write()` (dedup: exists → add a reference, absent → create) and `delete()` (decrement → remove at -zero) were unserialized read-modify-writes over the blob's metadata. Concurrent writes of -identical content could lose references — making a later delete remove bytes **another file still -referenced** (data loss), or leak unreferenced blobs. All reference-count-bearing mutations are -now serialized per content hash (distinct content never contends): N concurrent identical writes -yield exactly N references, and a blob is physically removed only when the true last reference -drops. Verified with concurrent write/delete storms. - -Both fixes are in-process complete by construction — storage enforces single-writer-per-directory, -so the process is the whole concurrency domain. No API changes. - -## v8.0.15 — 2026-07-08 (`ifRev` CAS is now atomic — exactly one winner under concurrency) - -Correctness fix for optimistic concurrency, reported from a production cutover rehearsal. N -**concurrent** `update({ ifRev })` calls carrying the same expected revision ALL succeeded — zero -`RevisionConflictError`s, last-writer-wins, the other N−1 writes silently lost. (Sequential calls -conflicted correctly.) The revision check ran before the commit mutex, so interleaved callers all -passed it before any apply landed — breaking the exactly-one-winner semantics the -[optimistic-concurrency guide](docs/guides/optimistic-concurrency.md) promises, which advisory -locks and per-entity ledgers/counters build on. - -The fix makes the check-and-apply a **conditional commit**: the generation store's commit paths -accept a precondition that runs under the commit mutex against the just-read authoritative -before-images — the per-record analogue of `ifAtGeneration`, which always ran there. `update()` -and `transact()` per-op `ifRev` both re-verify at that point (the earlier check remains as a cheap -fast-fail). A conflict aborts atomically: nothing staged, nothing applied, the same -`RevisionConflictError` as before. Two adjacent behaviors also became honest: - -- **`_rev` is now monotonic under concurrency.** The winner's stamp derives from the - authoritative before-image, so N concurrent plain updates advance `_rev` by N (previously they - could all stamp the same stale value). -- **CAS against a concurrently-deleted entity** now throws `EntityNotFoundError` instead of - silently resurrecting it (plain updates keep their last-writer-wins re-create semantics). - -Verified with a concurrency regression suite: 8 parallel same-rev updates → exactly 1 winner + 7 -conflicts; the documented read→CAS→retry ledger loop converges exactly (8 workers, 8 decrements, -0 lost) — `tests/integration/ifrev-concurrent-cas.test.ts`. If you serialized writes in your own -adapter as a workaround, it can come out after this upgrade. - -## v8.0.14 — 2026-07-07 (7→8 migration preserves branch-scoped non-entity state instead of deleting it) - -Defense-in-depth for the one-time 7→8 layout migration. The migration rescues the head branch's -entities (`branches//entities/*` → `entities/*`) and then drained the whole `branches//` -directory. If a 7.x engine had written durable state under the head branch *outside* `entities/` -(a branch-scoped index, blob area, or field registry), that drain would have silently deleted it — -the same failure class as the VFS content blobs a 7.x store kept in `_cow/`. The migration now drains -the branch only when nothing but the moved entities remains; if any non-entity object survives, it -**preserves the branch** (no delete) and logs a loud warning naming the leftover keys, so the state is -recoverable rather than lost. No effect on a normal migration (a clean branch still drains); purely a -guard against silent data loss. - -Cosmetic boot-log fix. Every persisted 8.0 store logged `📁 New installation: using depth 1 -sharding` on **every** open — even established brains holding thousands of entities — which is -alarming to read during a restart or incident. Cause: 8.0 stores nouns in the canonical -`entities/nouns///vectors.json` layout, but the legacy sharding probe inspected the -`entities/nouns/hnsw/` directory, which the 8.0 write path never populates, so it always concluded -"new". The new-vs-existing decision now consults the layout the database actually reads and writes -(plus the known noun count), so an established store logs `📁 Using depth 1 sharding (N entities)` -and only a genuinely empty store reports a new installation. No behavior change beyond the log line — -it never triggered a rebuild or migration; entities were always read correctly. - -## v8.0.12 — 2026-07-07 (7→8 VFS-content recovery · zero-rebuild cold open · strict query operators) - -Three consumer-facing fixes. - -**1 — A 7→8 upgrade recovers Virtual Filesystem content automatically (data-integrity).** -7.x stored VFS file content as blobs in the branch system's copy-on-write area (`_cow/`). 8.0 -removed that system and stores content blobs in the content-addressed store (`_cas/`), but the -one-time layout migration only moved entities — it never adopted the `_cow/` content blobs. On a -store that used the VFS (for example, a CMS with published pages), that left every VFS-backed read -throwing `Blob metadata not found` and the pages 500ing. **8.0.12 adds an on-open recovery pass** -that adopts every orphaned `_cow/` blob into `_cas/` — copying both the bytes and the metadata, -idempotently and **non-destructively** (the `_cow/` originals are never deleted). It heals both a -fresh 7→8 upgrade and a store **already** upgraded by an earlier 8.0.x that stranded them, with no -operator action: just open the store under 8.0.12. An explicit force path is exposed as -`brain.vfs.adoptOrphanedBlobs()` (returns `{ cowBlobs, adopted, alreadyPresent, incomplete }`). The -automatic pre-upgrade backup is now **retained** if any blob can't be fully adopted -(`incomplete > 0`) instead of being removed on entity-migration "success" alone. Affects any 7.x -store that used the VFS; native-8.0 and non-VFS stores no-op on a cheap existence check. Full guide: -`docs/guides/upgrading-7-to-8.md`. - -**2 — A cold open no longer re-derives durable indexes it can just load.** -Opening a persisted store rebuilt the vector, graph, and metadata indexes from the canonical -records even when the durable index state was present and loadable — an O(N) cost paid on every -boot (measured at ~48 s on an ~11k-entity store). The rebuild gate now consults an honest -per-provider durability signal (`init()` / `isReady()`) instead of an in-memory size heuristic, so -a provider that has loaded (or can cheaply demand-load) its persisted index is not rebuilt. The -built-in JS indexes keep today's behavior (a rebuild *is* their load path); the live query-time -guards that self-heal a genuinely lost index are unchanged. **The zero-rebuild boot activates when -the accelerator exposes the durability signal (`@soulcraft/cor@3.0.5`);** on earlier accelerator -versions 8.0.12 falls back to the previous behavior safely — no regression, just no speedup yet. - -**3 — Unknown query operators now throw instead of silently returning nothing.** -`find({ where })` had two gaps: the in-memory matcher (used for egress re-validation and historical -reads) was missing several documented operators (`in`, `greaterThanOrEqual`, `lessThanOrEqual`), so -it silently disagreed with the index path; and an unknown operator key (a typo, or `notIn` written -where `not: { in }` was meant) was treated as a nested-object field and silently matched nothing. -8.0.12 aligns the matcher to the full documented operator set and **validates the `where` filter -up front**, throwing a typed `BrainyError('INVALID_QUERY')` naming the bad operator. Dotted paths -remain the supported form for nested fields. If you relied on an unknown key silently returning `[]`, -it now throws — the fix is to use the documented operator or dot-notation. - -## v8.0.11 — 2026-07-02 (no script shape can hang on brainy's internals) - -Completes v8.0.10's exit fix for every operation class. Two further mechanisms found and fixed: -the `beforeExit` auto-flush hook looped forever on any script that never reaches `close()` (Node -re-emits `beforeExit` after each event-loop drain and the async flush schedules new work — it now -self-deregisters before its single flush, which still lands your buffered data before exit), and -every background-maintenance interval (graph auto-flush, LSM compaction, metadata write-buffer, -VFS/path-cache maintenance, statistics debounce) is now unref'd at creation. Verified against the -published package: an `add + relate` script exits cleanly both with `close()` (~0.5 s) and with no -teardown at all — with the data confirmed durable on reopen. A per-operation-class sweep test now -asserts no ref'd timer survives `close()`, so this bug class stays closed. - -## v8.0.10 — 2026-07-02 (a bare script exits cleanly after `close()`) - -A minimal `init → add → close` script used to hang forever after `close()` returned — brainy held -four process keep-alives it never released: the global SIGTERM/SIGINT shutdown hooks (never -removed; now deregistered when the last live instance closes), an internal cache-fairness interval -(unclearable by construction; now unref'd), and the VFS/PathResolver maintenance intervals -(`close()` never shut the VFS down; now wired). Verified against the published package: the same -script now exits ~1 ms after `close()` resolves. No API change; servers and long-running processes -are unaffected. - -## v8.0.9 — 2026-07-02 (guarded plugin auto-detection — the "install it and it's on" contract) - -With the default config (`plugins` unset), brainy now **auto-detects the first-party accelerator**: -installing `@soulcraft/cor` is the opt-in. The detection is guarded — everything except "not -installed" fails loud: - -- Not installed → plain brainy, silently (the free path — zero noise, zero cost). -- Installed and healthy → it loads and announces itself (`[brainy] Plugin activated`). -- Installed but broken (unresolvable, invalid shape, failed activation, version mismatch) → - **`init()` throws.** An installed accelerator never silently vanishes behind the JS engines. -- `plugins: []` / `false` = explicit opt-out; `plugins: ['@soulcraft/cor']` pins the exact list - (unchanged semantics). - -Also new: if a plugin activates but registers **zero** native providers (e.g. a licensing gate -declining to engage), brainy warns loudly instead of leaving you to discover every query is running -on the JS engines. - -> This supersedes v8.0.8's "plugins are explicit opt-in" wording, which documented the pre-GA -> loader accurately but contradicted the published product contract ("add the package and it -> activates"). 8.0.9 makes the contract true — with the loud-failure guarantees intact. - -## v8.0.8 — 2026-07-02 (docs-only patch on the GA) - -Corrects the README's scale-up section: the native provider is **not** auto-detected — plugins are -**explicit opt-in** (`new Brainy({ plugins: ['@soulcraft/cor'] })`; brainy never auto-imports a -package you didn't list, and a listed plugin that fails to load throws rather than silently falling -back to the JS engines). Also fixes the stale `plugins` config comment in the public types and one -phrase in the plugin-author guide. No code change. - -## v8.0.7 — 2026-07-02 (GA, npm tag `latest`) - -> **Why 8.0.7:** npm permanently retires unpublished version numbers, and `8.0.0`–`8.0.6` were -> consumed by a January development cycle (published and immediately unpublished). `8.0.7` is -> the first stable release of the 8.x line; there are no earlier stable 8.0.x releases. - -**8.0.7 is the first stable major on the u64-id core.** It ships in lockstep with the optional -native provider's `3.0` (the billion-scale path). Everything below works standalone on the open-core -JS engine — a native provider is feature-detected and only changes the scale ceiling, never the API. - -**Upgrading from 7.x: nothing to script.** The first time 8.0 opens a 7.x brain it upgrades itself — -an automatic, observable, **coordinated migration lock** rebuilds every derived index from your -canonical records while Brainy **blocks and queues** reads and writes, so no operation ever touches a -half-built index and no write is ever lost. Budget seconds-to-minutes per brain at large sizes; small -brains rebuild inline on open. A pre-upgrade hard-link **backup** is taken automatically (removed on -success, kept on failure for rollback). Observe it via `getIndexStatus().migration`; a caller that -hits the window gets a typed, retryable **`MigrationInProgressError`** (catch → HTTP 503 + -`Retry-After`). Bounded by `migrationWaitTimeoutMs` (default 30 s — it bounds *your wait*, not the -rebuild). Opt out of the backup with `migrationBackup: false`. - -### Headline changes - -- **The cold-open silent-`[]` class is gone.** On a cold reopen, filtered finds (`find({ where })`) - and graph reads (`find({ connected })` / `neighbors()` / `related()`) never silently return `[]` - for data that is actually present. With a native provider the durable indexes cold-serve every - filter and edge with zero rebuild; the open-core engine adds belt-and-suspenders guards that - self-heal from canonical data or throw a loud, typed error — never a silent empty result. Two new - exported errors: **`MetadataIndexNotReadyError`**, **`GraphIndexNotReadyError`**. - -- **Faster vector search (open-core).** The JS distance functions are allocation-free loops instead - of `reduce`: **~6× cosine, ~1.4× euclidean** (MEASURED — `tests/benchmarks/distance-microbench.mjs`, - 384-dim, median of 41). Numerically identical, so recall is unchanged. Exact metadata-filter - pushdown and scale-aware search width keep `find()` recall-correct as data grows. - -- **Per-write immutable history.** Every write is generation-stamped; `now()` / `asOf()` / - `transact()` read a consistent point in time, and a retention knob bounds history growth. Single - writes participate in the same immutable timeline as transactions. - -- **Billion-scale resident memory.** Nothing is O(N)-resident on the write or time-travel path — - per-id caches removed, the committed-generation ledger is an interval set, per-id history chains - are bounded (hot-window + LRU). Resident memory is independent of entity count. - -- **Deterministic, round-trip-free writes.** Supply your own ids, forward-reference not-yet-written - entities inside a `transact()`, `ifAbsent` upsert, and `brain.newId()` (uuidv7) — write graphs - without a write→wait→get round-trip. - -### Breaking changes (summary — the full migration guide follows below) - -- **Runtime floor: Node ≥ 22 / Bun ≥ 1.1** (Bun recommended). Compiler target ES2023; the DOM lib is - dropped — 8.0 is a Node/Bun/Deno engine with no browser path. -- **`neural()` removed** and **`Db.search` removed** — use `find()` on the brain. -- **Storage config is one `path` key.** Removed aliases now throw with a message pointing at `path`. -- **`get()` omits the vector by default** — pass `{ includeVectors: true }` when you need it. -- **Export/import type is `PortableGraph`** (was `BackupData`; wire tag `brainy-backup` → - `brainy-portable-graph`). Re-export any snapshots you version outside Brainy. -- **Reserved `visibility` field** (`public` / `internal` / `system`) on nouns and verbs; `internal` - and `system` are excluded from normal reads by default. -- **4 deprecated query operators removed** (`is`/`isNot`/`greaterEqual`/`lessEqual`) and reserved - keys in a `metadata` bag now **throw** by default — details in the rc notes below. - -> Post-rc.9 hardening folded into GA (no on-disk or provider-contract change vs `8.0.0-rc.9`): -> the metadata cold-read guard, the auto pre-upgrade backup (with an `_id_mapper/*` mmap byte-copy -> correctness fix), and a CI fix so a fresh clone builds green. - -### RC history (rc.1 → rc.9, npm tag `rc`) - -> **rc.9 changes (2026-07-01):** -> - **Whole-brain auto-upgrade is now a coordinated, observable LOCK — supersedes rc.8's no-freeze -> approach.** When a large brain's derived-index format changes (7.x→8.0), the native provider -> rebuilds the indexes from the canonical records **in place** while Brainy **blocks and queues** -> reads and writes — so no operation ever touches a half-built index. This is a clean *blocking -> upgrade to a known-good state* (vs rc.8's online background swap): unknown/halfway states are -> more dangerous than a bounded wait. Small brains still rebuild inline on open. New surface, all -> additive: a typed, exported, retryable **`MigrationInProgressError`** (catch it → HTTP 503 + -> `Retry-After`); `getIndexStatus()` gains **`migrating` + `migration`** progress (never gated — -> the readiness-probe signal); a **`migrationWaitTimeoutMs`** config (default 30 s — it bounds the -> *caller's wait*, NOT the rebuild, which is unbounded); `health()` / `checkHealth()` report the -> upgrade without blocking. No effect without a native provider. -> - **Faster vector search (open-core).** The JS distance functions are rewritten from `reduce` to -> allocation-free loops — **~6× cosine, ~1.4× euclidean** (MEASURED, `tests/benchmarks/distance-microbench.mjs`, -> 384-dim, median of 41). Numerically identical → recall unchanged. -> - **Runtime floor + Bun.** Engines are now **Node ≥22 / Bun ≥1.1** (fixes a Node-24 `EBADENGINE`); -> compiler target ES2023 with DOM dropped from the type lib (8.0 is Node/Bun/Deno-only, no browser -> path). **Bun is recommended as a runtime** (`bun add` / `bun run`); the single-binary -> `bun build --compile` is not a supported target (native addon can't embed + a Bun 1.3.10 codegen -> regression). `.d.ts` stays TS-5.x-parseable. - -> **rc.8 additions (2026-06-30) — additive, no breaking change:** -> - **No-freeze (online) whole-brain auto-upgrade.** When a derived-index format changes, a large -> brain upgrades **without blocking** — the native provider rebuilds the new indexes in the -> background and serves correct reads from the canonical records meanwhile, then atomically swaps; -> no minutes-long freeze on the first open/query. (Small brains still auto-rebuild inline on open.) -> New surface: an optional provider `isMigrating()` deference signal, a public `brain.stampBrainFormat()` -> the provider calls once its swap verifies, and a `@soulcraft/brainy/brain-format` export so the -> provider shares the `indexEpoch` constant. No effect without a native provider. - -> **rc.7 additions (2026-06-30) — additive, no breaking change:** -> - **8.0 cold-graph self-heal.** `find({ connected })` / `neighbors()` / `related()` never -> silently return `[]` on a cold open where the graph adjacency loaded its membership but not -> its source→target edges — it self-heals (rebuild from storage) or throws a loud -> `GraphIndexNotReadyError`, never empty-for-persisted-data. (The 8.0 equivalent of the 7.33.4 -> fix 7.x consumers already have; gates on the native provider's honest `isReady()` edge-readiness -> signal.) -> - **Billion-scale RAM — nothing O(N)-resident on the write/time-travel path.** Eliminated the -> per-id storage caches (per-type counts now sourced from the record), made the -> committed-generation ledger an interval set, and bounded the per-id time-travel history chains -> (hot-window + LRU, with lock-light reconstruction so historical reads never stall writers). -> Resident memory is now independent of entity count. -> - **Whole-brain version handshake.** A `_system/brain-format.json` marker + `brain.formatInfo()` -> + a shared, lockstep `indexEpoch`: a future derived-index format change auto-rebuilds the -> indexes from the canonical records on open (non-destructively) — the foundation for -> whole-brain auto-upgrade with the native provider. No effect on an unchanged brain. - -> **rc.6 additions (2026-06-29) — additive, no breaking change:** -> - **Open-core perf**: HNSW delete is now O(in-degree) (was O(N²) for bulk delete) via a -> reverse-adjacency index; the negation/absence operators (`ne`/`exists:false`/`missing:true`) -> are served as a roaring-bitmap difference instead of materializing the whole corpus. -> - **Native provider contract** (cor lockstep, optional/feature-detected — no effect without a -> native provider): a cold-open `probeConsistency()` self-heal hook, and a `getIdsForFilter` -> page bound so the native index can early-stop the unsorted `find({ type, where, limit })` path. -> - **Test hygiene**: re-homed previously-unrun test suites into CI + a guard so a test file can -> never silently fall outside every config again. No source/API change from these. - - -**Affected products:** every consumer — this is a major release with removed -surfaces, hard renames (no aliases, no deprecation period), and one flipped -default. This entry **is** the migration guide: the find/replace pairs, the -sed snippet, and the upgrade checklist below are the complete story — there is -no separate migration doc. **`8.0.7` is GA on `latest`** — install with -`npm i @soulcraft/brainy@latest`. - -> **rc.3–rc.5 additions (2026-06-24):** -> - **Showcase-quality GA hardening (rc.5).** A full readiness audit closed a cold-init -> version-coupling bug (a matched native provider could be rejected on first open), turned -> silent storage-read failures into named `BrainyError`s (no more empty-result-on-error), -> stopped the JS graph LSM from orphaning compacted SSTables, corrected the public docs to -> the real API, and documented the two headline methods. Plus a dead/deprecated-code sweep. -> - **BREAKING — 4 deprecated query operators removed.** `is`→`eq`, `isNot`→`ne`, -> `greaterEqual`→`gte`, `lessEqual`→`lte`. The canonical operators and their clean long-form -> aliases (`equals`/`notEquals`/`greaterThan`/`greaterThanOrEqual`/`lessThan`/`lessThanOrEqual`) -> are unchanged — only the four redundant spellings are gone. Find/replace if you used them. -> - **`find()` search mode** is now the single `searchMode: SearchMode` option (the redundant, -> silently-ignored `mode` alias and the unwired `explain` flag were removed from `FindParams`). -> - The phantom index-integrity guard (find() re-validates every result against its predicate) -> shipped in rc.3; rc.4 added the `accel.isInitialized` gate + #35 at-gen candidate vectors. - -> **rc.1 additions (this entry predates them — full writeup lands at GA):** entity-id -> normalization — `brain.newId()` (UUID v7) + v7 default ids, and non-UUID string ids are -> transparently normalized to a stable UUID v5 (original key preserved under `_originalId`); -> `brain.neural()` clustering and `Db.search()` removed (use `find({ vector })` / -> `find()` + aggregation `GROUP BY`); storage config collapsed to one `path` key (the -> pre-8.0 aliases now throw); and `queryAggregate` no longer hangs/staled min-max after a delete. -> **Reserved fields in a `metadata` bag now throw by default** — passing a Brainy-reserved key -> (`confidence`, `weight`, `subtype`, `visibility`, `service`, `createdBy`, `noun`/`verb`, `data`, -> `createdAt`, `updatedAt`, `_rev`) inside `metadata` on `add()`/`update()`/`relate()`/`updateRelation()` -> (untyped/JS callers — TypeScript already blocks it) is rejected with an Error naming the correct -> param, instead of the old silent remap-or-drop. Pass reserved values as their dedicated top-level -> params. Opt back into the legacy behavior with `new Brainy({ reservedFieldPolicy: 'warn' | 'remap' })`. - -> **rc.2 additions (2026-06-21):** -> - **Graph performance + the `brain.graph` namespace.** New `brain.graph.subgraph(seeds, opts)` -> (bounded multi-hop neighborhood → `{ nodes, edges, truncated }`) and `brain.graph.export(opts)` -> (stream the whole graph in one O(N+E) pass — the right primitive for visualizing all data -> instead of paging per node). `related({ node })` returns every edge incident to an entity in -> **both directions** in one O(degree) call. Under the hood: `related({ from/to })` now stays -> O(degree) under the default visibility filter (was a full scan), and the verb **and** noun -> pagination walks are cursor-based, so a full edge/node walk is **O(N), not O(N²)** (a consumer -> measured a 19k-edge whole-graph read drop from ~27s toward a single scan). These transparently -> use a native graph engine when present (the `@soulcraft/cor` 3.0 acceleration layer) and fall -> back to pure-TS adjacency otherwise. -> - **Additive ergonomics:** `add({ upsert: true })` (create-or-update in one call — merges into an -> existing id instead of overwriting), `find({ includeVectors: true })`, and `removeMany`'s batch -> size is now storage-adaptive (was a hardcoded 10). -> - **Correctness fixes (apply to all consumers, not just graph users):** `related()` results now -> carry `visibility`; whole-graph/`getNouns` streaming no longer leaks `system`/`internal` entities; -> and small-page cursor walks over nouns no longer loop. No API change — just correct behavior. - -> **rc.3 additions (2026-06-23):** -> - **Graph analytics on the `brain.graph` namespace.** Three intent-level reads that answer -> whole-graph questions in one call: -> - **`brain.graph.rank(opts?)`** → `{ id, score }[]` descending — "which entities matter most" -> (importance / centrality). `topK` to cap. -> - **`brain.graph.communities(opts?)`** → `{ groups: string[][], count }` — "which things group -> together". Weakly-connected components by default; `{ directed: true }` returns -> strongly-connected components. -> - **`brain.graph.path(from, to, opts?)`** → `{ nodes, relationships, cost } | null` — the best -> route between two entities. Fewest hops by default; `{ by: 'weight' }` minimizes summed edge -> weight (the 0–1 connection strength); `direction`, `type`, and `maxDepth` filters apply. -> These are **intent contracts, not algorithm contracts** — the question is the promise, the -> algorithm is the engine's choice. They transparently use the native `@soulcraft/cor` 3.0 graph -> engine when present, and fall back to pure-TS kernels (PageRank, connected components / Tarjan -> SCC, BFS / Dijkstra) otherwise. Both paths return identical shapes and respect the default -> visibility filter (opt in with `includeInternal` / `includeSystem`). -> - **Filtered vector search keeps its recall (`allowedIds` pushdown).** A `find({ query, where })` -> that combines semantic search with a metadata filter now restricts the vector walk to the -> matching candidates *inside* the search (walk-all, collect-allowed) instead of filtering the -> top-k afterward — so a query whose nearest vectors are all filtered out still returns the best -> matches that DO pass the filter, rather than coming back empty. No API change. With the native -> `@soulcraft/cor` 3.0 stack the matched universe is forwarded as an opaque roaring buffer with -> zero id materialization in TypeScript; the pure-JS path restricts the beam walk with a string set. -> - **Historical (`asOf`) semantic search can skip the rebuild.** A filtered semantic query at a -> past generation — `db.asOf(g).find({ query, where })` — no longer always rebuilds an ephemeral -> in-memory vector index over every at-`g` vector (O(n@G)). When a native versioned vector engine -> is registered and can serve the pinned generation, Brainy resolves the at-`g` filter universe from -> its record layer (no rebuild) and routes the vector leg to the engine with the matched ids + the -> generation; without one it falls back to the materialization (unchanged). The `VectorIndexProvider.search` -> options gained an optional `generation?: bigint` ("omitted = now") to carry this — additive, the -> built-in index ignores it. No public API change. -> - **`find({ where, orderBy })` bounds the sort to the page.** A broad filter + `orderBy` -> returning one page no longer materializes every matching sorted id (it produced the full sorted -> match set, then sliced) — the page bound (`offset + limit`) is threaded into the column store's -> top-K sort, so returning 20 rows from a million matches stays a bounded-K heap. No API change; -> ordering and pagination are unchanged. -> - **`brain.graph.subgraph()` accepts a query (query→expand).** The seed selector now takes not -> just entity id(s) but a `find()` result or a `FindParams` query — `brain.graph.subgraph({ where: -> { team: 'platform' } }, { depth: 1 })` runs the query and expands the neighborhood of every -> match in one call. With the native engine a metadata-only query's matched universe is handed to -> the traversal as an opaque set with no id materialization in TypeScript (the query→expand -> fusion); the pure-JS path materializes the matched ids and expands from them. Existing -> id-seeded calls are unchanged. - -### Headline: Database as a Value - -8.0 replaces Brainy's two overlapping version-control subsystems (copy-on-write -branching and per-entity versioning, ~5,100 LOC combined) with **one -mechanism**: generational MVCC over immutable, generation-stamped records, -exposed through a Datomic-style immutable database value — the **`Db`**. - -```ts -const db = brain.now() // pin the current state — O(1), no I/O - -await brain.transact([ - { op: 'update', id: invoiceId, metadata: { status: 'paid' } } -], { meta: { author: 'billing-service', reason: 'PO-7741' } }) - -await db.get(invoiceId) // still 'pending' — pinned, forever -await brain.get(invoiceId) // 'paid' — live -await db.release() // unpin when done -``` - -What you get: - -- **`brain.now()`** — pins the current generation as an immutable `Db` view. - True snapshot isolation: the view reads exactly its pinned state no matter - what commits afterwards, including deletes. Readers never block writers and - writers never block readers. -- **`brain.transact(ops, { meta, ifAtGeneration })`** — atomic multi-write - batches (`add` / `update` / `remove` / `relate` / `unrelate`) committed as - exactly one generation. Either every operation applies or none do. - `ifAtGeneration` is whole-store compare-and-swap (`GenerationConflictError` - on conflict) — the big sibling of the per-entity `ifRev` CAS from 7.31. - `meta` is reified Datomic-style into an append-only transaction log, - readable via `brain.transactionLog()`. -- **`brain.asOf(generation | Date | snapshotPath, { exclusive? })`** — time - travel with the **full query surface**: `get()`, `find()` in every mode, - semantic search, graph traversal, cursors, aggregation, all at the pinned past - state. `exclusive: true` pins the generation immediately before the target - (strict-before). -- **`db.with(ops)`** — speculative writes applied in memory on top of a view. - Nothing touches disk, the generation counter, or the indexes. What-if - analysis, then `transact()` the same ops for real. -- **`db.persist(path)`** — instant self-contained snapshots. On filesystem - storage they are built from hard links (no entity data is copied; later - writes to the source can never alter the snapshot because rewrites swap - inodes). `brain.restore(path, { confirm: true })` replaces the whole store - from one; `Brainy.load(path)` opens one read-only with the full query - surface. -- **Range verbs over history** — answer "what happened BETWEEN two points": - - **`db.since(Db | generation | Date)`** — the entity and relationship ids - committed transactions touched after an **exclusive** lower bound (now - accepts a generation or `Date`, not just a prior `Db`). - - **`brain.diff(a, b)`** — the touched ids CLASSIFIED as `{ added, removed, - modified }` (split by nouns/verbs) by resolving each at both endpoints; a - touched-but-reverted id lands in none of the buckets. Endpoints are a - generation, `Date`, or `Db`, in either order. - - **`brain.history(id, { from?, to? })`** — every distinct version of ONE - entity/relationship over a range, oldest first (`value: null` marks a - removal); each version equals `asOf(version.generation).get(id)`. - - **`brain.transactionLog({ from?, to?, limit? })`** — the commit log over an - **inclusive** generation/`Date` window (contrast `since`'s exclusive lower - bound); `limit` applies last. -- **Every write is versioned (Model-B).** History granularity is **per-write**: - EVERY write — `transact()` AND a single-operation `add`/`update`/`remove`/ - `relate` — is its own immutable generation. A `now()` pin always freezes - against later writes, and `asOf`/`since`/`diff`/`history`/`transactionLog` - reflect single-ops exactly like transacts. `transact()` groups several - operations into ONE atomic generation. Single-op history durability is **async - group-commit** (the live write is acknowledged immediately; its before-image - is batched to disk in one fsync) — a hard crash can lose only the last - un-flushed window's *history*, never live data, and a crash mid-flush is - recovered by drop-without-restore. A freshly-initialized brain is - `generation() === 0` with an empty `transactionLog()` (init-time - infrastructure is the un-versioned baseline); the first user write is gen 1. -- **`new Brainy({ retention })`** — the retention knob governs auto-compaction on - `flush()`/`close()`: **unset → ADAPTIVE** (disk/RAM-pressure byte budget, - zero-config; a coordinator can drive it via `brain.setRetentionBudget(bytes)`) - · **`'all'`** → unbounded · **`{ maxGenerations?, maxAge?, maxBytes? }`** → - explicit caps (reclaim oldest-unpinned while ANY cap is exceeded). Live pins - are ALWAYS exempt. -- **`brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? })`** — manual - reclaim on the same caps. Compaction never breaks a pinned read. `diff`/`since` - throw `GenerationCompactedError` below the horizon; `history` truncates to it. - -The precise guarantees are documented in: - -- [docs/concepts/consistency-model.md](docs/concepts/consistency-model.md) — - the guarantees, each proven by a dedicated test in - `tests/integration/db-mvcc.test.ts` -- [docs/guides/snapshots-and-time-travel.md](docs/guides/snapshots-and-time-travel.md) - — the recipes: backup, restore, time-travel debugging, what-if, audit trails -- [docs/ADR-001-generational-mvcc.md](docs/ADR-001-generational-mvcc.md) — the - design record: persisted layout, commit protocol, crash recovery, proof table - -### Portable export & import (PortableGraph v1) - -A portable, versioned graph format that serializes part or all of a brain to a -single JSON document and restores it — the cross-environment, cross-version -(7.x↔8.0), partial-or-whole companion to the native `persist()` snapshot. - -```ts -// Export is a method on the immutable Db, so it composes with now()/asOf()/with() -const graph = await brain.export({ collection: id }, { includeVectors: true }) -await otherBrain.import(graph, { onConflict: 'merge' }) // dedup-by-id - -;(await brain.asOf(gen)).export(sel) // time-travel export -brain.now().with(ops).export(sel) // what-if export -``` - -- **`brain.export(selector?, options?)` / `db.export(...)`** → a versioned - `PortableGraph` document. Selectors reuse `find()`'s grammar: `{ ids }`, - `{ collection }` (alias `memberOf`, transitive `Contains`), - `{ connected: { from, depth } }`, `{ vfsPath }`, predicate - (`{ type, subtype, where, service }`), or the whole brain (omit) — and they - compose. Options: `includeVectors`, `includeContent` (VFS file bytes), - `includeSystem`, `edges: 'induced' | 'incident' | 'none'`. -- **`brain.import(graph, options?)`** is polymorphic: a `PortableGraph` document is - restored as **one atomic transaction** (`onConflict: 'merge' | 'replace' | 'skip'`, - `reembed: 'auto' | 'never'`, `remapIds` for cloning); a file/buffer routes to the - existing CSV/PDF/Excel/JSON ingestion. No migration for ingestion callers. -- **`validatePortableGraph(data)`** — dry-run structural/version/endpoint check before import. -- **Format** (`format:'brainy-portable-graph'`, `formatVersion: 1`) is identical on - 7.x and 8.0; standard fields (`subtype`/`visibility`/`data`/…) are top-level, - `metadata` is custom-only. Current-state (no generation history — that lives in - `persist()`). Types exported from the package root. -- **Naming:** the type is `PortableGraph` (with `PortableGraphEntity` / - `PortableGraphRelation`) — it is the portable interchange form of a graph, not a - backup (that role is `persist()`/`load()`). Renamed from the short-lived - `BackupData`/`'brainy-backup'` (introduced in 7.32.0, never adopted) with no - compatibility shim. -- Guide: [docs/guides/export-and-import.md](docs/guides/export-and-import.md). - -### Aggregation: distinctCount over any value type - -`distinctCount` now counts distinct values of **any** type (strings, booleans, -numbers) — previously it silently returned `0` for non-numeric fields, missing its -primary use (distinct categories / users / tags). `percentile` (with `p`; median = -`p: 0.5`), `stddev`, and `variance` are exact and delete-safe alongside the -write-time `sum`/`count`/`avg`/`min`/`max`. - -### Removed surfaces and their replacements - -| Removed in 8.0 | Replacement | -|---|---| -| `brain.fork(name)` | Speculation: `db.with(ops)` (in-memory). Long-lived writable copy: `brain.restore()` a persisted snapshot into a fresh data directory. | -| `brain.checkout(branch)` | Open the snapshot you want — `Brainy.load(path)` read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store. | -| `brain.listBranches()` / `brain.getCurrentBranch()` / `brain.deleteBranch()` | A "branch" is now a name → snapshot-path mapping owned by your application. | -| `brain.commit({ message })` | `brain.transact(ops, { meta })` — every batch is an atomic, logged, time-travelable commit; audit fields live in the database, not in commit messages. | -| `brain.getHistory()` / `brain.streamHistory()` | `brain.transactionLog({ limit })` + `db.since(olderDb)`. | -| `brain.versions.*` (`save` / `list` / `compare` / `restore` / `prune`) | A pinned `Db` or persisted snapshot captures *every* entity at that moment; `asOf()` reads any entity's past state; `restore()` for whole-store rollback. | -| `brain.data()` (DataAPI: `clear` / `import` / `export` / `getStats`) | `brain.clear()`, `brain.import()`, `db.persist(path)` (the full-fidelity export format), `brain.restore(path, { confirm: true })`, `brain.stats()`. | -| Cloud + browser storage adapters (`s3` / `gcs` / `r2` / `opfs` storage types, Azure Blob, plus the `storage.branch` option) | `filesystem` and `memory` only. Cloud backup is now an operator concern: `db.persist()` produces a plain directory — sync it with `gsutil` / `aws s3 cp` / `rclone` / `azcopy`. Passing a removed storage type throws at construction with this exact guidance. | -| Browser support | Node.js 22 LTS or Bun ≥ 1.0 (`engines` enforces `node: 22.x`); Deno works through its Node compatibility layer. | -| `brain.migrateToDiskAnn()` / `brain.migrateToHnsw()` | Gone — there is one canonical `'vector'` provider slot. A registered native vector provider selects its own operating mode; there is nothing to call. | -| Distributed-clustering subsystem — `config.distributed`, the `DistributedRole` enum, the 13 `BRAINY_*` cluster env vars (`BRAINY_DISTRIBUTED`, `BRAINY_ROLE`, `BRAINY_HTTP_PORT`, `BRAINY_WS_PORT`, `BRAINY_DNS`, `BRAINY_SERVICE`, `BRAINY_NAMESPACE`, `BRAINY_CONSENSUS`, `BRAINY_COORDINATOR`, `BRAINY_NODES`, `BRAINY_REPLICAS`, `BRAINY_SHARDS`, `BRAINY_TRANSPORT`), the storage `setDistributedComponents` hook, and the `@soulcraft/brainy/config` preset/augmentation registry | Removed. Brainy 8.0 is a single-process library — there is no coordinator, peer discovery, or consensus to operate. Scale via: the optional native provider (`@soulcraft/cor` — on-disk DiskANN to 10B+ vectors on one machine); per-tenant pools (one instance + storage dir per tenant); and horizontal read scaling (many reader processes against one shared store, single writer). The `mode: 'reader' \| 'writer'` multi-process roles are unchanged. | -| CLI `fork` / `branch` / `checkout` / `migrate` | CLI `snapshot ` / `restore ` / `history` / `generation`. | -| `BrainyZeroConfig` type | Removed. 8.0 zero-config is automatic and internal — `new Brainy()` auto-detects storage, derives HNSW quality from `config.vector.recall`, picks `persistMode` from the adapter, and sizes caches to the detected container-memory limit. There is no config-generation type to import. | -| `brain.isFullyInitialized()` / `brain.awaitBackgroundInit()` | `await brain.ready`. The built-in filesystem/memory adapters finish initialization synchronously inside `init()`, so a single readiness promise is the whole story — these methods were no-ops once cloud adapters were removed. | - -**Opening a 7.x store auto-migrates it.** 8.0 stores entities at the root; 7.x -stored them branch-scoped under `branches//`. On first open, 8.0 collapses -the **HEAD branch** (`config.storage.branch`, default `main`) to the 8.0 layout in -place and rebuilds all derived state — no action required (`autoMigrate` defaults -to `true`; set it to `false` to make 8.0 refuse a legacy layout with an explicit -error instead). Two caveats: - -- **Non-HEAD branches are not imported.** 8.0 has no COW branches; only the head - branch's data is migrated. If you care about other branches, **export each one - while still on 7.x** (`fork → export`, or copy its store). -- **Back up first for rollback.** The migration mutates the directory in place and - 8.0 does not keep the old layout — copy the data directory before the first 8.0 - open if you need to roll back. - -### Renames — find/replace pairs - -Hard renames, no aliases. Every pair below is verified against the 8.0 source. - -| Brainy 7.x | Brainy 8.0 | -|---|---| -| `brain.delete(id)` | `brain.remove(id)` — matches the transact op vocabulary (`add` / `update` / `remove` / `relate` / `unrelate`) | -| `brain.deleteMany(params)` | `brain.removeMany(params)` | -| `DeleteManyParams` (type) | `RemoveManyParams` | -| `brain.getRelations(paramsOrId)` | `brain.related(paramsOrId)` — the same name a pinned `Db` exposes (`db.related()`) | -| `GetRelationsParams` (type) | `RelatedParams` | -| CLI `delete ` | CLI `remove ` (JSON output `{ id, removed: true }`, matching `unrelate`) | -| `HnswProvider` (from `@soulcraft/brainy/plugin`) | `VectorIndexProvider` | -| `HNSWIndex` (exported class) | `JsHnswVectorIndex` | -| Provider keys `'hnsw'` and `'diskann'` | `'vector'` (the only key Brainy consults for the vector index) | -| `config.hnsw = { quantization, vectorStorage }` | `config.vector = { recall?, persistMode? }` (shape change — see below) | -| `config.vector.quantization` (and 7.x `config.hnsw.quantization`) | **Removed.** The JS vector path is full-precision (exact float32 distances) only — quantization at scale is the native provider's job (e.g. DiskANN PQ). | -| `hnswPersistMode: 'immediate' \| 'deferred'` (top level) | `config.vector.persistMode` (auto-selected from the storage adapter when omitted: `'immediate'` on filesystem, `'deferred'` on memory) | -| `config.storage.type: 's3' \| 'gcs' \| 'r2' \| 'opfs'` | `'filesystem'` (or `'memory'` / `'auto'`) | -| `config.storage.branch` | Removed (no COW branches) | -| `brain.stats().indexHealth.hnsw` | `brain.stats().indexHealth.vector` | -| Storage adapter contract `saveHNSWData()` / `getHNSWData()` | `saveVectorIndexData()` / `getVectorIndexData()` | -| Cache category `'hnsw'` (`CacheProvider` union) | `'vectors'` | -| `config.history = { retainGenerations, retainMs, autoCompact }` | `config.retention` — `'all'` \| `'adaptive'` \| `{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }`; **unset → adaptive** (was keep-100/7-days). The fields are now **caps**, not floors. | -| `compactHistory({ retainGenerations, retainMs })` | `compactHistory({ maxGenerations, maxAge, maxBytes })` — `retainGenerations`→`maxGenerations`, `retainMs`→`maxAge` (now upper-bound CAPS), plus new `maxBytes`. | -| `CompactHistoryOptions.retainGenerations` / `.retainMs` | `.maxGenerations` / `.maxAge` (+ `.maxBytes`) | - -Mechanical renames as one command (run from your repo root, review the diff): - -```bash -find . -name '*.ts' -not -path '*/node_modules/*' -print0 | xargs -0 sed -i \ - -e 's/\bHnswProvider\b/VectorIndexProvider/g' \ - -e 's/\bHNSWIndex\b/JsHnswVectorIndex/g' \ - -e 's/\bsaveHNSWData\b/saveVectorIndexData/g' \ - -e 's/\bgetHNSWData\b/getVectorIndexData/g' \ - -e 's/indexHealth\.hnsw\b/indexHealth.vector/g' \ - -e "s/registerProvider('hnsw'/registerProvider('vector'/g" \ - -e "s/registerProvider('diskann'/registerProvider('vector'/g" \ - -e "s/getProvider('hnsw'/getProvider('vector'/g" \ - -e "s/getProvider('diskann'/getProvider('vector'/g" \ - -e 's/\.getRelations(/.related(/g' \ - -e 's/\bGetRelationsParams\b/RelatedParams/g' \ - -e 's/\.deleteMany(/.removeMany(/g' \ - -e 's/\bDeleteManyParams\b/RemoveManyParams/g' -``` - -`delete()` → `remove()` is deliberately **not** in the snippet: `.delete(` is -too common (`Map`/`Set`/storage adapters) for a blind sed. Find your -`brain.delete(...)` call sites and rename them by hand. - -The config shape changes need a human (they are not 1:1 textual): - -```ts -// 7.x -new Brainy({ - hnswPersistMode: 'deferred', - hnsw: { quantization: { enabled: true, bits: 8, rerankMultiplier: 3 } } -}) - -// 8.0 — config.vector is { recall?, persistMode? } -new Brainy({ - vector: { - recall: 'balanced', // 'fast' | 'balanced' | 'accurate' - persistMode: 'deferred' // optional; auto-selected from storage when omitted - } -}) -``` - -`config.vector.recall` replaces the algorithm-internal HNSW knobs: `'balanced'` -(the default) maps to exactly the 7.x default parameters, so an upgrade without -explicit knobs changes nothing about search behaviour. `config.vector.quantization` -and `config.hnsw.vectorStorage` are removed: the JS vector path now computes exact -float32 distances throughout (no rerank/approximate branch), which is what made -its quantization a memory *increase* — it stored both the full and the quantized -vectors in RAM. Quantization at scale belongs to the native provider's own PQ. -The on-disk vector index file names are **unchanged** (e.g. `hnsw-system.json`) — -existing filesystem stores need no data migration for this rename; only the API -surface moved. - -### Behavior changes - -- **`subtype` is required by default.** 7.30's opt-in strict mode is now the - default: every `add()` / `addMany()` / `update()` / `relate()` / - `relateMany()` / `updateRelation()` rejects writes whose type carries no - non-empty `subtype` (`src/brainy.ts:9759` — `requireSubtype ?? true`). - Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy - data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type - allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)` - still compose. `brain.audit()` reports entries missing a subtype and the new - `brain.fillSubtypes(rules)` migration helper backfills them — one rule per - NounType/VerbType (literal default or per-entry function), applied only to - entries still missing a subtype, returning - `{ scanned, filled, skipped, errors, byType }`. Idempotent: re-running fills - nothing, so a crashed run is resumed by running it again. -- **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws - a teaching error if a caller passes an `id` field - (`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently - ignored (a generated UUID was used anyway); 8.0 says so out loud, because - graph-index providers key verb-int interning on the raw UUID bytes. -- **`update({ vector })` is honored on its own.** 7.x only applied a supplied - pre-computed vector when `data` was also passed (it was silently dropped - otherwise). 8.0 applies an explicit `params.vector` with dimension - validation (`src/brainy.ts:1702-1713`; proven by - `tests/unit/brainy/update.test.ts:256`). -- **`brain.clear()` re-resolves all indexes exactly as `init()` does** — - including plugin-provided vector/metadata/entityIdMapper factories and VFS - root re-creation. In 7.x, clearing a plugin-accelerated brain could leave - the metadata index rebuilt without its native id-mapper wiring. -- **`find({ connected: { …, subtype } })` with `depth > 1` now works.** 7.30 - threw `NOT_SUPPORTED` for multi-hop subtype-filtered traversal; 8.0 - implements the BFS in the JS graph index and routes to a native provider - when one is registered. -- **Every write advances the generation clock; only `transact()` writes - history.** Single-operation writes (`add` / `update` / `remove` / `relate` - outside `transact()`) bump `brain.generation()` so watermarks and CAS stay - sound, but they do not stage historical records — they remain visible - through earlier pins and are not reported by `db.since()`. Writes you want - to travel back through go through `transact()`. This is the documented - contract, stated rather than papered over. -- **Reserved fields have one canonical location — enforced at every layer.** - The Brainy-owned field names (`RESERVED_ENTITY_FIELDS`: - `noun`/`subtype`/`createdAt`/`updatedAt`/`confidence`/`weight`/`service`/ - `data`/`createdBy`/`_rev`; verb mirror `RESERVED_RELATION_FIELDS` with - `verb` for the type key) are now (1) a **compile error** inside any - `metadata` param — `add`/`update`/`relate`/`updateRelation` and the - matching `transact()` ops; (2) **normalized at write time** for untyped - callers — user-settable fields remap to their dedicated top-level param - (top-level wins; `update({metadata:{confidence}})` no longer silently - no-ops, closing a 7.x trap), system-managed fields drop with a one-shot - warning naming the right path; (3) **split at read time** through one - canonical helper, so `entity.metadata` / `relation.metadata` contain ONLY - custom fields on every read path — `get`, `find`, `related` (several - paginated/by-source/by-target paths previously echoed the full stored - record, including the `verb` type key, inside `metadata`), batch reads, - and historical `asOf()` reads. `related()` results now also surface - `confidence`/`updatedAt` top-level, and `updateRelation()` no longer - erases a relationship's `service`/`createdBy`. See "Reserved fields" in - `docs/concepts/consistency-model.md`. -- **Named errors for not-found contract failures.** `update()`, `relate()`, - `updateRelation()`, `similar({ to: id })`, `transact()` planning, and - speculative `db.with()` planning now throw `EntityNotFoundError` / - `RelationNotFoundError` (both exported from the package root, both carrying - the missing `id` as a field) instead of a generic `Error`. Message texts are - unchanged, so existing `/not found/` matching keeps working — `instanceof` - is now the supported way to branch. -- **Unchanged from 7.31:** per-entity `_rev`, `update({ ifRev })` - (`RevisionConflictError`), and `add({ ifAbsent })` work exactly as before, - and `ifRev` is also accepted on `transact()` update operations (a conflict - rejects the whole batch). - -### For provider and plugin authors - -The provider contracts in `@soulcraft/brainy/plugin` (`src/plugin.ts`) changed -shape: - -- **`GraphIndexProvider` speaks BigInt at the boundary.** Reads take entity - ints and return entity/verb ints as `bigint[]`; the coordinator owns all - UUID ↔ int conversion. `addVerb(verb, sourceInt, targetInt)` receives the - resolved endpoint ints (also mirrored on the new optional - `GraphVerb.sourceInt` / `GraphVerb.targetInt` fields) and returns the - interned verb int. The batch reverse resolver - `verbIntsToIds(verbInts: bigint[]): Promise<(string | null)[]>` is - **required** — the provider owns durable verb-int interning; Brainy keeps - only a bounded in-memory warm cache. Implementations may stay u32 internally - (`Number(bigint)` narrowing is lossless under the `EntityIdSpaceExceeded` - guard) but must speak the bigint contract. -- **`VersionedIndexProvider`** is a new optional 4-method capability — - `generation()`, `isGenerationVisible(g)`, `pin(g)`, `release(g)`, BigInt - generations — feature-detected on every registered index provider. Providers - are *post-commit appliers*: the storage-record commit is the source of - truth; on open, a provider behind the committed watermark replays the gap or - requests a rebuild. Explicit pins override any time-based snapshot retention - the provider has. Speculative `db.with()` overlays never reach providers. A - provider implementing this serves historical reads with **no rebuild** — - the open-core materializer is the correctness baseline, the provider is the - accelerator. -- **The vector index registers under `'vector'`** and implements - `VectorIndexProvider`. The `'hnsw'` and `'diskann'` keys are retired and - never looked up. `CacheProvider`'s category union uses `'vectors'` in place - of `'hnsw'`. - -### Scale and cost — how the mechanisms behave - -Mechanism descriptions, not benchmark numbers (none are published for 8.0 yet): - -- `brain.now()` pins in O(1) and adds **zero read overhead until history - actually moves** — while nothing has committed past the pin, every read - delegates to the live fast paths. -- A `transact()` commit pays O(ids touched) extra writes (before-images + - delta + manifest) — never O(store size). Single-operation writes pay an - in-memory counter bump with coalesced persistence. -- `db.persist()` on filesystem storage is a hard-link farm: snapshot creation - copies no entity data and the snapshot shares disk space with the source. - Cross-device targets fall back to per-file byte copies; persisting an - in-memory brain serializes a real, durable, loadable store. -- Historical **index-accelerated** queries (semantic search, traversal, - cursors, aggregation) on the open-core path pay a one-time O(n at the - pinned generation) in-memory index materialization per `Db`, cached until - `release()`. Record-path reads (`get`, metadata `find`, filter `related`) - at any reachable generation are served directly from the immutable record - layer. A native `VersionedIndexProvider` serves the same reads rebuild-free. - -The full cost model is in -[docs/concepts/consistency-model.md](docs/concepts/consistency-model.md). - -### Upgrade checklist (from 7.x) - -1. **While still on 7.x:** back up your data directory (a plain file copy is - fine). If you used COW branches, materialize each branch you need — 8.0 - does not read branch state. If you ran a cloud storage adapter, export your - data with 7.x tooling (`(await brain.data()).export()`) — 8.0 opens - `filesystem` and `memory` stores only. -2. **Runtime:** Node.js 22 LTS or Bun ≥ 1.0. -3. **Config:** change removed storage types to `'filesystem'`; delete - `storage.branch`; move `config.hnsw` / `hnswPersistMode` to - `config.vector` per the shape example above. -4. **Renames:** run the sed snippet, then fix the manual shape changes. -5. **Removed APIs:** replace `fork` / `checkout` / branch methods / `commit` / - `getHistory` / `streamHistory` / `versions` / `data()` per the table above. -6. **Subtypes:** if your data predates subtype discipline, start with - `requireSubtype: false`, run `brain.audit()`, backfill with - `brain.fillSubtypes(rules)` (one rule per type — a literal default or a - function deriving the subtype from each entry), re-run `audit()` until - `total === 0`, then remove the opt-out so the 8.0 default enforcement - protects you going forward. -7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` → - `snapshot ` / `restore ` / `history` / `generation`. -8. **Plugin authors:** apply the contract renames and the BigInt graph - contract; optionally implement `VersionedIndexProvider`. -9. **Verify, then take your first snapshot:** run your test suite, then - `const db = brain.now(); await db.persist('/backups/post-8.0-upgrade'); - await db.release()`. - -What did **not** change: the core query surface (`find` / `search` / `get` / -`add` / `update` / `relate` and friends), the aggregation engine, the VFS, -neural extraction, and the on-disk entity/relationship layout — 8.0 creates -its MVCC bookkeeping (`_system/generation.json`, `_system/manifest.json`, -`_system/tx-log.jsonl`, `_generations/`) alongside the existing files on -first use. - -### Native-provider (Cor) compatibility - -Brainy 8.0 pairs with `@soulcraft/cor` 3.0 — the renamed successor to the -`@soulcraft/cortex` 2.x native provider — shipping in lockstep. The old -`@soulcraft/cortex` 2.x cannot accelerate Brainy 8.0: 8.0 never consults the -`'hnsw'` / `'diskann'` provider keys, and the graph/column provider contracts -are BigInt at the boundary. Upgrade both packages together (`@soulcraft/brainy@8` -+ `@soulcraft/cor@3`). - ---- - -## v7.31.2 — 2026-06-09 - -**Affected products:** none in behaviour; affects anyone reading Brainy's TypeScript -type definitions or coreTypes for the `hnsw.quantization.bits` option. Drop-in from 7.31.1. - -### Fix: stale comment claimed SQ4 vector quantization required a paid native provider - -Brainy ships a pure-JS SQ4 distance implementation (`distanceSQ4Js` in -`src/utils/vectorQuantization.ts`) that's been part of the open-core path the whole -time. Two type-definition comments incorrectly stated SQ4 required a native (paid) -provider: - -```ts -// coreTypes.ts:472 + types/brainy.types.ts:1123 — before -bits?: 8 | 4 // default: 8 (SQ8). SQ4 requires cortex native. -``` - -The comment was simply wrong — open-core users can use SQ4 today, and the native SIMD -acceleration is a drop-in via `setSQ4DistanceImplementation`, not a hard requirement. -Corrected to: - -```ts -bits?: 8 | 4 // default: 8 (SQ8). SQ4 has a pure-JS implementation; - // cortex's distance:sq4 SIMD provider accelerates it. -``` - -Both locations updated. Pure documentation; no behaviour change. Caught during an -upstream open-core-boundary audit — thanks to whoever read the types carefully enough -to spot the misleading comment. - -### Cortex compatibility - -Zero changes required. The fix is documentation only; runtime behaviour was already -correct (`vectorQuantization.ts:412` ships `distanceSQ4Js`). - ---- - -## v7.31.1 — 2026-06-09 - -**Affected products:** any consumer running `@soulcraft/brainy` against `FileSystemStorage` -that triggers concurrent flush / compaction (e.g. explicit `brain.flush()` calls overlapping -with periodic background compaction, or a backup job + a flush job racing). Production-impacting -when present: `brain.flush()` throws `ENOENT` on rename, downstream jobs that rely on flush -(GCS / S3 / Azure backups, snapshot exports) fail continuously. Drop-in from 7.31.0. - -### Fixes a same-key rename race in `saveBinaryBlob` - -`FileSystemStorage.saveBinaryBlob` used a bare `${filePath}.tmp` suffix for its atomic-write -temp file. Two concurrent same-key calls computed the **same** temp path; both `writeFile`d, -the first `rename` succeeded, the second `rename` fired against a missing temp and threw -`ENOENT`. The throw propagated up through `brain.flush()` and broke any downstream job that -called it. - -``` -[job-queue] gcs-backup: failed — ENOENT: no such file or directory, - rename '/data/brainy-data/.../_column_index/owner/DELETED.bin.tmp' - -> '/data/brainy-data/.../_column_index/owner/DELETED.bin' -``` - -**Patch:** unique per-writer temp suffix (matches the pattern at six other atomic-write -sites in the same file) + ENOENT swallow on rename (defensive: if the temp is gone, the -work has already landed; `saveBinaryBlob` is idempotent for a given key) + temp cleanup -on any other rename failure (no orphan `.tmp.*` files). - -```ts -// before (7.31.0) -const tmpPath = filePath + '.tmp' -await fs.promises.writeFile(tmpPath, data) -await fs.promises.rename(tmpPath, filePath) - -// after (7.31.1) -const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}` -await fs.promises.writeFile(tmpPath, data) -try { - await fs.promises.rename(tmpPath, filePath) -} catch (err) { - if ((err as NodeJS.ErrnoException).code === 'ENOENT') return - await fs.promises.unlink(tmpPath).catch(() => {}) - throw err -} -``` - -### Scope - -The fix is one site — `src/storage/adapters/fileSystemStorage.ts:992-999`. Audit results: - -- **`FileSystemStorage`** — six sibling atomic-write sites already used unique suffixes; - only `saveBinaryBlob` was the outlier. All six were rechecked. Clean. -- **`OPFSStorage`** — writes via WritableStream (no tmp+rename). Not affected. -- **`GCSStorage` / `R2Storage` / `AzureBlobStorage` / `S3CompatibleStorage`** — use - object-store `PUT` (atomic at the API). Not affected. -- **`MemoryStorage`** — in-memory. Not affected. -- **`HistoricalStorageAdapter`** — read-only. Not affected. -- **COW / versioning / snapshot / HNSW / aggregation** — all delegate to storage adapters - via `saveBinaryBlob` / `writeObjectToPath`. They get the fix automatically by virtue of - using the patched primitive. - -### Beneficiary surfaces (broader than the reported bug) - -Column-store compaction (`_column_index//DELETED.bin`, segment files) is what the -production report named, but `saveBinaryBlob` is also used by HNSW connection persistence -(`_hnsw_conn/` — `src/hnsw/hnswIndex.ts:252`). If two flush paths ever raced on -HNSW persistence for the same node, they would have hit the same ENOENT. No production -reports of HNSW-side failures, but the fix removes the latent race. - -### Tests - -- **New `tests/integration/savebinaryblob-concurrent-rename.test.ts`** (4 tests): - - 20 concurrent `saveBinaryBlob(sameKey, ...)` calls all resolve without throwing - - No orphan `.tmp.*` siblings remain in the blob directory after concurrent writes - - Production-shape: two concurrent compactor passes over 10 column-index fields - (`owner`, `path`, `permissions`, `vfsType`, `modified`, `createdAt`, `accessed`, - `updatedAt`, `mimeType`, `size`) - - Single-writer path still produces correct bytes (no regression) -- The first three tests reproduce the production `ENOENT: rename '...DELETED.bin.tmp' -> - '...DELETED.bin'` error verbatim on the pre-7.31.1 code path. Verified by stashing the - fix and watching the tests fail with the exact production error. -- Existing suites unchanged: 1468/1468 unit. All integration suites pass. - -### Cortex compatibility - -Zero changes required. The bug and the fix are pure JS in the filesystem storage adapter; -the Cortex mmap-filesystem adapter wraps `FileSystemStorage` and inherits the patch -automatically. - -### Forward-compat - -The bare-`.tmp` pattern is gone repo-wide. 8.0's `Db.persist()` will route through the -same patched primitive; no additional work needed when the immutable Db API ships. - ---- - -## v7.31.0 — 2026-06-09 - -**Affected products:** consumers that need multi-writer coordination (job -schedulers, idempotent state machines, distributed locks, optimistic UI updates) -or idempotent bootstrap of well-known singletons. Additive; drop-in from 7.30.2. - -### Per-entity `_rev` + `update({ ifRev })` + `add({ ifAbsent })` - -CouchDB / PouchDB / ETag-style optimistic concurrency. Every entity now carries a -monotonic `_rev: number` that Brainy auto-bumps on every successful `update()`. -Pass it back as `ifRev` to make read-modify-write race-safe without an external -lock service. `ifAbsent` adds by-ID idempotent insert for singletons / config rows -/ deterministic-ID bootstraps. - -**What's new:** - -- **`entity._rev: number`** — initialized to `1` on `add()`, bumped by `1` on every - successful `update()` that reaches storage. Surfaced on `get()` (both fast metadata- - only and full-vector paths), `find()`, `search()`, and on the `Result` flatten layer - for backward compatibility. Pre-7.31.0 entities without `_rev` are read as `1`. -- **`update({ id, ..., ifRev: number })`** — optimistic-concurrency check. When - provided, throws `RevisionConflictError` if the persisted `_rev` no longer matches. - Carries `{ id, expected, actual }` for principled recovery. Omitting `ifRev` keeps - the prior unconditional-update behavior. -- **`add({ id, ifAbsent: true })`** — by-ID idempotent insert. Returns the existing - `id` without writing if the entity already exists. No throw, no overwrite. Ignored - when `id` is omitted (a fresh UUID can never collide). -- **`addMany({ items, ifAbsent: true })`** — applies the flag to every item; per-item - `ifAbsent` overrides the batch flag. - -### The lock recipe (single-process or distributed) - -```ts -import { Brainy, RevisionConflictError } from '@soulcraft/brainy' - -const LOCK_ID = '...uuid...' - -await brain.add({ - id: LOCK_ID, - type: NounType.Document, - data: { owner: null, expiresAt: 0 }, - ifAbsent: true -}) - -async function tryAcquireLock(workerId: string, ttlMs: number) { - const lock = await brain.get(LOCK_ID) - if (!lock) throw new Error('lock document missing') - const state = lock.data as { owner: string | null; expiresAt: number } - if (state.owner && state.expiresAt > Date.now()) return false - - try { - await brain.update({ - id: LOCK_ID, - data: { owner: workerId, expiresAt: Date.now() + ttlMs }, - ifRev: lock._rev - }) - return true - } catch (err) { - if (err instanceof RevisionConflictError) return false - throw err - } -} -``` - -### Why this is scoped to `_rev` + `ifAbsent` - -A public `brain.transaction(fn)` wrapper was on the table but was cut. The internal -`TransactionManager` exposes raw `Operation` classes that take a `StorageAdapter` -constructor argument; a high-level facade in 7.31.0 would have meant either -(a) delegating to `brain.add()` / `update()` / `relate()` — each of which commits -its own internal transaction before the closure returns, so multi-write rollback -wouldn't actually work (a footgun), or (b) threading `tx?` through every internal -write path — ~2 days of refactor with regression surface, and the API shape changes -again in 8.0 anyway. - -The SDK-scheduler use case that drove the thread (read-then-CAS lock pattern) is -fully solved by `_rev` + `ifRev`. Multi-write atomicity is the 8.0 `brain.transact()` -use case, where the Datomic-style immutable Db makes it atomic by construction. We -chose not to ship a worse version in the interim. - -### How `_rev` interacts with branches and the snapshot API - -| System | What it tracks | When it advances | -|---|---|---| -| `_rev` (NEW) | Per-entity write counter | Auto, on every successful `update()` | -| `brain.versions.save()` | Named snapshots per entity | Explicit — you call `save()` | -| `brain.fork()` / branches | Whole-brain copy-on-write | Explicit — you call `fork(name)` | -| VFS file versioning | Per-VFS-file snapshots | Same as `brain.versions.save()` | - -`_rev` is independent. Each branch has its own copy of every entity, so each -branch has its own `_rev` per entity (same as every other field under COW). -Snapshots taken via `brain.versions.save()` capture the entity at that moment -including its `_rev` at that time; the snapshot's own `version: number` is the -snapshot index, distinct from `_rev`. - -### Docs - -- **New `docs/guides/optimistic-concurrency.md`** (public, indexed at - soulcraft.com/docs after portal deploy) — full reference: the lock pattern, - read-modify-write retry, idempotent bootstrap, how `_rev` interacts with branches / - snapshots / VFS, what's coming in 8.0. -- `docs/api/README.md` `add()` and `update()` entries get the new params + tips - pointing at the guide. - -### Tests - -- **New `tests/integration/rev-and-ifabsent.test.ts`** (18 tests): rev initialization, - rev bump on update, ifRev pass / fail / omit / legacy-entity-fallback, ifAbsent - with custom id / no id / batch propagation / per-item override, plus a full - SDK-scheduler-style two-concurrent-CAS-updaters scenario. -- Existing suites unchanged: subtype-and-facets 26/26, verb-subtype-and-enforcement - 30/30, strict-mode-self-test 13/13, find-limits 9/9. Unit 1468/1468. - -### Cortex compatibility - -Zero changes required. `_rev` is a metadata column already supported by -`NativeColumnStore`; the auto-bump runs in Brainy JS before any storage call. -Cortex never sees the per-entity counter — it's a single integer field updated -alongside every other metadata field on the existing write paths. - -### 8.0 forward-compat - -`_rev`, `ifRev`, `RevisionConflictError`, and `ifAbsent` all survive the 8.0 Db -redesign unchanged. 8.0 layers `brain.transact(tx, { ifAtGeneration })` for -whole-tx CAS on top of the same per-entity mechanism — per-entity for single-record -patterns, generation-based for "did the world move under me." - ---- - -## v7.30.2 — 2026-06-08 - -**Affected products:** any consumer with `find({ limit })` call sites that pass values -≥ ~9000 — a common safety-cap pattern in production code (`limit: 10_000` against -type-filtered queries that typically return 10–500 entities). Additive; drop-in -from 7.30.1. - -### Why - -Brainy 7.30 introduced a memory-derived cap on `find({ limit })` to prevent OOM -from runaway queries. The cap was sound in intent but **~4× too conservative in -calibration**: the formula assumed 100 KB per result while typical entity footprint -is 7-10 KB (384-dim float32 vector ≈ 1.5 KB + standard fields + metadata). On a 900 MB -free-memory box this capped `limit` at 9000, breaking production dashboards where -cascading 500s from queries with `limit: 10_000` silently degraded the `Promise.all` -loading the canvas. - -7.30.2 recalibrates the formula and changes the enforcement from synchronous-throw -to two-tier (warn-then-throw) so existing pre-7.30 code doesn't break on upgrade. - -### Recalibrated formula — `25 KB` per result (was `100 KB`) - -The auto-configured cap now uses a realistic per-result size: - -| Memory source | 7.30.0–7.30.1 cap | 7.30.2 cap | -|---|---|---| -| 4 GB Cloud Run container | 10 000 | 40 000 | -| 2 GB container | 5 000 | 20 000 | -| 900 MB free system memory | 9 000 | ~36 000 | -| `reservedQueryMemory: 1 GB` | 10 000 | 40 000 | - -Hard ceiling stays at 100 000. Existing `BrainyConfig.maxQueryLimit` / -`reservedQueryMemory` overrides continue to work unchanged. - -### Two-tier enforcement — warn, then throw - -**Below cap (`limit ≤ maxLimit`):** silent pass. No signal, no friction. Unchanged. - -**Soft tier (`maxLimit < limit ≤ 2 × maxLimit`)** *NEW*: one-time warning per call -site (dedup keyed on stack location + limit value), query proceeds. Pre-7.30.2 code -that relied on the cap silently allowing safety-cap limits keeps working — the -warning teaches the recipe so consumers can fix it intentionally. - -**Hard tier (`limit > 2 × maxLimit`)** *NEW*: throw with the same message format the -warning uses. Real OOM territory; the cap stops being a recommendation and becomes -a guardrail. - -The 2× soft margin is chosen to absorb existing safety-cap patterns (`limit: 10_000` -against a 9 K-cap box) without disabling OOM protection. Real OOM territory on a JS -in-memory brain is hundreds of thousands of results, not 10× the safety cap. - -### Improved error / warning message (mirrors the 7.30.1 enforcement-error format) - -Before (7.30 → 7.30.1): -``` -limit exceeds auto-configured maximum of 9000 (based on available memory) -``` - -After (7.30.2): -``` -find({ limit: 10000 }) exceeds the auto-configured query limit of 9000 (basis: -available free memory). Choose one: - • Increase the cap: new Brainy({ maxQueryLimit: 10000 }) - • Reserve more memory: new Brainy({ reservedQueryMemory: 256000000 }) - • Paginate: split the query with { limit, offset } pages - at OrderService.loadDashboard (/app/src/orders/dashboard.ts:142:18) -Docs: https://soulcraft.com/docs/guides/find-limits -``` - -Caller location comes from the same `findCallerLocation()` helper introduced in -7.30.1 (now extracted to `src/utils/callerLocation.ts` so both subtype enforcement -and limit enforcement share it). - -### New docs - -New `docs/guides/subtypes-and-facets.md`-style guide at -`docs/guides/find-limits.md` (public, indexed) — explains the cap, the four memory -sources the auto-config considers, the three escape valves (`maxQueryLimit`, -`reservedQueryMemory`, pagination), and when to use which. Explicit "pagination is -the future-proof pattern" callout — the cap can get tighter in 8.0 (Datomic-style -`Db.find()` may make per-call limits stricter to keep snapshot semantics cheap), -but pagination keeps working unchanged. - -`docs/api/README.md` `find()` entry gets a one-paragraph `limit` tip + pointer -to the new guide. - -### Tests - -- **New `tests/integration/find-limits.test.ts`** (9 tests): below-cap silent - pass; soft-tier warns once per call site (dedup verified by exercising same vs. - different source lines); soft-tier message format (names all three escape - valves + docs link); soft-tier message includes caller location; hard-tier - throws; hard-tier message format same as soft-tier; consumer `maxQueryLimit` - override raises the cap and shifts both tiers accordingly; **the pre-7.30.2 - regression scenario** explicitly covered — `limit: 10_000` against a - `maxQueryLimit: 9000` cap now warns and passes instead of throwing. -- Existing unit tests in `tests/unit/utils/memoryLimits.test.ts` updated to - reflect the recalibrated formula (5 tests: hardcoded values for - container / reserved / free-memory paths bumped 4×). -- Existing `paramValidation.test.ts` "should auto-limit based on system memory" - test extended to cover the new three-tier semantics (below-cap pass / - soft-tier silent / hard-tier throw). -- All prior suites still green: subtype-and-facets 26/26, verb-subtype-and- - enforcement 30/30, strict-mode-self-test 13/13. Unit 1468/1468. - -### Cortex compatibility - -**No Cortex changes required for 7.30.2.** Every change is JS-side: formula -recalibration runs in `ValidationConfig.constructor()`, two-tier enforcement runs -in `validateFindParams()`, both fire before any storage/index/Cortex call. Cortex -2.x and the pending Cortex 3.0 work both stay compatible without code changes. - -The new `docs/guides/find-limits.md` calls out that 8.0's Datomic-style `Db.find()` -may tighten per-call limits; that's a Brainy 8.0 / Cortex 3.0 coordination point -whose rollout staging now also covers query limits. - -### What consumers should do - -- **If you've been carrying a `limit: 9000` workaround for the 7.30.0–7.30.1 - cap:** drop it on bump to 7.30.2 — existing `limit: 10_000` patterns now - warn (teaching the recipe) but no longer throw. The warning is - one-time-per-call-site, so production logs don't spam. -- **All other consumers:** no action required if no enforcement-error was being - hit. If you see new `[Brainy]` warnings in logs after upgrade, follow the - recipe in the warning or read `docs/guides/find-limits.md`. -- **SDK wrappers:** wrapper-level pools that auto-construct `Brainy` instances - should consider surfacing `maxQueryLimit` / `reservedQueryMemory` as - consumer-facing config; Brainy now documents the knob explicitly. - ---- - -## v7.30.1 — 2026-06-08 - -**Affected products:** anyone running a brain where a platform layer (SDK, framework wrapper) -has registered `brain.requireSubtype()` rules on common NounTypes, AND anyone preparing for -the upcoming Brainy 8.0 default-on strict mode. Additive; drop-in from 7.30.0. No behavior -change for consumers not using strict-mode enforcement. - -### Why - -Production incident 2026-06-08: a consumer using SDK 3.20.0 (which registers -`requireSubtype()` rules on `NounType.{Event, Collection, Message, Contract, Media, Document}`) -saw their booking flow start returning 500s because `brain.add({ type: NounType.Event, ... })` -calls in their codebase lacked `subtype`. An audit of Brainy's OWN source revealed 14 HIGH-risk -internal write paths that also omit subtype — VFS move/copy/symlink edges, aggregation -materializer, neural extraction, importers, integrations (Sheets/OData), MCP client, CLI. -Any consumer running `requireSubtype()` rules on those NounTypes was one step away from breaking -Brainy's own infrastructure paths, not just their own code. 7.30.1 closes both gaps before -8.0 ships and makes strict mode the default. - -### New — `brain.audit()` diagnostic - -Find entities and relationships missing a `subtype` value, grouped by type. The companion to -`migrateField()` (and to 8.0's `fillSubtypes()`): answers "what would break if I enabled -strict subtype enforcement?". - -```typescript -const report = await brain.audit() -// { -// entitiesWithoutSubtype: { event: 24, document: 3 }, -// relationshipsWithoutSubtype: { relatedTo: 1402 }, -// total: 1429, -// scanned: 8400, -// recommendation: 'Found 1429 entries without subtype. Migrate via `brain.migrateField()` -// (7.x) — or wait for `brain.fillSubtypes()` (8.0) which closes the same -// gap with caller-supplied rules.' -// } -``` - -VFS infrastructure entities are excluded by default (they bypass enforcement via -`metadata.isVFSEntity` markers). Pass `{ includeVFS: true }` to surface them. - -### New — Improved enforcement error messages - -The error fired when subtype enforcement rejects a write now includes: - -1. **The caller's source location** — extracted from the JavaScript stack so you see your own - call site, not a Brainy internal frame. Eliminates the "grep your repo for `brain.add`" step. -2. **Specific guidance** — points at the registered vocabulary when one exists; mentions - brain-wide strict mode and the `except` escape valve when not; otherwise the - `brain.requireSubtype()` registration recipe. -3. **A documentation link** — `https://soulcraft.com/docs/guides/subtypes-and-facets#strict-mode` - for the canonical migration recipe. - -Before: -``` -add(): NounType.Event requires subtype but got undefined. Register vocabulary via brain.requireSubtype(). -``` - -After: -``` -add(): NounType.event requires subtype but got undefined. - at OrderService.getOrCreateByToken (/app/src/orders/draft.ts:42:23) - Pass one of: standard, recurring, milestone. - Migration recipe: https://soulcraft.com/docs/guides/subtypes-and-facets#strict-mode -``` - -### Internal subtype labels — Brainy's own infrastructure paths - -Every internal Brainy write path now sets a stable, queryable `subtype`. Consumers don't need -to do anything for these — they're documented here so you can query Brainy-managed data: - -| Code path | NounType / VerbType | Subtype label | -|---|---|---| -| VFS root directory `/` | `Collection` | `'vfs-root'` | -| VFS subdirectories | `Collection` | `'vfs-directory'` | -| VFS files | mime-driven (e.g. `Document`/`Code`/`Image`) | `'vfs-file'` | -| VFS symlinks | `File` | `'vfs-symlink'` (NEW — distinct from `'vfs-file'`) | -| VFS Contains edges (create + move/copy/symlink/batch) | `Contains` | `'vfs-contains'` | -| Aggregation materialized output | `Measurement` | `'materialized-aggregate'` | -| Import-source provenance entity | `Document` | `'import-source'` | -| Importer-extracted entities | extractor-driven type | `'imported'` | -| Importer placeholder targets | `Thing` | `'import-placeholder'` | -| Neural extraction | extractor-driven type | `'extracted'` | -| GoogleSheets API entity writes | request-driven | `'imported-from-sheets'` | -| OData API entity writes | request-driven | `'imported-from-odata'` | -| MCP message storage | `Message` | `'mcp-message'` | -| `brainy add` CLI default | user-supplied type | `'cli-add'` | -| `brainy relate` CLI default | user-supplied verb | `'cli-relate'` | - -Query examples: - -```typescript -// Every VFS-managed file in your brain -await brain.find({ subtype: 'vfs-file' }) - -// Document breakdown — distinguishes import-source from extracted/imported/user content -brain.counts.bySubtype(NounType.Document) -// → { 'import-source': 12, 'imported': 847, 'extracted': 34, 'vfs-file': 102, ... } -``` - -### Caller-supplied `defaultSubtype` on importers and extraction - -Importer + extraction paths now accept a caller-supplied `defaultSubtype` config so consumers -can tag a whole batch with their own provenance label instead of the Brainy default: - -```typescript -// SmartImportOrchestrator / ImportCoordinator / NeuralImport all accept this -await brain.importer.import(file, { - defaultSubtype: 'customer-upload-2026q2', // your batch label - // ... -}) -``` - -Precedence: extractor-set subtype (highest) → caller's `defaultSubtype` → Brainy default -(`'imported'` for importers, `'extracted'` for extractors). - -### CLI `--subtype` flag - -`brainy add` and `brainy relate` gain a `--subtype ` (`-s`) flag for use with -strict-mode brains: - -```bash -brainy add "Avery Brooks — runs the AI lab" --type person --subtype employee -brainy relate alice manages bob --subtype direct -``` - -When the flag isn't supplied, the CLI uses `'cli-add'` / `'cli-relate'` as defaults so -ad-hoc CLI usage still works against strict-mode brains. - -### Cortex compatibility - -**No Cortex changes required for 7.30.1.** Every change is JS-side: internal subtype labels are -arbitrary strings stored transparently by Cortex; `brain.audit()` runs purely on JS via existing -`storage.getNouns()` / `getVerbs()` pagination; the CLI is JS-only; error messages fire in -Brainy before any native call. - -**For Cortex 3.0 (forward-looking, not blocking):** - -- **Native `audit()` proxy.** For billion-scale brains, `audit()` walks every entity (O(N)). - A native implementation reading from a "null-subtype" bitmap in the column store would be - O(buckets). -- **Strict-mode parity test.** Cortex should mirror Brainy's new - `tests/integration/strict-mode-self-test.test.ts` against their native paths to catch any - latent bug where native writes bypass JS validation. -- **Reserved-label awareness (optional).** Brainy's internal labels (`'vfs-*'`, - `'materialized-aggregate'`, `'imported'`, `'extracted'`, `'mcp-message'`, `'cli-*'`) become - a documented part of the 8.0 contract; useful for telemetry that surfaces "X% of entities are - Brainy-managed infrastructure". - -### Docs - -Updated `docs/guides/subtypes-and-facets.md` with a new "Strict mode in practice" section -covering the SDK_CORE_VOCABULARY pattern, a 4-step migration recipe, the Brainy-internal label -reference table, and an 8.0 forward-look. `docs/api/README.md` documents `brain.audit()` and -adds a strict-mode tip to the `add()` / `relate()` reference entries. - -### Tests - -- **New `tests/integration/strict-mode-self-test.test.ts`** (13 tests): creates a brain under - a realistic consumer vocabulary shape + brain-wide strict mode, then exercises every - internal Brainy path (VFS root/mkdir/writeFile/cp/mv/ln, aggregation engine, audit - diagnostic, error-message UX). Zero rejections expected. -- Existing 7.30.0 + 7.29.0 integration suites unchanged: 26/26 + 30/30. -- Unit suite unchanged: 1468/1468. - ---- - -## v7.30.0 — 2026-06-05 - -**Affected products:** consumers modeling typed relationships with sub-classification -(direct vs dotted-line management; spouse / sibling / colleague; collaborator vs competitor; -etc.), and anyone wanting to enforce the pairing of `type` + `subtype` on every write. -Additive; drop-in from 7.29.x. No deprecations. - -### Symmetric — `subtype` on relationships (parity with 7.29.0 nouns) - -`subtype?: string` is now a first-class standard field on every relationship, mirroring the -noun-side work shipped in 7.29.0. Verbs and nouns are now first-class peers — every API -available on the noun side has a verb-side mirror. - -```typescript -const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' }) -const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' }) - -await brain.relate({ - from: ceoId, - to: vpId, - type: VerbType.ReportsTo, - subtype: 'direct' // sub-classification on the edge -}) -``` - -**Read & filter:** - -```typescript -// Fast-path filter — column-store hit, not metadata fallback -const direct = await brain.getRelations({ - from: ceoId, - type: VerbType.ReportsTo, - subtype: 'direct' -}) - -// Set membership -const all = await brain.getRelations({ - from: ceoId, - type: VerbType.ReportsTo, - subtype: ['direct', 'dotted-line'] -}) - -// Traversal filter (depth-1 in JS; multi-hop lands on Cortex native) -const reports = await brain.find({ - connected: { from: ceoId, via: VerbType.ReportsTo, subtype: 'direct', depth: 1 } -}) -``` - -### New — `updateRelation()` closes a long-standing gap - -Verbs previously had no update method — the only way to change a relationship was -delete-then-recreate, which lost the relation id. 7.30 ships `brain.updateRelation()`: - -```typescript -await brain.updateRelation({ id: relId, subtype: 'dotted-line' }) -await brain.updateRelation({ id: relId, weight: 0.5, confidence: 0.9 }) - -// Change verb type — re-indexes in graph adjacency, id preserved -await brain.updateRelation({ id: relId, type: VerbType.WorksWith }) -``` - -### New — O(1) verb subtype counts via the persisted rollup - -`_system/verb-subtype-statistics.json` mirrors the noun-side rollup shipped in 7.29.0. -Per-VerbType-per-subtype counts are maintained incrementally and persisted; reads are O(1): - -```typescript -brain.counts.byRelationshipSubtype(VerbType.ReportsTo) -// → { direct: 12, 'dotted-line': 3 } - -brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct') // O(1) point -// → 12 - -brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3) -// → [['direct', 12], ['dotted-line', 3]] - -brain.relationshipSubtypesOf(VerbType.ReportsTo) -// → ['direct', 'dotted-line'] -``` - -### New — `brain.requireSubtype(type, options)` per-type enforcement - -Unified API for noun OR verb types — register specific types as requiring a subtype, -optionally with a fixed vocabulary: - -```typescript -brain.requireSubtype(NounType.Person, { - values: ['employee', 'customer', 'vendor'], - required: true -}) - -brain.requireSubtype(VerbType.ReportsTo, { - values: ['direct', 'dotted-line'], - required: true -}) - -// Now this throws — Person requires subtype: -await brain.add({ type: NounType.Person, data: 'no subtype' }) - -// And this throws — 'matrix' isn't in the registered vocabulary: -await brain.relate({ from: a, to: b, type: VerbType.ReportsTo, subtype: 'matrix' }) -``` - -### New — Brain-wide strict mode - -`new Brainy({ requireSubtype: true })` enforces subtype on every public write across the -whole brain. Composes with per-type rules; per-type rules win when both apply. - -```typescript -// Every write must include subtype -const brain = new Brainy({ requireSubtype: true }) - -// Exempt specific types (e.g. catch-all Thing) -const brain2 = new Brainy({ - requireSubtype: { except: [NounType.Thing, NounType.Custom] } -}) -``` - -When strict mode is on: -- Every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()` - rejects writes missing a subtype on a non-exempt type. -- `addMany()` and `relateMany()` validate every item BEFORE any storage write — - atomic-fail semantics, no partial writes. -- Brainy's own infrastructure writes (VFS root, directories, files) bypass via the - `metadata.isVFSEntity: true` marker so existing consumers' VFS usage continues to work. - -The brain-wide flag becomes the default in 8.0.0; the type-level `subtype` field becomes -required at the type system level. The full 8.0 contract upgrade is coordinated through the -internal platform handoff (`CTX-SUBTYPE-8.0-CONTRACT`). - -### `migrateField()` extended to verbs - -The migration helper shipped in 7.29.0 now walks verbs too via the new `entityKind` option: - -```typescript -// Migrate verb-side metadata.kind → top-level subtype -await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype', - entityKind: 'verb' -}) - -// Or walk nouns and verbs in one pass -await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype', - entityKind: 'both' -}) -``` - -Default is `entityKind: 'noun'` (backward-compatible). - -### Symmetry — noun + verb capability matrix - -7.30 closes every gap between nouns and verbs. Every capability available on the noun side -has a verb-side mirror: - -| Capability | Nouns | Verbs | -|---|---|---| -| `subtype` top-level field | ✓ (7.29) | ✓ (7.30 new) | -| Standard-field set | `STANDARD_ENTITY_FIELDS` | `STANDARD_VERB_FIELDS` (new) | -| Field resolver helper | `resolveEntityField` | `resolveVerbField` (new) | -| Statistics rollup | `_system/subtype-statistics.json` | `_system/verb-subtype-statistics.json` (new) | -| Counts breakdown | `counts.bySubtype` / `topSubtypes` / `subtypesOf` | `counts.byRelationshipSubtype` / `topRelationshipSubtypes` / `relationshipSubtypesOf` (new) | -| Fast-path filter | `find({type, subtype})` | `getRelations({type, subtype})` + `find({connected, subtype})` (new) | -| Update method | `update()` | `updateRelation()` (new — closed pre-7.30 gap) | -| Migration helper | `migrateField()` | `migrateField({entityKind: 'verb'\|'both'})` (new) | -| Enforcement | `requireSubtype(NounType, ...)` | `requireSubtype(VerbType, ...)` — one unified API | -| Brain-wide strict mode | `new Brainy({ requireSubtype })` covers both | same | - -### Cortex compatibility - -Verb subtype works under Cortex out of the box via auto-field-indexing. Cortex parity items -for the verb side (native query planner recognition, `verbSubtypeCountsByType` native rollup, -per-edge subtype on native graph adjacency for multi-hop traversal filtering) ship in the -next Cortex release. Not a Brainy blocker. - -### Docs - -Full guide: `docs/guides/subtypes-and-facets.md` (extended with Layer V + Enforcement -sections). The verb subtype + enforcement APIs are documented in `docs/api/README.md`. - ---- - -## v7.29.0 — 2026-06-04 - -**Affected products:** anyone modeling entities with per-product sub-classification — every -consumer that's been reaching for `metadata.kind`, a custom `metadata.subtype` field, a -`data.kind` shape inside the payload, or similar ad-hoc conventions. Additive; drop-in from -7.28.x. No deprecations. - -### New — `subtype` promoted to a top-level standard field - -`subtype?: string` is now a first-class standard field on every entity, alongside `type` / -`confidence` / `weight`. Use it as the platform-standard primitive for sub-classifying entities -within a NounType (`Person` → `'employee'` / `'customer'`; `Document` → `'invoice'` / `'contract'`; -`Event` → `'milestone'` / `'meeting'`): - -```typescript -await brain.add({ - data: 'Avery Brooks — runs the AI lab', - type: NounType.Person, - subtype: 'employee', // top-level write param - metadata: { department: 'ai-lab' } -}) - -// Fast-path filter — column-store hit, not metadata fallback: -const employees = await brain.find({ type: NounType.Person, subtype: 'employee' }) - -// Set membership: -const internal = await brain.find({ - type: NounType.Person, - subtype: ['employee', 'contractor'] -}) -``` - -Flat string, no hierarchy — your vocabulary, your choice. Brainy stores and counts, never validates. - -### New — O(1) subtype counts via the persisted rollup - -A new `_system/subtype-statistics.json` rollup is maintained incrementally as entities are added, -updated, and deleted — mirroring the existing `nounCountsByType` machinery with the same self-heal -behavior on poison detection. Counts are O(1) at billion scale: - -```typescript -brain.counts.bySubtype(NounType.Person) -// → { employee: 12, customer: 847, vendor: 34 } - -brain.counts.bySubtype(NounType.Person, 'employee') // O(1) point count -// → 12 - -brain.counts.topSubtypes(NounType.Person, 3) -// → [['customer', 847], ['employee', 12], ['vendor', 34]] - -brain.subtypesOf(NounType.Person) -// → ['customer', 'employee', 'vendor'] -``` - -### New — `brain.trackField()` for other metadata facets - -For facets that aren't the *primary* sub-classification (`status`, `source`, `role`, `paradigm`), -`trackField` registers a field for cardinality + per-NounType breakdown stats without promoting -each one to a top-level slot. Piggybacks on the existing aggregation engine, so backfill-on-define -applies — registering on a populated brain scans existing entities on the first query: - -```typescript -brain.trackField('status', { perType: true }) - -await brain.counts.byField('status') -// → { todo: 12, doing: 3, done: 47 } - -await brain.counts.byField('status', { type: NounType.Task }) -// → { todo: 8, doing: 2, done: 30 } - -// Opt-in vocabulary validation: -brain.trackField('priority', { values: ['low', 'medium', 'high'] }) -// brain.add({ ..., metadata: { priority: 'urgent' } }) → throws -``` - -### New — generic `brain.migrateField()` for one-shot rewrites - -Streams every entity and copies a value from one field path to another. Supports top-level -standard fields, `metadata.*`, and `data.*` paths; idempotent; with optional `readBoth: true` -deprecation window that preserves the source field alongside the new one: - -```typescript -// Phase 1: dual-populate (legacy readers still work) -await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype', - readBoth: true -}) - -// ...readers migrate to subtype at their own pace... - -// Phase 2: clear the source field -await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) -// → { scanned: 1500, migrated: 1500, skipped: 0, errors: [] } -``` - -Supports `batchSize` and `onProgress` for large brains. - -### Aggregation composition - -`subtype` is a standard-field group-by dimension out of the box — no `where:` wrapper, no -metadata-fallback overhead: - -```typescript -await brain.find({ - aggregate: { - groupBy: ['type', 'subtype'], - metrics: { count: { op: 'COUNT' } } - } -}) -// [ -// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 }, -// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 }, -// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 }, -// ... -// ] -``` - -### Cortex compatibility - -Subtype works under Cortex out of the box via auto-field-indexing. Cortex will land its own -native-side query-planner recognition + `subtypeCountsByType` native rollup in the next Cortex -release for full parity with `type`'s fast path. Not a Brainy blocker — subtype reads/writes -function correctly with current Cortex. - -### Docs - -Full guide: `docs/guides/subtypes-and-facets.md`. Subtype is documented as a core primitive -throughout `README.md`, `docs/DATA_MODEL.md`, `docs/architecture/finite-type-system.md`, -`docs/api/README.md`, and `docs/QUERY_OPERATORS.md`. - ---- - -## v7.24.0 — 2026-05-26 - -**Affected products:** aggregation/reporting users + anyone calling `extractEntities` on -multi-entity text. Additive; drop-in from 7.23.x. - -### New — array-unnest `groupBy` (tag frequency / faceted counts) - -A `groupBy` dimension can now be `{ field, unnest: true }`: the field holds an array and the -entity contributes once per **distinct** element. Enables tag-frequency / label-count style -aggregates: - -```typescript -brain.defineAggregate({ - name: 'tag_frequency', - source: { type: NounType.Document }, - groupBy: [{ field: 'tags', unnest: true }], - metrics: { count: { op: 'count' } } -}) -// queryAggregate('tag_frequency', { orderBy: 'count', order: 'desc' }) -``` - -Duplicate elements on one entity count once; an entity with an empty/missing array joins no group. - -### Perf — entity extraction batch-embeds candidates - -`extractEntities` / `extractConcepts` now embed all unique candidate spans in a single -`embedBatch` call instead of one `embed()` per candidate (N sequential model calls before). No -behavior change — and with vectors reliably available, the embedding signal more consistently -reinforces correct types (e.g. clearer Organization confidence). Falls back to per-candidate -embedding if a batch call fails. - ---- - -## v7.23.0 — 2026-05-26 - -**Affected products:** anyone using aggregation (stats/dashboards), graph traversal, or entity -extraction. Adds two report APIs and fixes three correctness gaps surfaced by BR-ADV-FEATURES-BUN. -Drop-in upgrade from 7.22.x. - -### New — `brain.queryAggregate(name, params)` - -A first-class report API returning the clean `AggregateResult[]` shape -(`{ groupKey, metrics, count }[]`) directly, instead of the search-`Result` wrapper that -`find({ aggregate })` returns. Accepts `where` / `having` / `orderBy` / `order` / `limit` / `offset`. - -### New — HAVING (filter groups by metric value) - -`find({ aggregate, having: { revenue: { greaterThan: 1000 } } })` (and the same on -`queryAggregate`) filters groups by their computed metrics — the analytics equivalent of SQL -`HAVING`, complementing `where` (which filters group keys). Evaluated per group: **O(groups), -independent of entity count.** - -### Fix — aggregates now backfill when defined over existing data (R1) - -Defining an aggregate on a store that already holds matching entities returned `[]` because -write-time hooks only saw *future* writes. It now backfills from existing entities on first query -(one-time scan via `getNouns`, then incremental) — storage-agnostic, so it works under durable -backends (Cortex) where a brain reopens pre-populated. Also: `groupBy:['noun']` now resolves to -the entity type instead of a single null group. `find({ aggregate })` rows now expose -`groupKey`/`metrics`/`count` at the top level (previously only under `.metadata`). - -### Fix — multi-hop `find({ connected: { depth, via } })` (traversal) - -`depth` and `via` are now honored at **every hop** (previously only the immediate neighbour was -returned, and verb filtering applied to hop 1 only). The BFS is bounded by `limit`. - -### Fix — entity extraction type accuracy (R3) - -`extractEntities` no longer lets a type indicator in one candidate's surrounding text bleed onto -neighbours (e.g. "Corp" in "Sarah Chen founded Acme Corp" no longer types "Sarah Chen" as -Organization). Each candidate is typed by its own span; context may only reinforce the same type. - ---- - -## v7.22.1 — 2026-05-26 - -**Affected products:** Anyone using `extractEntities()` / `extractConcepts()`, native -aggregation (`find({ aggregate })`), or multi-hop graph traversal -(`find({ connected: { depth } })`). Three advanced-API correctness fixes — all reproduce on -Node, so they were **not** Bun-specific despite the original report (BR-ADV-FEATURES-BUN). - -### Entity/concept extraction no longer returns `[]` - -`extractEntities()` / `extractConcepts()` returned empty for entity-rich text. The SmartExtractor -ensemble scored agreeing signals with a weighted **sum** against an absolute 0.60 gate, so a -confident low-weight signal (e.g. a name pattern at 0.82) lost selection to a mediocre -high-weight signal that then failed the gate — dropping the whole result. It now selects and -gates on a normalized weighted **average**. Also: the `confidence` option now actually controls -the threshold (was a dead hardcoded 0.60), and the embedding-signal timeout was raised -100ms → 2000ms so the neural signal isn't silently dropped on slower runtimes. - -*Known limitation:* type accuracy on dense multi-entity sentences is still imperfect — a strong -indicator in one entity's surrounding text can bleed onto neighbours. Tracked separately. - -### `find({ aggregate })` exposes `groupKey` / `metrics` / `count` - -Aggregation always computed correct values, but rows nested them under `.metadata`, so callers -expecting the documented `AggregateResult` shape saw empty-looking rows. Those three fields are -now also present at the top level of each result row. - -### Multi-hop `find({ connected: { depth } })` honours depth - -Previously returned only the immediate (1-hop) neighbour at any depth because the graph-search -path ignored `depth` / `via`. It now performs the full depth-aware traversal. - ---- - -## v7.22.0 — 2026-05-15 - -**Affected products:** All. Fixes a silent-data-loss class affecting `find()` and -`brain.stats()`, plus Cortex-compatibility regression from 7.21.0. Recommended -upgrade for everyone on 7.20.x or 7.21.x. - -### Two correctness fixes - -#### 1. `find({ where })` no longer silently returns `[]` (BR-FIND-WHERE-ZERO) - -The 7.20.0 column-store refactor deleted the legacy sparse-index write path -but left `getStats()` and `getIds()` reading from sparse indices. New -workspaces had no sparse-index files, so: -- `brain.stats().entityCount` was `0` regardless of actual entity count -- `find({ where: { ... } })` returned `[]` regardless of actual matches -- `rebuildIndexesIfNeeded()` saw `0` entries → fired the `[Brainy] CRITICAL ... - Second rebuild result: 0 entries` log → application saw no indexed data - -7.22.0 makes the ColumnStore the single source of truth post-7.20.0: -- `MetadataIndex.getStats()` reads from `ColumnStore.getIndexedFields()` and - `idMapper.size`. Entity count is now accurate across writer → close → - reader-open cycles. -- `MetadataIndex.getIdsFromChunks()` throws `BrainyError(FIELD_NOT_INDEXED)` - when neither column store nor legacy sparse index has the field. -- `MetadataIndex.getIdsForFilter()` catches per-clause, logs - `[brainy] find() where-clause referenced unindexed field(s): ...`, returns - `[]`. Production `find()` is now consistent with `brain.explain()`'s - `path: 'none'` diagnostic. - -Separately, `BaseStorage.getNounType()` was hardcoded to return `'thing'` -since a type-cache removal in commit `42ae5be`. Every noun save attributed -the entity to `'thing'` regardless of its actual type, poisoning -`_system/type-statistics.json` and `brain.stats().entitiesByType`. 7.22.0 -restores type-correct attribution: -- New `nounTypeByIdCache: Map` on `BaseStorage`, populated - in `saveNounMetadata_internal` and consumed in `saveNoun_internal`. -- New `flushCounts()` override that persists `nounCountsByType` / - `verbCountsByType` alongside the per-entity counter. Readers opening the - same directory after a writer flush now see correct per-type counts. - -**Self-heal at init.** If `loadTypeStatistics()` detects the poisoned-state -signature (all nouns attributed to `'thing'` with multiple non-`thing` -entities on disk), it auto-runs `rebuildTypeCounts()` once and rewrites the -file. A `[BaseStorage] Detected poisoned type-statistics.json signature ...` -warning is logged. Operators don't need to take any action — the first 7.22 -open of a directory poisoned by 7.20.0/7.21.0 fixes it. - -**`brainy inspect repair`** can also be used to manually trigger -`rebuildTypeCounts()`. It's now public on `BaseStorage`. - -#### 2. Cortex 2.2.x no longer crashes 7.21.0 boot (BR-DEFENSIVE-INTERFACE) - -7.21.0 added new storage-adapter methods (`supportsMultiProcessLocking`, -`acquireWriterLock`, etc.) and called them unconditionally. Cortex 2.2.0's -mmap storage adapter bundles a pre-7.21 `BaseStorage` and doesn't define -them, so a consumer's 7.21.0 upgrade attempt threw -`TypeError: this.storage.supportsMultiProcessLocking is not a function` -at boot. - -7.22.0 introduces `hasStorageMethod(name)` on Brainy and guards every new- -method call site. Older adapters degrade with a one-line warning at init: -``` -[brainy] Storage adapter `CortexMmapStorage` predates the 7.21 -multi-process methods. Writer locking and the flush-request RPC are -disabled for this directory. Upgrade the plugin (e.g. -`@soulcraft/cortex` ≥2.3.0) for full enforcement. -``` - -### Dead-code cleanup - -Removed `MetadataIndexManager.dirtyChunks`, `dirtySparseIndices`, and the -`flushDirtyMetadata()` no-op machinery. These accumulators stopped being -populated when the sparse-index write path was deleted in 7.20.0; the -remaining 6 callers wasted async hops on an empty Map iteration. - -### Upgrade notes - -- **Behaviour change:** `find({ where: { unindexedField: ... } })` previously - returned `[]` silently. It now logs a one-line warning and still returns - `[]`. The diagnostic message names the field — use - `brain.explain({ where: { ... } })` for full diagnosis. No code change - needed for callers that already handle empty results. -- **Behaviour change (good):** `brain.stats()` and `brain.health()` now - report accurate per-type counts. If you previously relied on the buggy - `entitiesByType.thing` over-count, update consumers. -- **No API breaks.** Pure additive on the public surface. - -### Cortex consumers - -Cortex 2.2.x continues to work against 7.22.0 thanks to the defensive -guards, but for full multi-process protection on Cortex-backed stores and -to make sure Cortex's native MetadataIndex picks up the find-where-zero -fix, Cortex 2.3.0 is recommended. Tracked as `CX-MULTIPROC-METHOD` in the -internal platform handoff. - ---- - -## v7.21.0 — 2026-05-15 - -**Affected products:** All consumers using filesystem storage (production deploys, -local development). - -### Multi-process safety + first-class read-only inspector - -Filesystem storage now enforces **single-writer, many-reader**. Previously, opening -a second writer process against a live data directory silently produced wrong query -results — no error, no warning, just empty or stale data. This release replaces -that footgun with a hard refusal at init time and a dedicated read-only inspection -mode. - -#### New: `Brainy.openReadOnly()` -```ts -const reader = await Brainy.openReadOnly({ - storage: { type: 'filesystem', rootDirectory: '/data/brain' } -}) -const bookings = await reader.find({ where: { entityType: 'booking' } }) -``` -- Does NOT acquire the writer lock — coexists with a live writer. -- Every mutation method throws `Cannot mutate a read-only Brainy instance`. -- `flush()` and `close()` are safe (no-op + clean shutdown). - -#### New: writer lock at init -- `new Brainy({ ... })` (default `mode: 'writer'`) acquires - `/locks/_writer.lock` containing `{ pid, hostname, startedAt, lastHeartbeat, version }`. -- A second writer on the same directory **throws** with the PID, hostname, - heartbeat, and a pointer to `openReadOnly()`. -- Stale-lock detection: same hostname + dead PID OR heartbeat > 60s ago → overwritten with a warning. -- Heartbeat: writer rewrites `lastHeartbeat` every 10s (unref'd timer). -- Override: pass `{ force: true }` to bypass when you've verified the existing lock is stale. - -#### New: `brain.requestFlush({ timeoutMs })` -Cross-process RPC for inspectors to force a fresh snapshot: -- In-process: just calls `flush()`. -- Out-of-process: writes a request file, polls for ack. Times out gracefully. -- Watcher runs in every writer instance (filesystem only). - -#### New: `brain.stats()` -Operator-facing summary: `entityCount`, `entitiesByType`, `relationCount`, -`relationsByType`, `fieldRegistry`, `indexHealth`, `storage.backend`, `writerLock`, `version`. -Designed for `/api/health` endpoints and incident triage. - -#### New: `brain.explain(findParams)` -Shows which index path serves each `where` clause: `column-store` | `sparse-chunked` | `none`. -**This is the answer to "why is `find()` returning empty?"** — `path: "none"` means -the field has no index entries and the query will silently return `[]`. Includes -notes on likely causes (writer hasn't flushed; typo; field genuinely absent). - -#### New: `brain.health()` -Invariant-check battery: HNSW vs metadata count parity, field registry sanity, -`_seeded` entity sweep, writer heartbeat freshness. Returns `{ overall: 'pass' | 'warn' | 'fail', checks: [...] }`. - -#### New: `brainy inspect` CLI (13 subcommands) -All read-only by default (internally uses `openReadOnly()`): -``` -brainy inspect stats -brainy inspect find --type Event --where '{"status":"paid"}' --limit 20 -brainy inspect get -brainy inspect relations --direction both -brainy inspect explain --where '{"entityType":"booking"}' -brainy inspect health -brainy inspect sample --type Event --n 20 -brainy inspect fields -brainy inspect dump --type Event > backup.jsonl -brainy inspect watch --type Event -brainy inspect backup /backups/brain.tar -brainy inspect repair # writer mode — stop live writer first -brainy inspect diff -``` -Default behaviour: ask the writer to flush via the RPC first (skip with `--no-fresh`). - -### Brainy + Cortex - -Cortex segments (`*.cidx`) are immutable mmap files; `MANIFEST.json` uses atomic -rename. The single writer lock at `/locks/_writer.lock` covers both Brainy -and Cortex. Read-only inspectors mmap Cortex segments with zero coordination. - -### What's NOT enforced yet - -- **Cloud storage backends** (S3, GCS, R2, Azure) — no multi-process locking. - A best-effort warning logs in writer mode against non-filesystem backends. -- **The `find({ where })`-returns-0 root-cause bug** — tracked separately. The - new `brain.explain()` and `brain.health()` surface it loudly - (`path: 'none'`, `index-parity warn`) instead of letting it be silent. - -### Upgrade notes - -- **Default behaviour change:** opening a Brainy directory in writer mode while - another writer is live now THROWS where it previously silently produced wrong - results. If you have scripts that intentionally co-opened a writer directory, - switch them to `Brainy.openReadOnly()` or pass `{ force: true }`. -- **Cloud backends unchanged** — no lock acquisition for S3/GCS/R2/Azure. -- All existing APIs unchanged. Pure addition. - -### Docs - -- README "Single-Writer Model" section. -- `docs/concepts/multi-process.md` — full model + Cortex compatibility. -- `docs/guides/inspection.md` — operator recipes. -- JSDoc on every new API. - ---- - -## v7.19.10 — 2026-02-24 - -**Affected products:** All Bun/ESM consumers - -### ESM crypto fix in SSTable - -Replaced `require('crypto')` with `import { createHash } from 'node:crypto'` in the -SSTable implementation. Fixes a crash in Bun and strict ESM environments where -CommonJS `require` is unavailable. - -No API changes — upgrade and redeploy. - ---- - -## v7.19.2 — 2026-02-18 - -**Affected products:** All - -### Metadata index cleanup on delete - -Fixed: metadata indexes were not cleaned up after `delete()` / `deleteMany()`. Stale -index entries could cause phantom results in metadata-filtered queries after deletion. - -No API changes. If you were seeing ghost results in filtered queries, this fixes it. - ---- - -## v7.18.0 — 2026-02-16 - -**Affected products:** Analytics, reporting, and session-summary consumers - -### Aggregation engine - -New `brain.aggregate()` API — incremental SUM, COUNT, AVG, MIN, MAX with GROUP BY -and time window support. Computes over entity collections without loading all records -into memory. - -```typescript -const result = await brain.aggregate({ - collection: 'bookings', - metrics: [ - { field: 'revenue', fn: 'SUM' }, - { field: 'id', fn: 'COUNT' }, - ], - groupBy: 'staffId', - timeWindow: { field: 'createdAt', from: startOfMonth, to: now }, -}) -``` - -SDK exposure: `sdk.brainy.aggregate()` — available once SDK is updated to pass through. - ---- - -## v7.17.0 — 2026-02-09 - -**Affected products:** All (schema evolution, data migrations) - -### Migration system - -New `brain.migrate()` API with error handling, validation, and enterprise hardening. -Run schema migrations reliably across Brainy data directories. - -```typescript -await brain.migrate({ - version: 3, - up: async (brain) => { - // transform entities, rename fields, etc. - }, -}) -``` - ---- - -## v7.16.0 — 2026-02-09 - -**Affected products:** All - -### Data/metadata separation enforced + numeric range queries - -- Entity `data` and `metadata` fields are now strictly separated at the storage layer -- Numeric range queries now supported in metadata filters: `{ age: { $gte: 18, $lt: 65 } }` -- Fixes edge cases where mixed data/metadata storage caused inconsistent query results - -**Breaking for anyone storing numeric values in metadata and relying on range queries:** -verify your filter syntax matches the new `$gte/$lte/$gt/$lt` operators. - ---- - -## v7.15.5 — 2026-02-02 - -**Affected products:** Anyone using `@soulcraft/cortex` plugin - -### Plugin opt-in clarified - -Cortex and other plugins are opt-in. Pass explicitly: -```typescript -new Brainy({ plugins: ['@soulcraft/cortex'] }) -``` -Without `plugins`, no external plugins are loaded regardless of what's installed. - ---- - -## v7.15.2 — 2026-02-01 - -**Affected products:** All (data safety) - -### Graph LSM flush on close - -Fixed: graph LSM-trees were not flushed on `brain.close()`, risking data loss across -restarts. Graph edges written in the final seconds before shutdown are now guaranteed -to be persisted. - -No API changes — upgrade immediately if running Brainy in a long-lived server process. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..9d54e26f --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,389 @@ +# Security Best Practices for Brainy + +## 🔒 Data Security + +### Encryption at Rest +```typescript +const brain = new Brainy({ + storage: { + type: 's3', + options: { + encryption: 'AES256', // Server-side encryption + kmsKeyId: process.env.KMS_KEY_ID // Optional KMS key + } + } +}) +``` + +### Encryption in Transit +- Always use HTTPS/TLS for API endpoints +- Enable SSL for database connections +- Use VPN or private networks for internal communication + +## 🔑 Authentication & Authorization + +### API Key Management +```typescript +// Middleware example +app.use('/api/brainy', (req, res, next) => { + const apiKey = req.headers['x-api-key'] + + if (!apiKey || !isValidApiKey(apiKey)) { + return res.status(401).json({ error: 'Unauthorized' }) + } + + // Rate limit by API key + const limit = getRateLimitForKey(apiKey) + if (exceedsRateLimit(apiKey, limit)) { + return res.status(429).json({ error: 'Rate limit exceeded' }) + } + + next() +}) +``` + +### JWT Authentication +```typescript +import jwt from 'jsonwebtoken' + +// Verify JWT token +app.use('/api/brainy', (req, res, next) => { + const token = req.headers.authorization?.split(' ')[1] + + try { + const decoded = jwt.verify(token, process.env.JWT_SECRET) + req.user = decoded + next() + } catch (error) { + return res.status(401).json({ error: 'Invalid token' }) + } +}) +``` + +## 🛡️ Input Validation & Sanitization + +### Query Validation +```typescript +import { z } from 'zod' + +const SearchSchema = z.object({ + query: z.string().min(1).max(1000), + limit: z.number().min(1).max(100).default(10), + metadata: z.record(z.unknown()).optional() +}) + +app.post('/api/search', async (req, res) => { + try { + const params = SearchSchema.parse(req.body) + const results = await brain.find(params) + res.json(results) + } catch (error) { + if (error instanceof z.ZodError) { + return res.status(400).json({ error: 'Invalid input', details: error.errors }) + } + throw error + } +}) +``` + +### Metadata Sanitization +```typescript +function sanitizeMetadata(metadata: any): any { + // Remove potential XSS vectors + const sanitized = {} + + for (const [key, value] of Object.entries(metadata)) { + // Sanitize keys + const cleanKey = key.replace(/[<>'"]/g, '') + + // Sanitize values + if (typeof value === 'string') { + sanitized[cleanKey] = value.replace(/[<>'"]/g, '') + } else if (typeof value === 'object' && value !== null) { + sanitized[cleanKey] = sanitizeMetadata(value) + } else { + sanitized[cleanKey] = value + } + } + + return sanitized +} + +// Use before adding to brain +const sanitizedData = { + text: sanitizeText(input.text), + metadata: sanitizeMetadata(input.metadata) +} +await brain.add(sanitizedData) +``` + +## 🚦 Rate Limiting + +### Per-IP Rate Limiting +```typescript +import rateLimit from 'express-rate-limit' + +const limiter = rateLimit({ + windowMs: 15 * 60 * 1000, // 15 minutes + max: 100, // Limit each IP to 100 requests per windowMs + message: 'Too many requests from this IP' +}) + +app.use('/api/brainy', limiter) +``` + +### Per-User Rate Limiting +```typescript +const userLimits = new Map() + +function checkUserRateLimit(userId: string, limit = 1000): boolean { + const now = Date.now() + const userRequests = userLimits.get(userId) || [] + + // Remove old requests (older than 1 hour) + const recentRequests = userRequests.filter((time: number) => + now - time < 3600000 + ) + + if (recentRequests.length >= limit) { + return false + } + + recentRequests.push(now) + userLimits.set(userId, recentRequests) + return true +} +``` + +## 🔍 Audit Logging + +### Comprehensive Audit Trail +```typescript +interface AuditLog { + timestamp: Date + userId: string + action: string + resource: string + details: any + ip: string + userAgent: string +} + +class AuditLogger { + async log(entry: AuditLog): Promise { + // Log to secure storage + await this.storage.append('audit.log', JSON.stringify(entry) + '\n') + + // Alert on suspicious activity + if (this.isSuspicious(entry)) { + await this.alertSecurityTeam(entry) + } + } + + private isSuspicious(entry: AuditLog): boolean { + // Check for patterns like: + // - Multiple failed auth attempts + // - Unusual data access patterns + // - Bulk data exports + // - Access from new locations + return false // Implement your logic + } +} + +// Use in your API +app.use(async (req, res, next) => { + const entry: AuditLog = { + timestamp: new Date(), + userId: req.user?.id || 'anonymous', + action: req.method, + resource: req.path, + details: req.body, + ip: req.ip, + userAgent: req.headers['user-agent'] + } + + await auditLogger.log(entry) + next() +}) +``` + +## 🗑️ Data Privacy & GDPR Compliance + +### Right to Deletion +```typescript +async function deleteUserData(userId: string): Promise { + // Find all items belonging to user + const userItems = await brain.find({ + metadata: { userId } + }) + + // Delete each item + for (const item of userItems) { + await brain.delete(item.id) + } + + // Log the deletion + await auditLogger.log({ + timestamp: new Date(), + userId, + action: 'DELETE_USER_DATA', + resource: 'user_data', + details: { itemCount: userItems.length }, + ip: 'system', + userAgent: 'gdpr-compliance' + }) +} +``` + +### Data Export +```typescript +async function exportUserData(userId: string): Promise { + // Get all user data + const items = await brain.find({ + metadata: { userId } + }) + + // Get all relationships + const relationships = [] + for (const item of items) { + const relations = await brain.getRelations(item.id) + relationships.push(...relations) + } + + return { + exportDate: new Date().toISOString(), + userId, + items, + relationships, + metadata: { + itemCount: items.length, + relationshipCount: relationships.length + } + } +} +``` + +## 🚨 Security Headers + +### Express.js Security Headers +```typescript +import helmet from 'helmet' + +app.use(helmet({ + contentSecurityPolicy: { + directives: { + defaultSrc: ["'self'"], + styleSrc: ["'self'", "'unsafe-inline'"], + scriptSrc: ["'self'"], + imgSrc: ["'self'", "data:", "https:"], + }, + }, + hsts: { + maxAge: 31536000, + includeSubDomains: true, + preload: true + } +})) +``` + +## 🔐 Environment Variables + +### Secure Configuration +```bash +# .env.production +NODE_ENV=production +JWT_SECRET= +DATABASE_URL= +AWS_ACCESS_KEY_ID= +AWS_SECRET_ACCESS_KEY= +REDIS_PASSWORD= +ENCRYPTION_KEY=<32-byte-random-key> +``` + +### Runtime Validation +```typescript +import { z } from 'zod' + +const EnvSchema = z.object({ + NODE_ENV: z.enum(['development', 'production', 'test']), + JWT_SECRET: z.string().min(32), + DATABASE_URL: z.string().url(), + AWS_REGION: z.string(), + REDIS_HOST: z.string(), + REDIS_PORT: z.string().transform(Number), + ENCRYPTION_KEY: z.string().length(64) // Hex encoded 32 bytes +}) + +// Validate on startup +try { + const env = EnvSchema.parse(process.env) + console.log('✅ Environment configuration valid') +} catch (error) { + console.error('❌ Invalid environment configuration:', error) + process.exit(1) +} +``` + +## 🛠️ Security Checklist + +### Development +- [ ] Use `.env` files for secrets (never commit) +- [ ] Enable TypeScript strict mode +- [ ] Run security linting (eslint-plugin-security) +- [ ] Use dependency scanning (npm audit) +- [ ] Implement unit tests for auth logic + +### Staging +- [ ] Penetration testing +- [ ] Load testing with security scenarios +- [ ] Review audit logs +- [ ] Test rate limiting +- [ ] Verify encryption working + +### Production +- [ ] Enable all security headers +- [ ] Configure WAF (Web Application Firewall) +- [ ] Set up intrusion detection +- [ ] Enable DDoS protection +- [ ] Configure automated backups +- [ ] Set up security alerts +- [ ] Regular security audits +- [ ] Incident response plan + +## 📊 Monitoring & Alerts + +### Security Metrics +```typescript +// Track and alert on: +const securityMetrics = { + failedAuthAttempts: 0, + rateLimitHits: 0, + suspiciousQueries: 0, + largeDataExports: 0, + unusualAccessPatterns: 0 +} + +// Alert thresholds +const alertThresholds = { + failedAuthAttempts: 10, // per minute + rateLimitHits: 100, // per minute + suspiciousQueries: 5, // per minute + largeDataExports: 10, // per hour +} +``` + +## 🚪 Incident Response + +### Response Plan +1. **Detect** - Monitoring alerts trigger +2. **Contain** - Isolate affected systems +3. **Investigate** - Review audit logs +4. **Remediate** - Fix vulnerability +5. **Recover** - Restore normal operations +6. **Review** - Post-incident analysis + +### Emergency Contacts +- Security Team: security@yourcompany.com +- On-call Engineer: Use PagerDuty +- Legal Team: legal@yourcompany.com +- PR Team: pr@yourcompany.com \ No newline at end of file diff --git a/STORAGE_ADAPTER_QUICK_REFERENCE.md b/STORAGE_ADAPTER_QUICK_REFERENCE.md new file mode 100644 index 00000000..62ed62fe --- /dev/null +++ b/STORAGE_ADAPTER_QUICK_REFERENCE.md @@ -0,0 +1,291 @@ +# Brainy Storage Adapter - Quick Reference Guide + +## File Locations + +``` +src/storage/ +├── baseStorageAdapter.ts # Abstract base class (1,156 lines) +├── baseStorage.ts # Implementation layer (1,098 lines) +├── storageFactory.ts # Factory for adapter selection +├── sharding.ts # UUID-based sharding utilities +├── cacheManager.ts # Cache implementation +└── adapters/ + ├── fileSystemStorage.ts # Node.js file system (2,677 lines) + ├── memoryStorage.ts # In-memory storage (822 lines) + ├── s3CompatibleStorage.ts # AWS S3 / R2 / GCS compat (5000+ lines) + ├── gcsStorage.ts # Google Cloud Storage native (1,835 lines) + ├── opfsStorage.ts # Browser OPFS storage + └── baseStorageAdapter.ts # Base class + +src/coreTypes.ts +└── StorageAdapter interface (27 methods) +``` + +## Storage Adapter Hierarchy + +``` +┌─ StorageAdapter (interface) +│ ├─ StorageAdapter.init() +│ ├─ StorageAdapter.saveNoun() +│ ├─ StorageAdapter.getNouns() +│ └─ ... (23 more methods) +│ +└─ BaseStorageAdapter (abstract class) + ├─ Statistics management + ├─ Throttling detection + ├─ Count tracking (O(1)) + ├─ Service-level statistics + └─ Abstract methods for subclasses + + └─ BaseStorage (abstract class) + ├─ 2-file system routing + ├─ Pagination support + ├─ Metadata handling + ├─ Public API (saveNoun, getNoun, etc.) + └─ Abstract internal methods + + └─ Concrete Adapters + ├─ FileSystemStorage + ├─ MemoryStorage + ├─ S3CompatibleStorage + ├─ GcsStorage + └─ OPFSStorage +``` + +## Abstract Methods to Implement + +When creating a new adapter, extend `BaseStorage` and implement: + +### Noun/Verb Operations (6 methods) +```typescript +protected abstract saveNoun_internal(noun: HNSWNoun): Promise +protected abstract getNoun_internal(id: string): Promise +protected abstract deleteNoun_internal(id: string): Promise +protected abstract saveVerb_internal(verb: HNSWVerb): Promise +protected abstract getVerb_internal(id: string): Promise +protected abstract deleteVerb_internal(id: string): Promise +``` + +### Path Operations (4 methods) +```typescript +protected abstract writeObjectToPath(path: string, data: any): Promise +protected abstract readObjectFromPath(path: string): Promise +protected abstract deleteObjectFromPath(path: string): Promise +protected abstract listObjectsUnderPath(prefix: string): Promise +``` + +### Count Management (2 methods) +```typescript +protected abstract initializeCounts(): Promise +protected abstract persistCounts(): Promise +``` + +### Statistics (2 methods) +```typescript +protected abstract saveStatisticsData(statistics: StatisticsData): Promise +protected abstract getStatisticsData(): Promise +``` + +### Lifecycle (3 methods) +```typescript +abstract init(): Promise +abstract clear(): Promise +abstract getStorageStatus(): Promise +``` + +**Total: 17 abstract methods to implement** + +## Storage Path Structure + +### Modern Entity-Based Structure +``` +storage-root/ +├── entities/ +│ ├── nouns/vectors/{shard}/{id}.json ← Vector data +│ ├── nouns/metadata/{shard}/{id}.json ← Metadata +│ ├── nouns/hnsw/{shard}/{id}.json ← HNSW graph +│ ├── verbs/vectors/{shard}/{id}.json +│ ├── verbs/metadata/{shard}/{id}.json +│ └── verbs/hnsw/{shard}/{id}.json +├── indexes/ +│ ├── metadata/... ← Search indexes +│ └── graph/... +└── _system/ + ├── statistics.json ← Aggregate stats + ├── counts.json ← O(1) totals + └── hnsw-system.json ← HNSW metadata +``` + +### Shard Format +- First 2 hex chars of UUID (00-ff) = 256 shards +- Example: `ab123456-...` → stored in `ab/` directory +- Enables 2.5M+ entities with consistent performance + +## 2-File System Design + +### Vector File (always loaded with HNSW) +```json +{ + "id": "ab123456-...", + "vector": [0.1, 0.2, ...], + "connections": { "0": [...], "1": [...] }, + "level": 2 +} +``` + +### Metadata File (loaded separately) +```json +{ + "noun": "Person", + "name": "Alice", + "email": "alice@example.com", + "createdAt": "...", + "service": "user-service" +} +``` + +**Benefit:** Decouple vector operations from flexible metadata queries + +## Existing Adapters Overview + +### FileSystemStorage (Node.js) +- 2,677 lines +- Sharding with migration support +- File-based locking for multi-process +- Production-ready + +### MemoryStorage (Testing) +- 822 lines +- In-memory Maps +- Fast for testing +- No persistence + +### S3CompatibleStorage (Cloud) +- 5,000+ lines +- AWS S3, Cloudflare R2, GCS (via S3 API) +- Adaptive batching, request coalescing +- High-volume mode, write buffers + +### GcsStorage (Google Cloud) +- 1,835 lines +- Native @google-cloud/storage SDK +- ADC, service account, HMAC auth +- Cache managers, backpressure + +### OPFSStorage (Browser) +- Browser Origin Private File System +- Persistent across sessions +- Modern browsers only + +## Factory Integration + +```typescript +// src/storage/storageFactory.ts +const storage = await createStorage({ + type: 'filesystem', // auto, memory, filesystem, s3, gcs, gcs-native, opfs + path: './data', + s3Storage: { bucketName, region, ... }, + gcsStorage: { bucketName, credentials, ... }, +}) +``` + +## Adding TypeAwareStorageAdapter + +### Recommended Approach: Direct Implementation +```typescript +// src/storage/adapters/typeAwareStorageAdapter.ts +export class TypeAwareStorageAdapter extends BaseStorage { + // Implement 17 abstract methods + // Add type indexing logic + // Track noun/verb types in separate indexes +} +``` + +### Integration Steps +1. Create `/src/storage/adapters/typeAwareStorageAdapter.ts` +2. Add to factory in `/src/storage/storageFactory.ts` +3. Update StorageOptions interface with `type: 'type-aware'` +4. No changes to Brainy.ts or existing adapters needed + +## Key Features Inherited from BaseStorageAdapter + +- **Statistics Caching:** Batches updates for efficiency +- **Throttling Detection:** Handles 429/503 errors +- **Count Management:** O(1) operations with persistence +- **Service Tracking:** Per-service statistics +- **Field Name Tracking:** Metadata field discovery + +## Performance Characteristics + +### O(1) Operations +- `getNounCount()` - total noun count +- `getVerbCount()` - total verb count + +### O(n) Operations +- `getNouns()` - paginated listing (n = page size) +- `getVerbs()` - paginated listing +- `getNounsByNounType()` - filter by type +- `getVerbsBySource()` - filter by source + +### Cloud Storage Features (GCS, S3) +- High-volume mode detection +- Adaptive batching +- Request coalescing for deduplication +- Write buffers for bulk operations +- Backpressure management +- Socket pool management + +## Testing Storage Adapters + +All adapters implement the same interface, so: + +```typescript +// Test with MemoryStorage (fastest) +const storage = new MemoryStorage() + +// Test with FileSystemStorage (persistent) +const storage = new FileSystemStorage('./test-data') + +// All adapters support the same operations +await storage.init() +await storage.saveNoun(noun) +const result = await storage.getNoun(id) +await storage.clear() +``` + +## Brainy Integration + +```typescript +export class Brainy { + private storage!: BaseStorage + + async init(config: BrainyConfig): Promise { + // Factory creates appropriate adapter + this.storage = await createStorage(config.storage) as BaseStorage + await this.storage.init() + + // Pass to HNSW index + this.index = new HNSWIndex(this.storage, ...) + } +} +``` + +Brainy depends on `BaseStorage` interface, not specific adapters. + +## Design Patterns Used + +1. **Factory Pattern** - `createStorage()` selects adapter +2. **Strategy Pattern** - Adapters are interchangeable +3. **Template Method** - BaseStorage defines skeleton +4. **Decorator Pattern** - Can wrap adapters (e.g., TypeAware wrapper) +5. **Adapter Pattern** - Maps different storage backends to same interface + +## Conclusion + +TypeAwareStorageAdapter can be added as a **new adapter alongside existing ones** without: +- Modifying Brainy.ts +- Replacing existing adapters +- Breaking the StorageAdapter interface +- Changing how storage is used throughout the codebase + +Simply extend `BaseStorage`, implement 17 abstract methods, and register in `storageFactory.ts`. diff --git a/STORAGE_FILES_REFERENCE.md b/STORAGE_FILES_REFERENCE.md new file mode 100644 index 00000000..8ddc531d --- /dev/null +++ b/STORAGE_FILES_REFERENCE.md @@ -0,0 +1,431 @@ +# Brainy Storage System - Complete File Reference + +## Core Storage Files + +### 1. Storage Interface Definition +**File:** `src/coreTypes.ts` +- **Lines:** ~250 (StorageAdapter interface) +- **Type:** Interface definition +- **Content:** + - `StorageAdapter` interface (27 methods) + - Supporting types: `HNSWNoun`, `HNSWVerb`, `GraphVerb` + - Statistics types: `StatisticsData`, `ServiceStatistics` + +### 2. Base Storage Adapter (Abstract Class) +**File:** `src/storage/adapters/baseStorageAdapter.ts` +- **Lines:** 1,156 +- **Type:** Abstract base class +- **Purpose:** Common functionality for all adapters +- **Key Features:** + - Statistics caching and batching + - Throttling detection and backoff + - Count management (O(1) operations) + - Service-level statistics tracking + - Field name discovery +- **Implements:** StorageAdapter interface +- **Key Methods:** + - `flushStatistics()` - Write statistics to storage + - `isThrottlingError()` - Detect cloud storage throttling + - `handleThrottling()` - Exponential backoff + - `trackThrottlingEvent()` - Record throttling events + - `incrementStatistic()` - Increment service stats + - `decrementStatistic()` - Decrement service stats + +### 3. Base Storage Implementation +**File:** `src/storage/baseStorage.ts` +- **Lines:** 1,098 +- **Type:** Abstract class extending BaseStorageAdapter +- **Purpose:** Core storage logic (routing, sharding, metadata) +- **Key Features:** + - 2-file system implementation (vectors + metadata) + - UUID-based sharding (256 directories) + - Metadata routing and separation + - Pagination support + - Backward compatibility with legacy paths +- **Implements:** + - Public API (saveNoun, getNoun, etc.) + - Metadata operations (saveNounMetadata, etc.) +- **Key Abstract Methods:** + - `saveNoun_internal()` - Adapter-specific noun save + - `getNoun_internal()` - Adapter-specific noun read + - `writeObjectToPath()` - Generic write + - `readObjectFromPath()` - Generic read + - `listObjectsUnderPath()` - Listing support + +### 4. Storage Factory +**File:** `src/storage/storageFactory.ts` +- **Lines:** ~200 +- **Purpose:** Factory function for adapter selection +- **Function:** `createStorage(options: StorageOptions): Promise` +- **Selection Logic:** + 1. Forced memory/filesystem (testing) + 2. Explicit type selection + 3. Auto-detection (browser vs Node.js) +- **Supported Types:** + - `'memory'` - MemoryStorage + - `'filesystem'` - FileSystemStorage + - `'s3'` - S3CompatibleStorage (AWS S3, R2) + - `'gcs'` - S3CompatibleStorage (GCS S3 API) + - `'gcs-native'` - GcsStorage (native SDK) + - `'opfs'` - OPFSStorage (browser) + +### 5. Sharding Utilities +**File:** `src/storage/sharding.ts` +- **Purpose:** UUID-based sharding helpers +- **Key Functions:** + - `getShardIdFromUuid(id: string): string` - Get first 2 hex chars + - `getShardIdByIndex(index: number): string` - Get shard by index + - `getAllShardIds(): string[]` - All 256 shard IDs +- **Constants:** + - `TOTAL_SHARDS = 256` + - `MIN_SHARD_ID = '00'` + - `MAX_SHARD_ID = 'ff'` + +### 6. Cache Manager +**File:** `src/storage/cacheManager.ts` +- **Purpose:** Generic LRU cache for storage +- **Type:** Generic class `CacheManager` +- **Used By:** GcsStorage, S3CompatibleStorage +- **Features:** + - LRU eviction + - TTL support + - Max size limits + - Batch operations + +## Storage Adapter Implementations + +### FileSystemStorage (Node.js) +**File:** `src/storage/adapters/fileSystemStorage.ts` +- **Lines:** 2,677 +- **Type:** Concrete implementation extending BaseStorage +- **Platform:** Node.js only +- **Storage Target:** Local file system +- **Key Features:** + - Sharding with automatic migration + - File-based locking (multi-process) + - O(1) count persistence + - HNSW index persistence + - Dual-write for migrations + - Path depth migration (0→1, 2→1) + +**Key Methods:** +```typescript +private getNounPath(id: string, depth: number): string +private getVerbPath(id: string, depth: number): string +private getNounMetadataPath(id: string, depth: number): string +async migrateShardingStructure(fromDepth: number, toDepth: number): Promise +async migrateFromOldStructure(): Promise +``` + +**Statistics Tracking:** +- Persists counts to `_system/counts.json` +- Tracks noun/verb type distributions +- Service-level activity timestamps +- Field name discovery + +### MemoryStorage (Testing) +**File:** `src/storage/adapters/memoryStorage.ts` +- **Lines:** 822 +- **Type:** Concrete implementation extending BaseStorage +- **Platform:** Browser & Node.js +- **Storage Target:** In-memory Maps +- **Key Features:** + - Fast for testing + - No persistence (ephemeral) + - Full pagination support + - Filtering capabilities + - 2-file system simulation + +**Key Maps:** +```typescript +private nouns: Map +private verbs: Map +private objectStore: Map // Unified metadata store +``` + +**Methods:** +- `getNouns()` - Paginated noun listing +- `getVerbs()` - Paginated verb listing +- `getMetadataBatch()` - Batch metadata loading + +### S3CompatibleStorage (Cloud) +**File:** `src/storage/adapters/s3CompatibleStorage.ts` +- **Lines:** 5,000+ +- **Type:** Concrete implementation extending BaseStorage +- **Platform:** Node.js (server-side) +- **Storage Targets:** + - Amazon S3 + - Cloudflare R2 (via S3 API) + - Google Cloud Storage (via S3 API) + +**Key Features:** +- Adaptive batching (10-1000 items) +- Request coalescing for deduplication +- High-volume mode detection +- Write buffers for bulk operations +- Socket pool management +- Backpressure system +- Change log tracking +- Cache management (nouns & verbs) + +**Performance Optimization:** +- `nounWriteBuffer` - Batches noun writes +- `verbWriteBuffer` - Batches verb writes +- `requestCoalescer` - Deduplicates requests +- `highVolumeMode` - Activates at >20 pending ops +- `baseBatchSize` - Adaptive from 10 to 1000 + +### GcsStorage (Google Cloud Native) +**File:** `src/storage/adapters/gcsStorage.ts` +- **Lines:** 1,835 +- **Type:** Concrete implementation extending BaseStorage +- **Platform:** Node.js (server-side) +- **Storage Target:** Google Cloud Storage +- **SDK:** `@google-cloud/storage` (native) + +**Key Features:** +- Application Default Credentials (ADC) +- Service Account Key File support +- Service Account Credentials Object +- HMAC Keys (backward compatible) +- Multi-level cache managers +- Backpressure management +- High-volume mode +- Request coalescing + +**Authentication Priority:** +1. ADC (Application Default Credentials) +2. Service Account Key File +3. Service Account Credentials +4. HMAC Keys + +**Cache Managers:** +```typescript +private nounCacheManager: CacheManager +private verbCacheManager: CacheManager +``` + +### OPFSStorage (Browser Storage) +**File:** `src/storage/adapters/opfsStorage.ts` +- **Type:** Concrete implementation extending BaseStorage +- **Platform:** Browser only +- **Storage Target:** Origin Private File System (OPFS) +- **Fallback:** MemoryStorage if OPFS unavailable + +**Browser Compatibility:** +- Chrome 96+ +- Edge 96+ +- Safari 15.1+ + +## Storage Patterns and Utilities + +### Backward Compatibility +**File:** `src/storage/backwardCompatibility.ts` +- **Purpose:** Handle legacy storage paths +- **Features:** + - Path migration detection + - Dual-write during transition + - Graceful fallback to old locations + - Read-from-new, fallback-to-old + +### Metadata Index +**File:** `src/utils/metadataIndex.ts` +- **Purpose:** Build searchable indexes from metadata +- **Used By:** Brainy for fast metadata queries +- **Features:** + - Field name discovery + - Standard field mapping + - Service-level statistics + +### Storage Discovery (Distributed) +**File:** `src/distributed/storageDiscovery.ts` +- **Purpose:** Discover storage config in distributed systems +- **Features:** + - Node coordination + - Storage synchronization + +### Adaptive Backpressure +**File:** `src/utils/adaptiveBackpressure.ts` +- **Purpose:** Flow control for storage operations +- **Used By:** All cloud storage adapters +- **Features:** + - Request queuing + - Throttling detection + - Backoff scheduling + +### Write Buffer +**File:** `src/utils/writeBuffer.ts` +- **Purpose:** Batch write operations +- **Used By:** S3, GCS adapters +- **Features:** + - Configurable batch size + - Flush on timeout + - Deduplication + +### Request Coalescer +**File:** `src/utils/requestCoalescer.ts` +- **Purpose:** Deduplicate concurrent requests +- **Used By:** S3, GCS adapters +- **Features:** + - Request deduplication + - Batch processing + +## Integration Points + +### In Brainy.ts (Main Class) +```typescript +private storage!: BaseStorage + +async init(): Promise { + // Create storage from factory + const storageAdapter = await createStorage(this.config.storage) + this.storage = storageAdapter as BaseStorage + + // Initialize + await this.storage.init() + + // Pass to HNSW index + this.index = new HNSWIndex(this.storage, ...) +} +``` + +### In HNSW Index +```typescript +export class HNSWIndex { + constructor(private storage: StorageAdapter, ...) + + // Uses storage for node/edge persistence + async saveNode(noun: HNSWNoun): Promise + async getNode(id: string): Promise +} +``` + +### In Metadata Index +```typescript +export class MetadataIndexManager { + constructor(storage: StorageAdapter, ...) + + // Uses storage for metadata queries + async getNounsByFilter(filter: Filter): Promise +} +``` + +## Data Flow + +### Saving a Noun +``` +Brainy.add() + ↓ +HNSWIndex.insert() + ↓ +BaseStorage.saveNoun() + ├─ saveNoun_internal() → adapter-specific + ├─ saveNounMetadata() → path routing + └─ updateStatistics() + +FileSystemStorage.saveNoun_internal() + ├─ Create shard directory (ab/) + ├─ Write JSON file + └─ Update counts +``` + +### Querying Nouns +``` +Brainy.search() + ↓ +HNSW.search() + ↓ +BaseStorage.getNoun() + ├─ getNoun_internal() → adapter-specific + └─ getNounMetadata() → path routing + +S3CompatibleStorage.getNoun_internal() + ├─ Check cache + ├─ Download from S3 + ├─ Parse JSON + └─ Update cache +``` + +## Storage Statistics Tracking + +### Stored in `_system/statistics.json` +```json +{ + "nounCount": { "Person": 5, "Company": 2 }, + "verbCount": { "knows": 10, "works_at": 3 }, + "metadataCount": { "user-service": 50 }, + "hnswIndexSize": 15, + "totalNodes": 7, + "totalEdges": 13, + "services": [ + { "name": "user-service", "totalNouns": 5, ... } + ], + "lastUpdated": "2024-10-15T..." +} +``` + +### Stored in `_system/counts.json` +```json +{ + "totalNounCount": 7, + "totalVerbCount": 13, + "entityCounts": { "Person": 5, "Company": 2 }, + "verbCounts": { "knows": 10, "works_at": 3 } +} +``` + +## Type Definitions + +### HNSWNoun (Vector + HNSW) +```typescript +interface HNSWNoun { + id: string + vector: number[] + connections: Map> // level → node IDs + level: number + metadata?: any // Optional in vector file, separate file system +} +``` + +### GraphVerb (Relationship) +```typescript +interface GraphVerb { + id: string + sourceId: string + targetId: string + vector: number[] + type?: string + weight?: number + metadata?: any + // Plus aliases: source, target, verb, embedding + createdAt?: Timestamp + createdBy?: { augmentation: string; version: string } +} +``` + +## Summary Statistics + +| Component | File | Lines | Purpose | +|-----------|------|-------|---------| +| StorageAdapter Interface | coreTypes.ts | 250 | Interface definition | +| BaseStorageAdapter | baseStorageAdapter.ts | 1,156 | Common functionality | +| BaseStorage | baseStorage.ts | 1,098 | Core routing & pagination | +| storageFactory | storageFactory.ts | 200 | Adapter selection | +| FileSystemStorage | fileSystemStorage.ts | 2,677 | Node.js FS | +| MemoryStorage | memoryStorage.ts | 822 | In-memory (test) | +| S3CompatibleStorage | s3CompatibleStorage.ts | 5,000+ | AWS S3 / R2 / GCS | +| GcsStorage | gcsStorage.ts | 1,835 | Google Cloud native | +| sharding.ts | sharding.ts | ~100 | UUID sharding | +| cacheManager.ts | cacheManager.ts | ~200 | LRU cache | +| **TOTAL** | | **~13,000+** | **Complete storage system** | + +## Conclusion + +Brainy's storage architecture is: +- Well-layered (Interface → Abstract → Concrete) +- Extensible (factory pattern) +- Flexible (multiple backends) +- Scalable (sharding, caching, batching) +- Type-safe (full TypeScript support) + +New adapters like TypeAwareStorageAdapter simply extend BaseStorage and implement 17 abstract methods. diff --git a/assets/models/all-MiniLM-L6-v2/tokenizer.json b/assets/models/all-MiniLM-L6-v2/tokenizer.json deleted file mode 100644 index cb202bfe..00000000 --- a/assets/models/all-MiniLM-L6-v2/tokenizer.json +++ /dev/null @@ -1 +0,0 @@ -{"version":"1.0","truncation":{"max_length":128,"strategy":"LongestFirst","stride":0},"padding":{"strategy":{"Fixed":128},"direction":"Right","pad_to_multiple_of":null,"pad_id":0,"pad_type_id":0,"pad_token":"[PAD]"},"added_tokens":[{"id":0,"special":true,"content":"[PAD]","single_word":false,"lstrip":false,"rstrip":false,"normalized":false},{"id":100,"special":true,"content":"[UNK]","single_word":false,"lstrip":false,"rstrip":false,"normalized":false},{"id":101,"special":true,"content":"[CLS]","single_word":false,"lstrip":false,"rstrip":false,"normalized":false},{"id":102,"special":true,"content":"[SEP]","single_word":false,"lstrip":false,"rstrip":false,"normalized":false},{"id":103,"special":true,"content":"[MASK]","single_word":false,"lstrip":false,"rstrip":false,"normalized":false}],"normalizer":{"type":"BertNormalizer","clean_text":true,"handle_chinese_chars":true,"strip_accents":null,"lowercase":true},"pre_tokenizer":{"type":"BertPreTokenizer"},"post_processor":{"type":"TemplateProcessing","single":[{"SpecialToken":{"id":"[CLS]","type_id":0}},{"Sequence":{"id":"A","type_id":0}},{"SpecialToken":{"id":"[SEP]","type_id":0}}],"pair":[{"SpecialToken":{"id":"[CLS]","type_id":0}},{"Sequence":{"id":"A","type_id":0}},{"SpecialToken":{"id":"[SEP]","type_id":0}},{"Sequence":{"id":"B","type_id":1}},{"SpecialToken":{"id":"[SEP]","type_id":1}}],"special_tokens":{"[CLS]":{"id":"[CLS]","ids":[101],"tokens":["[CLS]"]},"[SEP]":{"id":"[SEP]","ids":[102],"tokens":["[SEP]"]}}},"decoder":{"type":"WordPiece","prefix":"##","cleanup":true},"model":{"type":"WordPiece","unk_token":"[UNK]","continuing_subword_prefix":"##","max_input_chars_per_word":100,"vocab":{"[PAD]":0,"[unused0]":1,"[unused1]":2,"[unused2]":3,"[unused3]":4,"[unused4]":5,"[unused5]":6,"[unused6]":7,"[unused7]":8,"[unused8]":9,"[unused9]":10,"[unused10]":11,"[unused11]":12,"[unused12]":13,"[unused13]":14,"[unused14]":15,"[unused15]":16,"[unused16]":17,"[unused17]":18,"[unused18]":19,"[unused19]":20,"[unused20]":21,"[unused21]":22,"[unused22]":23,"[unused23]":24,"[unused24]":25,"[unused25]":26,"[unused26]":27,"[unused27]":28,"[unused28]":29,"[unused29]":30,"[unused30]":31,"[unused31]":32,"[unused32]":33,"[unused33]":34,"[unused34]":35,"[unused35]":36,"[unused36]":37,"[unused37]":38,"[unused38]":39,"[unused39]":40,"[unused40]":41,"[unused41]":42,"[unused42]":43,"[unused43]":44,"[unused44]":45,"[unused45]":46,"[unused46]":47,"[unused47]":48,"[unused48]":49,"[unused49]":50,"[unused50]":51,"[unused51]":52,"[unused52]":53,"[unused53]":54,"[unused54]":55,"[unused55]":56,"[unused56]":57,"[unused57]":58,"[unused58]":59,"[unused59]":60,"[unused60]":61,"[unused61]":62,"[unused62]":63,"[unused63]":64,"[unused64]":65,"[unused65]":66,"[unused66]":67,"[unused67]":68,"[unused68]":69,"[unused69]":70,"[unused70]":71,"[unused71]":72,"[unused72]":73,"[unused73]":74,"[unused74]":75,"[unused75]":76,"[unused76]":77,"[unused77]":78,"[unused78]":79,"[unused79]":80,"[unused80]":81,"[unused81]":82,"[unused82]":83,"[unused83]":84,"[unused84]":85,"[unused85]":86,"[unused86]":87,"[unused87]":88,"[unused88]":89,"[unused89]":90,"[unused90]":91,"[unused91]":92,"[unused92]":93,"[unused93]":94,"[unused94]":95,"[unused95]":96,"[unused96]":97,"[unused97]":98,"[unused98]":99,"[UNK]":100,"[CLS]":101,"[SEP]":102,"[MASK]":103,"[unused99]":104,"[unused100]":105,"[unused101]":106,"[unused102]":107,"[unused103]":108,"[unused104]":109,"[unused105]":110,"[unused106]":111,"[unused107]":112,"[unused108]":113,"[unused109]":114,"[unused110]":115,"[unused111]":116,"[unused112]":117,"[unused113]":118,"[unused114]":119,"[unused115]":120,"[unused116]":121,"[unused117]":122,"[unused118]":123,"[unused119]":124,"[unused120]":125,"[unused121]":126,"[unused122]":127,"[unused123]":128,"[unused124]":129,"[unused125]":130,"[unused126]":131,"[unused127]":132,"[unused128]":133,"[unused129]":134,"[unused130]":135,"[unused131]":136,"[unused132]":137,"[unused133]":138,"[unused134]":139,"[unused135]":140,"[unused136]":141,"[unused137]":142,"[unused138]":143,"[unused139]":144,"[unused140]":145,"[unused141]":146,"[unused142]":147,"[unused143]":148,"[unused144]":149,"[unused145]":150,"[unused146]":151,"[unused147]":152,"[unused148]":153,"[unused149]":154,"[unused150]":155,"[unused151]":156,"[unused152]":157,"[unused153]":158,"[unused154]":159,"[unused155]":160,"[unused156]":161,"[unused157]":162,"[unused158]":163,"[unused159]":164,"[unused160]":165,"[unused161]":166,"[unused162]":167,"[unused163]":168,"[unused164]":169,"[unused165]":170,"[unused166]":171,"[unused167]":172,"[unused168]":173,"[unused169]":174,"[unused170]":175,"[unused171]":176,"[unused172]":177,"[unused173]":178,"[unused174]":179,"[unused175]":180,"[unused176]":181,"[unused177]":182,"[unused178]":183,"[unused179]":184,"[unused180]":185,"[unused181]":186,"[unused182]":187,"[unused183]":188,"[unused184]":189,"[unused185]":190,"[unused186]":191,"[unused187]":192,"[unused188]":193,"[unused189]":194,"[unused190]":195,"[unused191]":196,"[unused192]":197,"[unused193]":198,"[unused194]":199,"[unused195]":200,"[unused196]":201,"[unused197]":202,"[unused198]":203,"[unused199]":204,"[unused200]":205,"[unused201]":206,"[unused202]":207,"[unused203]":208,"[unused204]":209,"[unused205]":210,"[unused206]":211,"[unused207]":212,"[unused208]":213,"[unused209]":214,"[unused210]":215,"[unused211]":216,"[unused212]":217,"[unused213]":218,"[unused214]":219,"[unused215]":220,"[unused216]":221,"[unused217]":222,"[unused218]":223,"[unused219]":224,"[unused220]":225,"[unused221]":226,"[unused222]":227,"[unused223]":228,"[unused224]":229,"[unused225]":230,"[unused226]":231,"[unused227]":232,"[unused228]":233,"[unused229]":234,"[unused230]":235,"[unused231]":236,"[unused232]":237,"[unused233]":238,"[unused234]":239,"[unused235]":240,"[unused236]":241,"[unused237]":242,"[unused238]":243,"[unused239]":244,"[unused240]":245,"[unused241]":246,"[unused242]":247,"[unused243]":248,"[unused244]":249,"[unused245]":250,"[unused246]":251,"[unused247]":252,"[unused248]":253,"[unused249]":254,"[unused250]":255,"[unused251]":256,"[unused252]":257,"[unused253]":258,"[unused254]":259,"[unused255]":260,"[unused256]":261,"[unused257]":262,"[unused258]":263,"[unused259]":264,"[unused260]":265,"[unused261]":266,"[unused262]":267,"[unused263]":268,"[unused264]":269,"[unused265]":270,"[unused266]":271,"[unused267]":272,"[unused268]":273,"[unused269]":274,"[unused270]":275,"[unused271]":276,"[unused272]":277,"[unused273]":278,"[unused274]":279,"[unused275]":280,"[unused276]":281,"[unused277]":282,"[unused278]":283,"[unused279]":284,"[unused280]":285,"[unused281]":286,"[unused282]":287,"[unused283]":288,"[unused284]":289,"[unused285]":290,"[unused286]":291,"[unused287]":292,"[unused288]":293,"[unused289]":294,"[unused290]":295,"[unused291]":296,"[unused292]":297,"[unused293]":298,"[unused294]":299,"[unused295]":300,"[unused296]":301,"[unused297]":302,"[unused298]":303,"[unused299]":304,"[unused300]":305,"[unused301]":306,"[unused302]":307,"[unused303]":308,"[unused304]":309,"[unused305]":310,"[unused306]":311,"[unused307]":312,"[unused308]":313,"[unused309]":314,"[unused310]":315,"[unused311]":316,"[unused312]":317,"[unused313]":318,"[unused314]":319,"[unused315]":320,"[unused316]":321,"[unused317]":322,"[unused318]":323,"[unused319]":324,"[unused320]":325,"[unused321]":326,"[unused322]":327,"[unused323]":328,"[unused324]":329,"[unused325]":330,"[unused326]":331,"[unused327]":332,"[unused328]":333,"[unused329]":334,"[unused330]":335,"[unused331]":336,"[unused332]":337,"[unused333]":338,"[unused334]":339,"[unused335]":340,"[unused336]":341,"[unused337]":342,"[unused338]":343,"[unused339]":344,"[unused340]":345,"[unused341]":346,"[unused342]":347,"[unused343]":348,"[unused344]":349,"[unused345]":350,"[unused346]":351,"[unused347]":352,"[unused348]":353,"[unused349]":354,"[unused350]":355,"[unused351]":356,"[unused352]":357,"[unused353]":358,"[unused354]":359,"[unused355]":360,"[unused356]":361,"[unused357]":362,"[unused358]":363,"[unused359]":364,"[unused360]":365,"[unused361]":366,"[unused362]":367,"[unused363]":368,"[unused364]":369,"[unused365]":370,"[unused366]":371,"[unused367]":372,"[unused368]":373,"[unused369]":374,"[unused370]":375,"[unused371]":376,"[unused372]":377,"[unused373]":378,"[unused374]":379,"[unused375]":380,"[unused376]":381,"[unused377]":382,"[unused378]":383,"[unused379]":384,"[unused380]":385,"[unused381]":386,"[unused382]":387,"[unused383]":388,"[unused384]":389,"[unused385]":390,"[unused386]":391,"[unused387]":392,"[unused388]":393,"[unused389]":394,"[unused390]":395,"[unused391]":396,"[unused392]":397,"[unused393]":398,"[unused394]":399,"[unused395]":400,"[unused396]":401,"[unused397]":402,"[unused398]":403,"[unused399]":404,"[unused400]":405,"[unused401]":406,"[unused402]":407,"[unused403]":408,"[unused404]":409,"[unused405]":410,"[unused406]":411,"[unused407]":412,"[unused408]":413,"[unused409]":414,"[unused410]":415,"[unused411]":416,"[unused412]":417,"[unused413]":418,"[unused414]":419,"[unused415]":420,"[unused416]":421,"[unused417]":422,"[unused418]":423,"[unused419]":424,"[unused420]":425,"[unused421]":426,"[unused422]":427,"[unused423]":428,"[unused424]":429,"[unused425]":430,"[unused426]":431,"[unused427]":432,"[unused428]":433,"[unused429]":434,"[unused430]":435,"[unused431]":436,"[unused432]":437,"[unused433]":438,"[unused434]":439,"[unused435]":440,"[unused436]":441,"[unused437]":442,"[unused438]":443,"[unused439]":444,"[unused440]":445,"[unused441]":446,"[unused442]":447,"[unused443]":448,"[unused444]":449,"[unused445]":450,"[unused446]":451,"[unused447]":452,"[unused448]":453,"[unused449]":454,"[unused450]":455,"[unused451]":456,"[unused452]":457,"[unused453]":458,"[unused454]":459,"[unused455]":460,"[unused456]":461,"[unused457]":462,"[unused458]":463,"[unused459]":464,"[unused460]":465,"[unused461]":466,"[unused462]":467,"[unused463]":468,"[unused464]":469,"[unused465]":470,"[unused466]":471,"[unused467]":472,"[unused468]":473,"[unused469]":474,"[unused470]":475,"[unused471]":476,"[unused472]":477,"[unused473]":478,"[unused474]":479,"[unused475]":480,"[unused476]":481,"[unused477]":482,"[unused478]":483,"[unused479]":484,"[unused480]":485,"[unused481]":486,"[unused482]":487,"[unused483]":488,"[unused484]":489,"[unused485]":490,"[unused486]":491,"[unused487]":492,"[unused488]":493,"[unused489]":494,"[unused490]":495,"[unused491]":496,"[unused492]":497,"[unused493]":498,"[unused494]":499,"[unused495]":500,"[unused496]":501,"[unused497]":502,"[unused498]":503,"[unused499]":504,"[unused500]":505,"[unused501]":506,"[unused502]":507,"[unused503]":508,"[unused504]":509,"[unused505]":510,"[unused506]":511,"[unused507]":512,"[unused508]":513,"[unused509]":514,"[unused510]":515,"[unused511]":516,"[unused512]":517,"[unused513]":518,"[unused514]":519,"[unused515]":520,"[unused516]":521,"[unused517]":522,"[unused518]":523,"[unused519]":524,"[unused520]":525,"[unused521]":526,"[unused522]":527,"[unused523]":528,"[unused524]":529,"[unused525]":530,"[unused526]":531,"[unused527]":532,"[unused528]":533,"[unused529]":534,"[unused530]":535,"[unused531]":536,"[unused532]":537,"[unused533]":538,"[unused534]":539,"[unused535]":540,"[unused536]":541,"[unused537]":542,"[unused538]":543,"[unused539]":544,"[unused540]":545,"[unused541]":546,"[unused542]":547,"[unused543]":548,"[unused544]":549,"[unused545]":550,"[unused546]":551,"[unused547]":552,"[unused548]":553,"[unused549]":554,"[unused550]":555,"[unused551]":556,"[unused552]":557,"[unused553]":558,"[unused554]":559,"[unused555]":560,"[unused556]":561,"[unused557]":562,"[unused558]":563,"[unused559]":564,"[unused560]":565,"[unused561]":566,"[unused562]":567,"[unused563]":568,"[unused564]":569,"[unused565]":570,"[unused566]":571,"[unused567]":572,"[unused568]":573,"[unused569]":574,"[unused570]":575,"[unused571]":576,"[unused572]":577,"[unused573]":578,"[unused574]":579,"[unused575]":580,"[unused576]":581,"[unused577]":582,"[unused578]":583,"[unused579]":584,"[unused580]":585,"[unused581]":586,"[unused582]":587,"[unused583]":588,"[unused584]":589,"[unused585]":590,"[unused586]":591,"[unused587]":592,"[unused588]":593,"[unused589]":594,"[unused590]":595,"[unused591]":596,"[unused592]":597,"[unused593]":598,"[unused594]":599,"[unused595]":600,"[unused596]":601,"[unused597]":602,"[unused598]":603,"[unused599]":604,"[unused600]":605,"[unused601]":606,"[unused602]":607,"[unused603]":608,"[unused604]":609,"[unused605]":610,"[unused606]":611,"[unused607]":612,"[unused608]":613,"[unused609]":614,"[unused610]":615,"[unused611]":616,"[unused612]":617,"[unused613]":618,"[unused614]":619,"[unused615]":620,"[unused616]":621,"[unused617]":622,"[unused618]":623,"[unused619]":624,"[unused620]":625,"[unused621]":626,"[unused622]":627,"[unused623]":628,"[unused624]":629,"[unused625]":630,"[unused626]":631,"[unused627]":632,"[unused628]":633,"[unused629]":634,"[unused630]":635,"[unused631]":636,"[unused632]":637,"[unused633]":638,"[unused634]":639,"[unused635]":640,"[unused636]":641,"[unused637]":642,"[unused638]":643,"[unused639]":644,"[unused640]":645,"[unused641]":646,"[unused642]":647,"[unused643]":648,"[unused644]":649,"[unused645]":650,"[unused646]":651,"[unused647]":652,"[unused648]":653,"[unused649]":654,"[unused650]":655,"[unused651]":656,"[unused652]":657,"[unused653]":658,"[unused654]":659,"[unused655]":660,"[unused656]":661,"[unused657]":662,"[unused658]":663,"[unused659]":664,"[unused660]":665,"[unused661]":666,"[unused662]":667,"[unused663]":668,"[unused664]":669,"[unused665]":670,"[unused666]":671,"[unused667]":672,"[unused668]":673,"[unused669]":674,"[unused670]":675,"[unused671]":676,"[unused672]":677,"[unused673]":678,"[unused674]":679,"[unused675]":680,"[unused676]":681,"[unused677]":682,"[unused678]":683,"[unused679]":684,"[unused680]":685,"[unused681]":686,"[unused682]":687,"[unused683]":688,"[unused684]":689,"[unused685]":690,"[unused686]":691,"[unused687]":692,"[unused688]":693,"[unused689]":694,"[unused690]":695,"[unused691]":696,"[unused692]":697,"[unused693]":698,"[unused694]":699,"[unused695]":700,"[unused696]":701,"[unused697]":702,"[unused698]":703,"[unused699]":704,"[unused700]":705,"[unused701]":706,"[unused702]":707,"[unused703]":708,"[unused704]":709,"[unused705]":710,"[unused706]":711,"[unused707]":712,"[unused708]":713,"[unused709]":714,"[unused710]":715,"[unused711]":716,"[unused712]":717,"[unused713]":718,"[unused714]":719,"[unused715]":720,"[unused716]":721,"[unused717]":722,"[unused718]":723,"[unused719]":724,"[unused720]":725,"[unused721]":726,"[unused722]":727,"[unused723]":728,"[unused724]":729,"[unused725]":730,"[unused726]":731,"[unused727]":732,"[unused728]":733,"[unused729]":734,"[unused730]":735,"[unused731]":736,"[unused732]":737,"[unused733]":738,"[unused734]":739,"[unused735]":740,"[unused736]":741,"[unused737]":742,"[unused738]":743,"[unused739]":744,"[unused740]":745,"[unused741]":746,"[unused742]":747,"[unused743]":748,"[unused744]":749,"[unused745]":750,"[unused746]":751,"[unused747]":752,"[unused748]":753,"[unused749]":754,"[unused750]":755,"[unused751]":756,"[unused752]":757,"[unused753]":758,"[unused754]":759,"[unused755]":760,"[unused756]":761,"[unused757]":762,"[unused758]":763,"[unused759]":764,"[unused760]":765,"[unused761]":766,"[unused762]":767,"[unused763]":768,"[unused764]":769,"[unused765]":770,"[unused766]":771,"[unused767]":772,"[unused768]":773,"[unused769]":774,"[unused770]":775,"[unused771]":776,"[unused772]":777,"[unused773]":778,"[unused774]":779,"[unused775]":780,"[unused776]":781,"[unused777]":782,"[unused778]":783,"[unused779]":784,"[unused780]":785,"[unused781]":786,"[unused782]":787,"[unused783]":788,"[unused784]":789,"[unused785]":790,"[unused786]":791,"[unused787]":792,"[unused788]":793,"[unused789]":794,"[unused790]":795,"[unused791]":796,"[unused792]":797,"[unused793]":798,"[unused794]":799,"[unused795]":800,"[unused796]":801,"[unused797]":802,"[unused798]":803,"[unused799]":804,"[unused800]":805,"[unused801]":806,"[unused802]":807,"[unused803]":808,"[unused804]":809,"[unused805]":810,"[unused806]":811,"[unused807]":812,"[unused808]":813,"[unused809]":814,"[unused810]":815,"[unused811]":816,"[unused812]":817,"[unused813]":818,"[unused814]":819,"[unused815]":820,"[unused816]":821,"[unused817]":822,"[unused818]":823,"[unused819]":824,"[unused820]":825,"[unused821]":826,"[unused822]":827,"[unused823]":828,"[unused824]":829,"[unused825]":830,"[unused826]":831,"[unused827]":832,"[unused828]":833,"[unused829]":834,"[unused830]":835,"[unused831]":836,"[unused832]":837,"[unused833]":838,"[unused834]":839,"[unused835]":840,"[unused836]":841,"[unused837]":842,"[unused838]":843,"[unused839]":844,"[unused840]":845,"[unused841]":846,"[unused842]":847,"[unused843]":848,"[unused844]":849,"[unused845]":850,"[unused846]":851,"[unused847]":852,"[unused848]":853,"[unused849]":854,"[unused850]":855,"[unused851]":856,"[unused852]":857,"[unused853]":858,"[unused854]":859,"[unused855]":860,"[unused856]":861,"[unused857]":862,"[unused858]":863,"[unused859]":864,"[unused860]":865,"[unused861]":866,"[unused862]":867,"[unused863]":868,"[unused864]":869,"[unused865]":870,"[unused866]":871,"[unused867]":872,"[unused868]":873,"[unused869]":874,"[unused870]":875,"[unused871]":876,"[unused872]":877,"[unused873]":878,"[unused874]":879,"[unused875]":880,"[unused876]":881,"[unused877]":882,"[unused878]":883,"[unused879]":884,"[unused880]":885,"[unused881]":886,"[unused882]":887,"[unused883]":888,"[unused884]":889,"[unused885]":890,"[unused886]":891,"[unused887]":892,"[unused888]":893,"[unused889]":894,"[unused890]":895,"[unused891]":896,"[unused892]":897,"[unused893]":898,"[unused894]":899,"[unused895]":900,"[unused896]":901,"[unused897]":902,"[unused898]":903,"[unused899]":904,"[unused900]":905,"[unused901]":906,"[unused902]":907,"[unused903]":908,"[unused904]":909,"[unused905]":910,"[unused906]":911,"[unused907]":912,"[unused908]":913,"[unused909]":914,"[unused910]":915,"[unused911]":916,"[unused912]":917,"[unused913]":918,"[unused914]":919,"[unused915]":920,"[unused916]":921,"[unused917]":922,"[unused918]":923,"[unused919]":924,"[unused920]":925,"[unused921]":926,"[unused922]":927,"[unused923]":928,"[unused924]":929,"[unused925]":930,"[unused926]":931,"[unused927]":932,"[unused928]":933,"[unused929]":934,"[unused930]":935,"[unused931]":936,"[unused932]":937,"[unused933]":938,"[unused934]":939,"[unused935]":940,"[unused936]":941,"[unused937]":942,"[unused938]":943,"[unused939]":944,"[unused940]":945,"[unused941]":946,"[unused942]":947,"[unused943]":948,"[unused944]":949,"[unused945]":950,"[unused946]":951,"[unused947]":952,"[unused948]":953,"[unused949]":954,"[unused950]":955,"[unused951]":956,"[unused952]":957,"[unused953]":958,"[unused954]":959,"[unused955]":960,"[unused956]":961,"[unused957]":962,"[unused958]":963,"[unused959]":964,"[unused960]":965,"[unused961]":966,"[unused962]":967,"[unused963]":968,"[unused964]":969,"[unused965]":970,"[unused966]":971,"[unused967]":972,"[unused968]":973,"[unused969]":974,"[unused970]":975,"[unused971]":976,"[unused972]":977,"[unused973]":978,"[unused974]":979,"[unused975]":980,"[unused976]":981,"[unused977]":982,"[unused978]":983,"[unused979]":984,"[unused980]":985,"[unused981]":986,"[unused982]":987,"[unused983]":988,"[unused984]":989,"[unused985]":990,"[unused986]":991,"[unused987]":992,"[unused988]":993,"[unused989]":994,"[unused990]":995,"[unused991]":996,"[unused992]":997,"[unused993]":998,"!":999,"\"":1000,"#":1001,"$":1002,"%":1003,"&":1004,"'":1005,"(":1006,")":1007,"*":1008,"+":1009,",":1010,"-":1011,".":1012,"/":1013,"0":1014,"1":1015,"2":1016,"3":1017,"4":1018,"5":1019,"6":1020,"7":1021,"8":1022,"9":1023,":":1024,";":1025,"<":1026,"=":1027,">":1028,"?":1029,"@":1030,"[":1031,"\\":1032,"]":1033,"^":1034,"_":1035,"`":1036,"a":1037,"b":1038,"c":1039,"d":1040,"e":1041,"f":1042,"g":1043,"h":1044,"i":1045,"j":1046,"k":1047,"l":1048,"m":1049,"n":1050,"o":1051,"p":1052,"q":1053,"r":1054,"s":1055,"t":1056,"u":1057,"v":1058,"w":1059,"x":1060,"y":1061,"z":1062,"{":1063,"|":1064,"}":1065,"~":1066,"¡":1067,"¢":1068,"£":1069,"¤":1070,"¥":1071,"¦":1072,"§":1073,"¨":1074,"©":1075,"ª":1076,"«":1077,"¬":1078,"®":1079,"°":1080,"±":1081,"²":1082,"³":1083,"´":1084,"µ":1085,"¶":1086,"·":1087,"¹":1088,"º":1089,"»":1090,"¼":1091,"½":1092,"¾":1093,"¿":1094,"×":1095,"ß":1096,"æ":1097,"ð":1098,"÷":1099,"ø":1100,"þ":1101,"đ":1102,"ħ":1103,"ı":1104,"ł":1105,"ŋ":1106,"œ":1107,"ƒ":1108,"ɐ":1109,"ɑ":1110,"ɒ":1111,"ɔ":1112,"ɕ":1113,"ə":1114,"ɛ":1115,"ɡ":1116,"ɣ":1117,"ɨ":1118,"ɪ":1119,"ɫ":1120,"ɬ":1121,"ɯ":1122,"ɲ":1123,"ɴ":1124,"ɹ":1125,"ɾ":1126,"ʀ":1127,"ʁ":1128,"ʂ":1129,"ʃ":1130,"ʉ":1131,"ʊ":1132,"ʋ":1133,"ʌ":1134,"ʎ":1135,"ʐ":1136,"ʑ":1137,"ʒ":1138,"ʔ":1139,"ʰ":1140,"ʲ":1141,"ʳ":1142,"ʷ":1143,"ʸ":1144,"ʻ":1145,"ʼ":1146,"ʾ":1147,"ʿ":1148,"ˈ":1149,"ː":1150,"ˡ":1151,"ˢ":1152,"ˣ":1153,"ˤ":1154,"α":1155,"β":1156,"γ":1157,"δ":1158,"ε":1159,"ζ":1160,"η":1161,"θ":1162,"ι":1163,"κ":1164,"λ":1165,"μ":1166,"ν":1167,"ξ":1168,"ο":1169,"π":1170,"ρ":1171,"ς":1172,"σ":1173,"τ":1174,"υ":1175,"φ":1176,"χ":1177,"ψ":1178,"ω":1179,"а":1180,"б":1181,"в":1182,"г":1183,"д":1184,"е":1185,"ж":1186,"з":1187,"и":1188,"к":1189,"л":1190,"м":1191,"н":1192,"о":1193,"п":1194,"р":1195,"с":1196,"т":1197,"у":1198,"ф":1199,"х":1200,"ц":1201,"ч":1202,"ш":1203,"щ":1204,"ъ":1205,"ы":1206,"ь":1207,"э":1208,"ю":1209,"я":1210,"ђ":1211,"є":1212,"і":1213,"ј":1214,"љ":1215,"њ":1216,"ћ":1217,"ӏ":1218,"ա":1219,"բ":1220,"գ":1221,"դ":1222,"ե":1223,"թ":1224,"ի":1225,"լ":1226,"կ":1227,"հ":1228,"մ":1229,"յ":1230,"ն":1231,"ո":1232,"պ":1233,"ս":1234,"վ":1235,"տ":1236,"ր":1237,"ւ":1238,"ք":1239,"־":1240,"א":1241,"ב":1242,"ג":1243,"ד":1244,"ה":1245,"ו":1246,"ז":1247,"ח":1248,"ט":1249,"י":1250,"ך":1251,"כ":1252,"ל":1253,"ם":1254,"מ":1255,"ן":1256,"נ":1257,"ס":1258,"ע":1259,"ף":1260,"פ":1261,"ץ":1262,"צ":1263,"ק":1264,"ר":1265,"ש":1266,"ת":1267,"،":1268,"ء":1269,"ا":1270,"ب":1271,"ة":1272,"ت":1273,"ث":1274,"ج":1275,"ح":1276,"خ":1277,"د":1278,"ذ":1279,"ر":1280,"ز":1281,"س":1282,"ش":1283,"ص":1284,"ض":1285,"ط":1286,"ظ":1287,"ع":1288,"غ":1289,"ـ":1290,"ف":1291,"ق":1292,"ك":1293,"ل":1294,"م":1295,"ن":1296,"ه":1297,"و":1298,"ى":1299,"ي":1300,"ٹ":1301,"پ":1302,"چ":1303,"ک":1304,"گ":1305,"ں":1306,"ھ":1307,"ہ":1308,"ی":1309,"ے":1310,"अ":1311,"आ":1312,"उ":1313,"ए":1314,"क":1315,"ख":1316,"ग":1317,"च":1318,"ज":1319,"ट":1320,"ड":1321,"ण":1322,"त":1323,"थ":1324,"द":1325,"ध":1326,"न":1327,"प":1328,"ब":1329,"भ":1330,"म":1331,"य":1332,"र":1333,"ल":1334,"व":1335,"श":1336,"ष":1337,"स":1338,"ह":1339,"ा":1340,"ि":1341,"ी":1342,"ो":1343,"।":1344,"॥":1345,"ং":1346,"অ":1347,"আ":1348,"ই":1349,"উ":1350,"এ":1351,"ও":1352,"ক":1353,"খ":1354,"গ":1355,"চ":1356,"ছ":1357,"জ":1358,"ট":1359,"ড":1360,"ণ":1361,"ত":1362,"থ":1363,"দ":1364,"ধ":1365,"ন":1366,"প":1367,"ব":1368,"ভ":1369,"ম":1370,"য":1371,"র":1372,"ল":1373,"শ":1374,"ষ":1375,"স":1376,"হ":1377,"া":1378,"ি":1379,"ী":1380,"ে":1381,"க":1382,"ச":1383,"ட":1384,"த":1385,"ந":1386,"ன":1387,"ப":1388,"ம":1389,"ய":1390,"ர":1391,"ல":1392,"ள":1393,"வ":1394,"ா":1395,"ி":1396,"ு":1397,"ே":1398,"ை":1399,"ನ":1400,"ರ":1401,"ಾ":1402,"ක":1403,"ය":1404,"ර":1405,"ල":1406,"ව":1407,"ා":1408,"ก":1409,"ง":1410,"ต":1411,"ท":1412,"น":1413,"พ":1414,"ม":1415,"ย":1416,"ร":1417,"ล":1418,"ว":1419,"ส":1420,"อ":1421,"า":1422,"เ":1423,"་":1424,"།":1425,"ག":1426,"ང":1427,"ད":1428,"ན":1429,"པ":1430,"བ":1431,"མ":1432,"འ":1433,"ར":1434,"ལ":1435,"ས":1436,"မ":1437,"ა":1438,"ბ":1439,"გ":1440,"დ":1441,"ე":1442,"ვ":1443,"თ":1444,"ი":1445,"კ":1446,"ლ":1447,"მ":1448,"ნ":1449,"ო":1450,"რ":1451,"ს":1452,"ტ":1453,"უ":1454,"ᄀ":1455,"ᄂ":1456,"ᄃ":1457,"ᄅ":1458,"ᄆ":1459,"ᄇ":1460,"ᄉ":1461,"ᄊ":1462,"ᄋ":1463,"ᄌ":1464,"ᄎ":1465,"ᄏ":1466,"ᄐ":1467,"ᄑ":1468,"ᄒ":1469,"ᅡ":1470,"ᅢ":1471,"ᅥ":1472,"ᅦ":1473,"ᅧ":1474,"ᅩ":1475,"ᅪ":1476,"ᅭ":1477,"ᅮ":1478,"ᅯ":1479,"ᅲ":1480,"ᅳ":1481,"ᅴ":1482,"ᅵ":1483,"ᆨ":1484,"ᆫ":1485,"ᆯ":1486,"ᆷ":1487,"ᆸ":1488,"ᆼ":1489,"ᴬ":1490,"ᴮ":1491,"ᴰ":1492,"ᴵ":1493,"ᴺ":1494,"ᵀ":1495,"ᵃ":1496,"ᵇ":1497,"ᵈ":1498,"ᵉ":1499,"ᵍ":1500,"ᵏ":1501,"ᵐ":1502,"ᵒ":1503,"ᵖ":1504,"ᵗ":1505,"ᵘ":1506,"ᵢ":1507,"ᵣ":1508,"ᵤ":1509,"ᵥ":1510,"ᶜ":1511,"ᶠ":1512,"‐":1513,"‑":1514,"‒":1515,"–":1516,"—":1517,"―":1518,"‖":1519,"‘":1520,"’":1521,"‚":1522,"“":1523,"”":1524,"„":1525,"†":1526,"‡":1527,"•":1528,"…":1529,"‰":1530,"′":1531,"″":1532,"›":1533,"‿":1534,"⁄":1535,"⁰":1536,"ⁱ":1537,"⁴":1538,"⁵":1539,"⁶":1540,"⁷":1541,"⁸":1542,"⁹":1543,"⁺":1544,"⁻":1545,"ⁿ":1546,"₀":1547,"₁":1548,"₂":1549,"₃":1550,"₄":1551,"₅":1552,"₆":1553,"₇":1554,"₈":1555,"₉":1556,"₊":1557,"₍":1558,"₎":1559,"ₐ":1560,"ₑ":1561,"ₒ":1562,"ₓ":1563,"ₕ":1564,"ₖ":1565,"ₗ":1566,"ₘ":1567,"ₙ":1568,"ₚ":1569,"ₛ":1570,"ₜ":1571,"₤":1572,"₩":1573,"€":1574,"₱":1575,"₹":1576,"ℓ":1577,"№":1578,"ℝ":1579,"™":1580,"⅓":1581,"⅔":1582,"←":1583,"↑":1584,"→":1585,"↓":1586,"↔":1587,"↦":1588,"⇄":1589,"⇌":1590,"⇒":1591,"∂":1592,"∅":1593,"∆":1594,"∇":1595,"∈":1596,"−":1597,"∗":1598,"∘":1599,"√":1600,"∞":1601,"∧":1602,"∨":1603,"∩":1604,"∪":1605,"≈":1606,"≡":1607,"≤":1608,"≥":1609,"⊂":1610,"⊆":1611,"⊕":1612,"⊗":1613,"⋅":1614,"─":1615,"│":1616,"■":1617,"▪":1618,"●":1619,"★":1620,"☆":1621,"☉":1622,"♠":1623,"♣":1624,"♥":1625,"♦":1626,"♭":1627,"♯":1628,"⟨":1629,"⟩":1630,"ⱼ":1631,"⺩":1632,"⺼":1633,"⽥":1634,"、":1635,"。":1636,"〈":1637,"〉":1638,"《":1639,"》":1640,"「":1641,"」":1642,"『":1643,"』":1644,"〜":1645,"あ":1646,"い":1647,"う":1648,"え":1649,"お":1650,"か":1651,"き":1652,"く":1653,"け":1654,"こ":1655,"さ":1656,"し":1657,"す":1658,"せ":1659,"そ":1660,"た":1661,"ち":1662,"っ":1663,"つ":1664,"て":1665,"と":1666,"な":1667,"に":1668,"ぬ":1669,"ね":1670,"の":1671,"は":1672,"ひ":1673,"ふ":1674,"へ":1675,"ほ":1676,"ま":1677,"み":1678,"む":1679,"め":1680,"も":1681,"や":1682,"ゆ":1683,"よ":1684,"ら":1685,"り":1686,"る":1687,"れ":1688,"ろ":1689,"を":1690,"ん":1691,"ァ":1692,"ア":1693,"ィ":1694,"イ":1695,"ウ":1696,"ェ":1697,"エ":1698,"オ":1699,"カ":1700,"キ":1701,"ク":1702,"ケ":1703,"コ":1704,"サ":1705,"シ":1706,"ス":1707,"セ":1708,"タ":1709,"チ":1710,"ッ":1711,"ツ":1712,"テ":1713,"ト":1714,"ナ":1715,"ニ":1716,"ノ":1717,"ハ":1718,"ヒ":1719,"フ":1720,"ヘ":1721,"ホ":1722,"マ":1723,"ミ":1724,"ム":1725,"メ":1726,"モ":1727,"ャ":1728,"ュ":1729,"ョ":1730,"ラ":1731,"リ":1732,"ル":1733,"レ":1734,"ロ":1735,"ワ":1736,"ン":1737,"・":1738,"ー":1739,"一":1740,"三":1741,"上":1742,"下":1743,"不":1744,"世":1745,"中":1746,"主":1747,"久":1748,"之":1749,"也":1750,"事":1751,"二":1752,"五":1753,"井":1754,"京":1755,"人":1756,"亻":1757,"仁":1758,"介":1759,"代":1760,"仮":1761,"伊":1762,"会":1763,"佐":1764,"侍":1765,"保":1766,"信":1767,"健":1768,"元":1769,"光":1770,"八":1771,"公":1772,"内":1773,"出":1774,"分":1775,"前":1776,"劉":1777,"力":1778,"加":1779,"勝":1780,"北":1781,"区":1782,"十":1783,"千":1784,"南":1785,"博":1786,"原":1787,"口":1788,"古":1789,"史":1790,"司":1791,"合":1792,"吉":1793,"同":1794,"名":1795,"和":1796,"囗":1797,"四":1798,"国":1799,"國":1800,"土":1801,"地":1802,"坂":1803,"城":1804,"堂":1805,"場":1806,"士":1807,"夏":1808,"外":1809,"大":1810,"天":1811,"太":1812,"夫":1813,"奈":1814,"女":1815,"子":1816,"学":1817,"宀":1818,"宇":1819,"安":1820,"宗":1821,"定":1822,"宣":1823,"宮":1824,"家":1825,"宿":1826,"寺":1827,"將":1828,"小":1829,"尚":1830,"山":1831,"岡":1832,"島":1833,"崎":1834,"川":1835,"州":1836,"巿":1837,"帝":1838,"平":1839,"年":1840,"幸":1841,"广":1842,"弘":1843,"張":1844,"彳":1845,"後":1846,"御":1847,"德":1848,"心":1849,"忄":1850,"志":1851,"忠":1852,"愛":1853,"成":1854,"我":1855,"戦":1856,"戸":1857,"手":1858,"扌":1859,"政":1860,"文":1861,"新":1862,"方":1863,"日":1864,"明":1865,"星":1866,"春":1867,"昭":1868,"智":1869,"曲":1870,"書":1871,"月":1872,"有":1873,"朝":1874,"木":1875,"本":1876,"李":1877,"村":1878,"東":1879,"松":1880,"林":1881,"森":1882,"楊":1883,"樹":1884,"橋":1885,"歌":1886,"止":1887,"正":1888,"武":1889,"比":1890,"氏":1891,"民":1892,"水":1893,"氵":1894,"氷":1895,"永":1896,"江":1897,"沢":1898,"河":1899,"治":1900,"法":1901,"海":1902,"清":1903,"漢":1904,"瀬":1905,"火":1906,"版":1907,"犬":1908,"王":1909,"生":1910,"田":1911,"男":1912,"疒":1913,"発":1914,"白":1915,"的":1916,"皇":1917,"目":1918,"相":1919,"省":1920,"真":1921,"石":1922,"示":1923,"社":1924,"神":1925,"福":1926,"禾":1927,"秀":1928,"秋":1929,"空":1930,"立":1931,"章":1932,"竹":1933,"糹":1934,"美":1935,"義":1936,"耳":1937,"良":1938,"艹":1939,"花":1940,"英":1941,"華":1942,"葉":1943,"藤":1944,"行":1945,"街":1946,"西":1947,"見":1948,"訁":1949,"語":1950,"谷":1951,"貝":1952,"貴":1953,"車":1954,"軍":1955,"辶":1956,"道":1957,"郎":1958,"郡":1959,"部":1960,"都":1961,"里":1962,"野":1963,"金":1964,"鈴":1965,"镇":1966,"長":1967,"門":1968,"間":1969,"阝":1970,"阿":1971,"陳":1972,"陽":1973,"雄":1974,"青":1975,"面":1976,"風":1977,"食":1978,"香":1979,"馬":1980,"高":1981,"龍":1982,"龸":1983,"fi":1984,"fl":1985,"!":1986,"(":1987,")":1988,",":1989,"-":1990,".":1991,"/":1992,":":1993,"?":1994,"~":1995,"the":1996,"of":1997,"and":1998,"in":1999,"to":2000,"was":2001,"he":2002,"is":2003,"as":2004,"for":2005,"on":2006,"with":2007,"that":2008,"it":2009,"his":2010,"by":2011,"at":2012,"from":2013,"her":2014,"##s":2015,"she":2016,"you":2017,"had":2018,"an":2019,"were":2020,"but":2021,"be":2022,"this":2023,"are":2024,"not":2025,"my":2026,"they":2027,"one":2028,"which":2029,"or":2030,"have":2031,"him":2032,"me":2033,"first":2034,"all":2035,"also":2036,"their":2037,"has":2038,"up":2039,"who":2040,"out":2041,"been":2042,"when":2043,"after":2044,"there":2045,"into":2046,"new":2047,"two":2048,"its":2049,"##a":2050,"time":2051,"would":2052,"no":2053,"what":2054,"about":2055,"said":2056,"we":2057,"over":2058,"then":2059,"other":2060,"so":2061,"more":2062,"##e":2063,"can":2064,"if":2065,"like":2066,"back":2067,"them":2068,"only":2069,"some":2070,"could":2071,"##i":2072,"where":2073,"just":2074,"##ing":2075,"during":2076,"before":2077,"##n":2078,"do":2079,"##o":2080,"made":2081,"school":2082,"through":2083,"than":2084,"now":2085,"years":2086,"most":2087,"world":2088,"may":2089,"between":2090,"down":2091,"well":2092,"three":2093,"##d":2094,"year":2095,"while":2096,"will":2097,"##ed":2098,"##r":2099,"##y":2100,"later":2101,"##t":2102,"city":2103,"under":2104,"around":2105,"did":2106,"such":2107,"being":2108,"used":2109,"state":2110,"people":2111,"part":2112,"know":2113,"against":2114,"your":2115,"many":2116,"second":2117,"university":2118,"both":2119,"national":2120,"##er":2121,"these":2122,"don":2123,"known":2124,"off":2125,"way":2126,"until":2127,"re":2128,"how":2129,"even":2130,"get":2131,"head":2132,"...":2133,"didn":2134,"##ly":2135,"team":2136,"american":2137,"because":2138,"de":2139,"##l":2140,"born":2141,"united":2142,"film":2143,"since":2144,"still":2145,"long":2146,"work":2147,"south":2148,"us":2149,"became":2150,"any":2151,"high":2152,"again":2153,"day":2154,"family":2155,"see":2156,"right":2157,"man":2158,"eyes":2159,"house":2160,"season":2161,"war":2162,"states":2163,"including":2164,"took":2165,"life":2166,"north":2167,"same":2168,"each":2169,"called":2170,"name":2171,"much":2172,"place":2173,"however":2174,"go":2175,"four":2176,"group":2177,"another":2178,"found":2179,"won":2180,"area":2181,"here":2182,"going":2183,"10":2184,"away":2185,"series":2186,"left":2187,"home":2188,"music":2189,"best":2190,"make":2191,"hand":2192,"number":2193,"company":2194,"several":2195,"never":2196,"last":2197,"john":2198,"000":2199,"very":2200,"album":2201,"take":2202,"end":2203,"good":2204,"too":2205,"following":2206,"released":2207,"game":2208,"played":2209,"little":2210,"began":2211,"district":2212,"##m":2213,"old":2214,"want":2215,"those":2216,"side":2217,"held":2218,"own":2219,"early":2220,"county":2221,"ll":2222,"league":2223,"use":2224,"west":2225,"##u":2226,"face":2227,"think":2228,"##es":2229,"2010":2230,"government":2231,"##h":2232,"march":2233,"came":2234,"small":2235,"general":2236,"town":2237,"june":2238,"##on":2239,"line":2240,"based":2241,"something":2242,"##k":2243,"september":2244,"thought":2245,"looked":2246,"along":2247,"international":2248,"2011":2249,"air":2250,"july":2251,"club":2252,"went":2253,"january":2254,"october":2255,"our":2256,"august":2257,"april":2258,"york":2259,"12":2260,"few":2261,"2012":2262,"2008":2263,"east":2264,"show":2265,"member":2266,"college":2267,"2009":2268,"father":2269,"public":2270,"##us":2271,"come":2272,"men":2273,"five":2274,"set":2275,"station":2276,"church":2277,"##c":2278,"next":2279,"former":2280,"november":2281,"room":2282,"party":2283,"located":2284,"december":2285,"2013":2286,"age":2287,"got":2288,"2007":2289,"##g":2290,"system":2291,"let":2292,"love":2293,"2006":2294,"though":2295,"every":2296,"2014":2297,"look":2298,"song":2299,"water":2300,"century":2301,"without":2302,"body":2303,"black":2304,"night":2305,"within":2306,"great":2307,"women":2308,"single":2309,"ve":2310,"building":2311,"large":2312,"population":2313,"river":2314,"named":2315,"band":2316,"white":2317,"started":2318,"##an":2319,"once":2320,"15":2321,"20":2322,"should":2323,"18":2324,"2015":2325,"service":2326,"top":2327,"built":2328,"british":2329,"open":2330,"death":2331,"king":2332,"moved":2333,"local":2334,"times":2335,"children":2336,"february":2337,"book":2338,"why":2339,"11":2340,"door":2341,"need":2342,"president":2343,"order":2344,"final":2345,"road":2346,"wasn":2347,"although":2348,"due":2349,"major":2350,"died":2351,"village":2352,"third":2353,"knew":2354,"2016":2355,"asked":2356,"turned":2357,"st":2358,"wanted":2359,"say":2360,"##p":2361,"together":2362,"received":2363,"main":2364,"son":2365,"served":2366,"different":2367,"##en":2368,"behind":2369,"himself":2370,"felt":2371,"members":2372,"power":2373,"football":2374,"law":2375,"voice":2376,"play":2377,"##in":2378,"near":2379,"park":2380,"history":2381,"30":2382,"having":2383,"2005":2384,"16":2385,"##man":2386,"saw":2387,"mother":2388,"##al":2389,"army":2390,"point":2391,"front":2392,"help":2393,"english":2394,"street":2395,"art":2396,"late":2397,"hands":2398,"games":2399,"award":2400,"##ia":2401,"young":2402,"14":2403,"put":2404,"published":2405,"country":2406,"division":2407,"across":2408,"told":2409,"13":2410,"often":2411,"ever":2412,"french":2413,"london":2414,"center":2415,"six":2416,"red":2417,"2017":2418,"led":2419,"days":2420,"include":2421,"light":2422,"25":2423,"find":2424,"tell":2425,"among":2426,"species":2427,"really":2428,"according":2429,"central":2430,"half":2431,"2004":2432,"form":2433,"original":2434,"gave":2435,"office":2436,"making":2437,"enough":2438,"lost":2439,"full":2440,"opened":2441,"must":2442,"included":2443,"live":2444,"given":2445,"german":2446,"player":2447,"run":2448,"business":2449,"woman":2450,"community":2451,"cup":2452,"might":2453,"million":2454,"land":2455,"2000":2456,"court":2457,"development":2458,"17":2459,"short":2460,"round":2461,"ii":2462,"km":2463,"seen":2464,"class":2465,"story":2466,"always":2467,"become":2468,"sure":2469,"research":2470,"almost":2471,"director":2472,"council":2473,"la":2474,"##2":2475,"career":2476,"things":2477,"using":2478,"island":2479,"##z":2480,"couldn":2481,"car":2482,"##is":2483,"24":2484,"close":2485,"force":2486,"##1":2487,"better":2488,"free":2489,"support":2490,"control":2491,"field":2492,"students":2493,"2003":2494,"education":2495,"married":2496,"##b":2497,"nothing":2498,"worked":2499,"others":2500,"record":2501,"big":2502,"inside":2503,"level":2504,"anything":2505,"continued":2506,"give":2507,"james":2508,"##3":2509,"military":2510,"established":2511,"non":2512,"returned":2513,"feel":2514,"does":2515,"title":2516,"written":2517,"thing":2518,"feet":2519,"william":2520,"far":2521,"co":2522,"association":2523,"hard":2524,"already":2525,"2002":2526,"##ra":2527,"championship":2528,"human":2529,"western":2530,"100":2531,"##na":2532,"department":2533,"hall":2534,"role":2535,"various":2536,"production":2537,"21":2538,"19":2539,"heart":2540,"2001":2541,"living":2542,"fire":2543,"version":2544,"##ers":2545,"##f":2546,"television":2547,"royal":2548,"##4":2549,"produced":2550,"working":2551,"act":2552,"case":2553,"society":2554,"region":2555,"present":2556,"radio":2557,"period":2558,"looking":2559,"least":2560,"total":2561,"keep":2562,"england":2563,"wife":2564,"program":2565,"per":2566,"brother":2567,"mind":2568,"special":2569,"22":2570,"##le":2571,"am":2572,"works":2573,"soon":2574,"##6":2575,"political":2576,"george":2577,"services":2578,"taken":2579,"created":2580,"##7":2581,"further":2582,"able":2583,"reached":2584,"david":2585,"union":2586,"joined":2587,"upon":2588,"done":2589,"important":2590,"social":2591,"information":2592,"either":2593,"##ic":2594,"##x":2595,"appeared":2596,"position":2597,"ground":2598,"lead":2599,"rock":2600,"dark":2601,"election":2602,"23":2603,"board":2604,"france":2605,"hair":2606,"course":2607,"arms":2608,"site":2609,"police":2610,"girl":2611,"instead":2612,"real":2613,"sound":2614,"##v":2615,"words":2616,"moment":2617,"##te":2618,"someone":2619,"##8":2620,"summer":2621,"project":2622,"announced":2623,"san":2624,"less":2625,"wrote":2626,"past":2627,"followed":2628,"##5":2629,"blue":2630,"founded":2631,"al":2632,"finally":2633,"india":2634,"taking":2635,"records":2636,"america":2637,"##ne":2638,"1999":2639,"design":2640,"considered":2641,"northern":2642,"god":2643,"stop":2644,"battle":2645,"toward":2646,"european":2647,"outside":2648,"described":2649,"track":2650,"today":2651,"playing":2652,"language":2653,"28":2654,"call":2655,"26":2656,"heard":2657,"professional":2658,"low":2659,"australia":2660,"miles":2661,"california":2662,"win":2663,"yet":2664,"green":2665,"##ie":2666,"trying":2667,"blood":2668,"##ton":2669,"southern":2670,"science":2671,"maybe":2672,"everything":2673,"match":2674,"square":2675,"27":2676,"mouth":2677,"video":2678,"race":2679,"recorded":2680,"leave":2681,"above":2682,"##9":2683,"daughter":2684,"points":2685,"space":2686,"1998":2687,"museum":2688,"change":2689,"middle":2690,"common":2691,"##0":2692,"move":2693,"tv":2694,"post":2695,"##ta":2696,"lake":2697,"seven":2698,"tried":2699,"elected":2700,"closed":2701,"ten":2702,"paul":2703,"minister":2704,"##th":2705,"months":2706,"start":2707,"chief":2708,"return":2709,"canada":2710,"person":2711,"sea":2712,"release":2713,"similar":2714,"modern":2715,"brought":2716,"rest":2717,"hit":2718,"formed":2719,"mr":2720,"##la":2721,"1997":2722,"floor":2723,"event":2724,"doing":2725,"thomas":2726,"1996":2727,"robert":2728,"care":2729,"killed":2730,"training":2731,"star":2732,"week":2733,"needed":2734,"turn":2735,"finished":2736,"railway":2737,"rather":2738,"news":2739,"health":2740,"sent":2741,"example":2742,"ran":2743,"term":2744,"michael":2745,"coming":2746,"currently":2747,"yes":2748,"forces":2749,"despite":2750,"gold":2751,"areas":2752,"50":2753,"stage":2754,"fact":2755,"29":2756,"dead":2757,"says":2758,"popular":2759,"2018":2760,"originally":2761,"germany":2762,"probably":2763,"developed":2764,"result":2765,"pulled":2766,"friend":2767,"stood":2768,"money":2769,"running":2770,"mi":2771,"signed":2772,"word":2773,"songs":2774,"child":2775,"eventually":2776,"met":2777,"tour":2778,"average":2779,"teams":2780,"minutes":2781,"festival":2782,"current":2783,"deep":2784,"kind":2785,"1995":2786,"decided":2787,"usually":2788,"eastern":2789,"seemed":2790,"##ness":2791,"episode":2792,"bed":2793,"added":2794,"table":2795,"indian":2796,"private":2797,"charles":2798,"route":2799,"available":2800,"idea":2801,"throughout":2802,"centre":2803,"addition":2804,"appointed":2805,"style":2806,"1994":2807,"books":2808,"eight":2809,"construction":2810,"press":2811,"mean":2812,"wall":2813,"friends":2814,"remained":2815,"schools":2816,"study":2817,"##ch":2818,"##um":2819,"institute":2820,"oh":2821,"chinese":2822,"sometimes":2823,"events":2824,"possible":2825,"1992":2826,"australian":2827,"type":2828,"brown":2829,"forward":2830,"talk":2831,"process":2832,"food":2833,"debut":2834,"seat":2835,"performance":2836,"committee":2837,"features":2838,"character":2839,"arts":2840,"herself":2841,"else":2842,"lot":2843,"strong":2844,"russian":2845,"range":2846,"hours":2847,"peter":2848,"arm":2849,"##da":2850,"morning":2851,"dr":2852,"sold":2853,"##ry":2854,"quickly":2855,"directed":2856,"1993":2857,"guitar":2858,"china":2859,"##w":2860,"31":2861,"list":2862,"##ma":2863,"performed":2864,"media":2865,"uk":2866,"players":2867,"smile":2868,"##rs":2869,"myself":2870,"40":2871,"placed":2872,"coach":2873,"province":2874,"towards":2875,"wouldn":2876,"leading":2877,"whole":2878,"boy":2879,"official":2880,"designed":2881,"grand":2882,"census":2883,"##el":2884,"europe":2885,"attack":2886,"japanese":2887,"henry":2888,"1991":2889,"##re":2890,"##os":2891,"cross":2892,"getting":2893,"alone":2894,"action":2895,"lower":2896,"network":2897,"wide":2898,"washington":2899,"japan":2900,"1990":2901,"hospital":2902,"believe":2903,"changed":2904,"sister":2905,"##ar":2906,"hold":2907,"gone":2908,"sir":2909,"hadn":2910,"ship":2911,"##ka":2912,"studies":2913,"academy":2914,"shot":2915,"rights":2916,"below":2917,"base":2918,"bad":2919,"involved":2920,"kept":2921,"largest":2922,"##ist":2923,"bank":2924,"future":2925,"especially":2926,"beginning":2927,"mark":2928,"movement":2929,"section":2930,"female":2931,"magazine":2932,"plan":2933,"professor":2934,"lord":2935,"longer":2936,"##ian":2937,"sat":2938,"walked":2939,"hill":2940,"actually":2941,"civil":2942,"energy":2943,"model":2944,"families":2945,"size":2946,"thus":2947,"aircraft":2948,"completed":2949,"includes":2950,"data":2951,"captain":2952,"##or":2953,"fight":2954,"vocals":2955,"featured":2956,"richard":2957,"bridge":2958,"fourth":2959,"1989":2960,"officer":2961,"stone":2962,"hear":2963,"##ism":2964,"means":2965,"medical":2966,"groups":2967,"management":2968,"self":2969,"lips":2970,"competition":2971,"entire":2972,"lived":2973,"technology":2974,"leaving":2975,"federal":2976,"tournament":2977,"bit":2978,"passed":2979,"hot":2980,"independent":2981,"awards":2982,"kingdom":2983,"mary":2984,"spent":2985,"fine":2986,"doesn":2987,"reported":2988,"##ling":2989,"jack":2990,"fall":2991,"raised":2992,"itself":2993,"stay":2994,"true":2995,"studio":2996,"1988":2997,"sports":2998,"replaced":2999,"paris":3000,"systems":3001,"saint":3002,"leader":3003,"theatre":3004,"whose":3005,"market":3006,"capital":3007,"parents":3008,"spanish":3009,"canadian":3010,"earth":3011,"##ity":3012,"cut":3013,"degree":3014,"writing":3015,"bay":3016,"christian":3017,"awarded":3018,"natural":3019,"higher":3020,"bill":3021,"##as":3022,"coast":3023,"provided":3024,"previous":3025,"senior":3026,"ft":3027,"valley":3028,"organization":3029,"stopped":3030,"onto":3031,"countries":3032,"parts":3033,"conference":3034,"queen":3035,"security":3036,"interest":3037,"saying":3038,"allowed":3039,"master":3040,"earlier":3041,"phone":3042,"matter":3043,"smith":3044,"winning":3045,"try":3046,"happened":3047,"moving":3048,"campaign":3049,"los":3050,"##ley":3051,"breath":3052,"nearly":3053,"mid":3054,"1987":3055,"certain":3056,"girls":3057,"date":3058,"italian":3059,"african":3060,"standing":3061,"fell":3062,"artist":3063,"##ted":3064,"shows":3065,"deal":3066,"mine":3067,"industry":3068,"1986":3069,"##ng":3070,"everyone":3071,"republic":3072,"provide":3073,"collection":3074,"library":3075,"student":3076,"##ville":3077,"primary":3078,"owned":3079,"older":3080,"via":3081,"heavy":3082,"1st":3083,"makes":3084,"##able":3085,"attention":3086,"anyone":3087,"africa":3088,"##ri":3089,"stated":3090,"length":3091,"ended":3092,"fingers":3093,"command":3094,"staff":3095,"skin":3096,"foreign":3097,"opening":3098,"governor":3099,"okay":3100,"medal":3101,"kill":3102,"sun":3103,"cover":3104,"job":3105,"1985":3106,"introduced":3107,"chest":3108,"hell":3109,"feeling":3110,"##ies":3111,"success":3112,"meet":3113,"reason":3114,"standard":3115,"meeting":3116,"novel":3117,"1984":3118,"trade":3119,"source":3120,"buildings":3121,"##land":3122,"rose":3123,"guy":3124,"goal":3125,"##ur":3126,"chapter":3127,"native":3128,"husband":3129,"previously":3130,"unit":3131,"limited":3132,"entered":3133,"weeks":3134,"producer":3135,"operations":3136,"mountain":3137,"takes":3138,"covered":3139,"forced":3140,"related":3141,"roman":3142,"complete":3143,"successful":3144,"key":3145,"texas":3146,"cold":3147,"##ya":3148,"channel":3149,"1980":3150,"traditional":3151,"films":3152,"dance":3153,"clear":3154,"approximately":3155,"500":3156,"nine":3157,"van":3158,"prince":3159,"question":3160,"active":3161,"tracks":3162,"ireland":3163,"regional":3164,"silver":3165,"author":3166,"personal":3167,"sense":3168,"operation":3169,"##ine":3170,"economic":3171,"1983":3172,"holding":3173,"twenty":3174,"isbn":3175,"additional":3176,"speed":3177,"hour":3178,"edition":3179,"regular":3180,"historic":3181,"places":3182,"whom":3183,"shook":3184,"movie":3185,"km²":3186,"secretary":3187,"prior":3188,"report":3189,"chicago":3190,"read":3191,"foundation":3192,"view":3193,"engine":3194,"scored":3195,"1982":3196,"units":3197,"ask":3198,"airport":3199,"property":3200,"ready":3201,"immediately":3202,"lady":3203,"month":3204,"listed":3205,"contract":3206,"##de":3207,"manager":3208,"themselves":3209,"lines":3210,"##ki":3211,"navy":3212,"writer":3213,"meant":3214,"##ts":3215,"runs":3216,"##ro":3217,"practice":3218,"championships":3219,"singer":3220,"glass":3221,"commission":3222,"required":3223,"forest":3224,"starting":3225,"culture":3226,"generally":3227,"giving":3228,"access":3229,"attended":3230,"test":3231,"couple":3232,"stand":3233,"catholic":3234,"martin":3235,"caught":3236,"executive":3237,"##less":3238,"eye":3239,"##ey":3240,"thinking":3241,"chair":3242,"quite":3243,"shoulder":3244,"1979":3245,"hope":3246,"decision":3247,"plays":3248,"defeated":3249,"municipality":3250,"whether":3251,"structure":3252,"offered":3253,"slowly":3254,"pain":3255,"ice":3256,"direction":3257,"##ion":3258,"paper":3259,"mission":3260,"1981":3261,"mostly":3262,"200":3263,"noted":3264,"individual":3265,"managed":3266,"nature":3267,"lives":3268,"plant":3269,"##ha":3270,"helped":3271,"except":3272,"studied":3273,"computer":3274,"figure":3275,"relationship":3276,"issue":3277,"significant":3278,"loss":3279,"die":3280,"smiled":3281,"gun":3282,"ago":3283,"highest":3284,"1972":3285,"##am":3286,"male":3287,"bring":3288,"goals":3289,"mexico":3290,"problem":3291,"distance":3292,"commercial":3293,"completely":3294,"location":3295,"annual":3296,"famous":3297,"drive":3298,"1976":3299,"neck":3300,"1978":3301,"surface":3302,"caused":3303,"italy":3304,"understand":3305,"greek":3306,"highway":3307,"wrong":3308,"hotel":3309,"comes":3310,"appearance":3311,"joseph":3312,"double":3313,"issues":3314,"musical":3315,"companies":3316,"castle":3317,"income":3318,"review":3319,"assembly":3320,"bass":3321,"initially":3322,"parliament":3323,"artists":3324,"experience":3325,"1974":3326,"particular":3327,"walk":3328,"foot":3329,"engineering":3330,"talking":3331,"window":3332,"dropped":3333,"##ter":3334,"miss":3335,"baby":3336,"boys":3337,"break":3338,"1975":3339,"stars":3340,"edge":3341,"remember":3342,"policy":3343,"carried":3344,"train":3345,"stadium":3346,"bar":3347,"sex":3348,"angeles":3349,"evidence":3350,"##ge":3351,"becoming":3352,"assistant":3353,"soviet":3354,"1977":3355,"upper":3356,"step":3357,"wing":3358,"1970":3359,"youth":3360,"financial":3361,"reach":3362,"##ll":3363,"actor":3364,"numerous":3365,"##se":3366,"##st":3367,"nodded":3368,"arrived":3369,"##ation":3370,"minute":3371,"##nt":3372,"believed":3373,"sorry":3374,"complex":3375,"beautiful":3376,"victory":3377,"associated":3378,"temple":3379,"1968":3380,"1973":3381,"chance":3382,"perhaps":3383,"metal":3384,"##son":3385,"1945":3386,"bishop":3387,"##et":3388,"lee":3389,"launched":3390,"particularly":3391,"tree":3392,"le":3393,"retired":3394,"subject":3395,"prize":3396,"contains":3397,"yeah":3398,"theory":3399,"empire":3400,"##ce":3401,"suddenly":3402,"waiting":3403,"trust":3404,"recording":3405,"##to":3406,"happy":3407,"terms":3408,"camp":3409,"champion":3410,"1971":3411,"religious":3412,"pass":3413,"zealand":3414,"names":3415,"2nd":3416,"port":3417,"ancient":3418,"tom":3419,"corner":3420,"represented":3421,"watch":3422,"legal":3423,"anti":3424,"justice":3425,"cause":3426,"watched":3427,"brothers":3428,"45":3429,"material":3430,"changes":3431,"simply":3432,"response":3433,"louis":3434,"fast":3435,"##ting":3436,"answer":3437,"60":3438,"historical":3439,"1969":3440,"stories":3441,"straight":3442,"create":3443,"feature":3444,"increased":3445,"rate":3446,"administration":3447,"virginia":3448,"el":3449,"activities":3450,"cultural":3451,"overall":3452,"winner":3453,"programs":3454,"basketball":3455,"legs":3456,"guard":3457,"beyond":3458,"cast":3459,"doctor":3460,"mm":3461,"flight":3462,"results":3463,"remains":3464,"cost":3465,"effect":3466,"winter":3467,"##ble":3468,"larger":3469,"islands":3470,"problems":3471,"chairman":3472,"grew":3473,"commander":3474,"isn":3475,"1967":3476,"pay":3477,"failed":3478,"selected":3479,"hurt":3480,"fort":3481,"box":3482,"regiment":3483,"majority":3484,"journal":3485,"35":3486,"edward":3487,"plans":3488,"##ke":3489,"##ni":3490,"shown":3491,"pretty":3492,"irish":3493,"characters":3494,"directly":3495,"scene":3496,"likely":3497,"operated":3498,"allow":3499,"spring":3500,"##j":3501,"junior":3502,"matches":3503,"looks":3504,"mike":3505,"houses":3506,"fellow":3507,"##tion":3508,"beach":3509,"marriage":3510,"##ham":3511,"##ive":3512,"rules":3513,"oil":3514,"65":3515,"florida":3516,"expected":3517,"nearby":3518,"congress":3519,"sam":3520,"peace":3521,"recent":3522,"iii":3523,"wait":3524,"subsequently":3525,"cell":3526,"##do":3527,"variety":3528,"serving":3529,"agreed":3530,"please":3531,"poor":3532,"joe":3533,"pacific":3534,"attempt":3535,"wood":3536,"democratic":3537,"piece":3538,"prime":3539,"##ca":3540,"rural":3541,"mile":3542,"touch":3543,"appears":3544,"township":3545,"1964":3546,"1966":3547,"soldiers":3548,"##men":3549,"##ized":3550,"1965":3551,"pennsylvania":3552,"closer":3553,"fighting":3554,"claimed":3555,"score":3556,"jones":3557,"physical":3558,"editor":3559,"##ous":3560,"filled":3561,"genus":3562,"specific":3563,"sitting":3564,"super":3565,"mom":3566,"##va":3567,"therefore":3568,"supported":3569,"status":3570,"fear":3571,"cases":3572,"store":3573,"meaning":3574,"wales":3575,"minor":3576,"spain":3577,"tower":3578,"focus":3579,"vice":3580,"frank":3581,"follow":3582,"parish":3583,"separate":3584,"golden":3585,"horse":3586,"fifth":3587,"remaining":3588,"branch":3589,"32":3590,"presented":3591,"stared":3592,"##id":3593,"uses":3594,"secret":3595,"forms":3596,"##co":3597,"baseball":3598,"exactly":3599,"##ck":3600,"choice":3601,"note":3602,"discovered":3603,"travel":3604,"composed":3605,"truth":3606,"russia":3607,"ball":3608,"color":3609,"kiss":3610,"dad":3611,"wind":3612,"continue":3613,"ring":3614,"referred":3615,"numbers":3616,"digital":3617,"greater":3618,"##ns":3619,"metres":3620,"slightly":3621,"direct":3622,"increase":3623,"1960":3624,"responsible":3625,"crew":3626,"rule":3627,"trees":3628,"troops":3629,"##no":3630,"broke":3631,"goes":3632,"individuals":3633,"hundred":3634,"weight":3635,"creek":3636,"sleep":3637,"memory":3638,"defense":3639,"provides":3640,"ordered":3641,"code":3642,"value":3643,"jewish":3644,"windows":3645,"1944":3646,"safe":3647,"judge":3648,"whatever":3649,"corps":3650,"realized":3651,"growing":3652,"pre":3653,"##ga":3654,"cities":3655,"alexander":3656,"gaze":3657,"lies":3658,"spread":3659,"scott":3660,"letter":3661,"showed":3662,"situation":3663,"mayor":3664,"transport":3665,"watching":3666,"workers":3667,"extended":3668,"##li":3669,"expression":3670,"normal":3671,"##ment":3672,"chart":3673,"multiple":3674,"border":3675,"##ba":3676,"host":3677,"##ner":3678,"daily":3679,"mrs":3680,"walls":3681,"piano":3682,"##ko":3683,"heat":3684,"cannot":3685,"##ate":3686,"earned":3687,"products":3688,"drama":3689,"era":3690,"authority":3691,"seasons":3692,"join":3693,"grade":3694,"##io":3695,"sign":3696,"difficult":3697,"machine":3698,"1963":3699,"territory":3700,"mainly":3701,"##wood":3702,"stations":3703,"squadron":3704,"1962":3705,"stepped":3706,"iron":3707,"19th":3708,"##led":3709,"serve":3710,"appear":3711,"sky":3712,"speak":3713,"broken":3714,"charge":3715,"knowledge":3716,"kilometres":3717,"removed":3718,"ships":3719,"article":3720,"campus":3721,"simple":3722,"##ty":3723,"pushed":3724,"britain":3725,"##ve":3726,"leaves":3727,"recently":3728,"cd":3729,"soft":3730,"boston":3731,"latter":3732,"easy":3733,"acquired":3734,"poland":3735,"##sa":3736,"quality":3737,"officers":3738,"presence":3739,"planned":3740,"nations":3741,"mass":3742,"broadcast":3743,"jean":3744,"share":3745,"image":3746,"influence":3747,"wild":3748,"offer":3749,"emperor":3750,"electric":3751,"reading":3752,"headed":3753,"ability":3754,"promoted":3755,"yellow":3756,"ministry":3757,"1942":3758,"throat":3759,"smaller":3760,"politician":3761,"##by":3762,"latin":3763,"spoke":3764,"cars":3765,"williams":3766,"males":3767,"lack":3768,"pop":3769,"80":3770,"##ier":3771,"acting":3772,"seeing":3773,"consists":3774,"##ti":3775,"estate":3776,"1961":3777,"pressure":3778,"johnson":3779,"newspaper":3780,"jr":3781,"chris":3782,"olympics":3783,"online":3784,"conditions":3785,"beat":3786,"elements":3787,"walking":3788,"vote":3789,"##field":3790,"needs":3791,"carolina":3792,"text":3793,"featuring":3794,"global":3795,"block":3796,"shirt":3797,"levels":3798,"francisco":3799,"purpose":3800,"females":3801,"et":3802,"dutch":3803,"duke":3804,"ahead":3805,"gas":3806,"twice":3807,"safety":3808,"serious":3809,"turning":3810,"highly":3811,"lieutenant":3812,"firm":3813,"maria":3814,"amount":3815,"mixed":3816,"daniel":3817,"proposed":3818,"perfect":3819,"agreement":3820,"affairs":3821,"3rd":3822,"seconds":3823,"contemporary":3824,"paid":3825,"1943":3826,"prison":3827,"save":3828,"kitchen":3829,"label":3830,"administrative":3831,"intended":3832,"constructed":3833,"academic":3834,"nice":3835,"teacher":3836,"races":3837,"1956":3838,"formerly":3839,"corporation":3840,"ben":3841,"nation":3842,"issued":3843,"shut":3844,"1958":3845,"drums":3846,"housing":3847,"victoria":3848,"seems":3849,"opera":3850,"1959":3851,"graduated":3852,"function":3853,"von":3854,"mentioned":3855,"picked":3856,"build":3857,"recognized":3858,"shortly":3859,"protection":3860,"picture":3861,"notable":3862,"exchange":3863,"elections":3864,"1980s":3865,"loved":3866,"percent":3867,"racing":3868,"fish":3869,"elizabeth":3870,"garden":3871,"volume":3872,"hockey":3873,"1941":3874,"beside":3875,"settled":3876,"##ford":3877,"1940":3878,"competed":3879,"replied":3880,"drew":3881,"1948":3882,"actress":3883,"marine":3884,"scotland":3885,"steel":3886,"glanced":3887,"farm":3888,"steve":3889,"1957":3890,"risk":3891,"tonight":3892,"positive":3893,"magic":3894,"singles":3895,"effects":3896,"gray":3897,"screen":3898,"dog":3899,"##ja":3900,"residents":3901,"bus":3902,"sides":3903,"none":3904,"secondary":3905,"literature":3906,"polish":3907,"destroyed":3908,"flying":3909,"founder":3910,"households":3911,"1939":3912,"lay":3913,"reserve":3914,"usa":3915,"gallery":3916,"##ler":3917,"1946":3918,"industrial":3919,"younger":3920,"approach":3921,"appearances":3922,"urban":3923,"ones":3924,"1950":3925,"finish":3926,"avenue":3927,"powerful":3928,"fully":3929,"growth":3930,"page":3931,"honor":3932,"jersey":3933,"projects":3934,"advanced":3935,"revealed":3936,"basic":3937,"90":3938,"infantry":3939,"pair":3940,"equipment":3941,"visit":3942,"33":3943,"evening":3944,"search":3945,"grant":3946,"effort":3947,"solo":3948,"treatment":3949,"buried":3950,"republican":3951,"primarily":3952,"bottom":3953,"owner":3954,"1970s":3955,"israel":3956,"gives":3957,"jim":3958,"dream":3959,"bob":3960,"remain":3961,"spot":3962,"70":3963,"notes":3964,"produce":3965,"champions":3966,"contact":3967,"ed":3968,"soul":3969,"accepted":3970,"ways":3971,"del":3972,"##ally":3973,"losing":3974,"split":3975,"price":3976,"capacity":3977,"basis":3978,"trial":3979,"questions":3980,"##ina":3981,"1955":3982,"20th":3983,"guess":3984,"officially":3985,"memorial":3986,"naval":3987,"initial":3988,"##ization":3989,"whispered":3990,"median":3991,"engineer":3992,"##ful":3993,"sydney":3994,"##go":3995,"columbia":3996,"strength":3997,"300":3998,"1952":3999,"tears":4000,"senate":4001,"00":4002,"card":4003,"asian":4004,"agent":4005,"1947":4006,"software":4007,"44":4008,"draw":4009,"warm":4010,"supposed":4011,"com":4012,"pro":4013,"##il":4014,"transferred":4015,"leaned":4016,"##at":4017,"candidate":4018,"escape":4019,"mountains":4020,"asia":4021,"potential":4022,"activity":4023,"entertainment":4024,"seem":4025,"traffic":4026,"jackson":4027,"murder":4028,"36":4029,"slow":4030,"product":4031,"orchestra":4032,"haven":4033,"agency":4034,"bbc":4035,"taught":4036,"website":4037,"comedy":4038,"unable":4039,"storm":4040,"planning":4041,"albums":4042,"rugby":4043,"environment":4044,"scientific":4045,"grabbed":4046,"protect":4047,"##hi":4048,"boat":4049,"typically":4050,"1954":4051,"1953":4052,"damage":4053,"principal":4054,"divided":4055,"dedicated":4056,"mount":4057,"ohio":4058,"##berg":4059,"pick":4060,"fought":4061,"driver":4062,"##der":4063,"empty":4064,"shoulders":4065,"sort":4066,"thank":4067,"berlin":4068,"prominent":4069,"account":4070,"freedom":4071,"necessary":4072,"efforts":4073,"alex":4074,"headquarters":4075,"follows":4076,"alongside":4077,"des":4078,"simon":4079,"andrew":4080,"suggested":4081,"operating":4082,"learning":4083,"steps":4084,"1949":4085,"sweet":4086,"technical":4087,"begin":4088,"easily":4089,"34":4090,"teeth":4091,"speaking":4092,"settlement":4093,"scale":4094,"##sh":4095,"renamed":4096,"ray":4097,"max":4098,"enemy":4099,"semi":4100,"joint":4101,"compared":4102,"##rd":4103,"scottish":4104,"leadership":4105,"analysis":4106,"offers":4107,"georgia":4108,"pieces":4109,"captured":4110,"animal":4111,"deputy":4112,"guest":4113,"organized":4114,"##lin":4115,"tony":4116,"combined":4117,"method":4118,"challenge":4119,"1960s":4120,"huge":4121,"wants":4122,"battalion":4123,"sons":4124,"rise":4125,"crime":4126,"types":4127,"facilities":4128,"telling":4129,"path":4130,"1951":4131,"platform":4132,"sit":4133,"1990s":4134,"##lo":4135,"tells":4136,"assigned":4137,"rich":4138,"pull":4139,"##ot":4140,"commonly":4141,"alive":4142,"##za":4143,"letters":4144,"concept":4145,"conducted":4146,"wearing":4147,"happen":4148,"bought":4149,"becomes":4150,"holy":4151,"gets":4152,"ocean":4153,"defeat":4154,"languages":4155,"purchased":4156,"coffee":4157,"occurred":4158,"titled":4159,"##q":4160,"declared":4161,"applied":4162,"sciences":4163,"concert":4164,"sounds":4165,"jazz":4166,"brain":4167,"##me":4168,"painting":4169,"fleet":4170,"tax":4171,"nick":4172,"##ius":4173,"michigan":4174,"count":4175,"animals":4176,"leaders":4177,"episodes":4178,"##line":4179,"content":4180,"##den":4181,"birth":4182,"##it":4183,"clubs":4184,"64":4185,"palace":4186,"critical":4187,"refused":4188,"fair":4189,"leg":4190,"laughed":4191,"returning":4192,"surrounding":4193,"participated":4194,"formation":4195,"lifted":4196,"pointed":4197,"connected":4198,"rome":4199,"medicine":4200,"laid":4201,"taylor":4202,"santa":4203,"powers":4204,"adam":4205,"tall":4206,"shared":4207,"focused":4208,"knowing":4209,"yards":4210,"entrance":4211,"falls":4212,"##wa":4213,"calling":4214,"##ad":4215,"sources":4216,"chosen":4217,"beneath":4218,"resources":4219,"yard":4220,"##ite":4221,"nominated":4222,"silence":4223,"zone":4224,"defined":4225,"##que":4226,"gained":4227,"thirty":4228,"38":4229,"bodies":4230,"moon":4231,"##ard":4232,"adopted":4233,"christmas":4234,"widely":4235,"register":4236,"apart":4237,"iran":4238,"premier":4239,"serves":4240,"du":4241,"unknown":4242,"parties":4243,"##les":4244,"generation":4245,"##ff":4246,"continues":4247,"quick":4248,"fields":4249,"brigade":4250,"quiet":4251,"teaching":4252,"clothes":4253,"impact":4254,"weapons":4255,"partner":4256,"flat":4257,"theater":4258,"supreme":4259,"1938":4260,"37":4261,"relations":4262,"##tor":4263,"plants":4264,"suffered":4265,"1936":4266,"wilson":4267,"kids":4268,"begins":4269,"##age":4270,"1918":4271,"seats":4272,"armed":4273,"internet":4274,"models":4275,"worth":4276,"laws":4277,"400":4278,"communities":4279,"classes":4280,"background":4281,"knows":4282,"thanks":4283,"quarter":4284,"reaching":4285,"humans":4286,"carry":4287,"killing":4288,"format":4289,"kong":4290,"hong":4291,"setting":4292,"75":4293,"architecture":4294,"disease":4295,"railroad":4296,"inc":4297,"possibly":4298,"wish":4299,"arthur":4300,"thoughts":4301,"harry":4302,"doors":4303,"density":4304,"##di":4305,"crowd":4306,"illinois":4307,"stomach":4308,"tone":4309,"unique":4310,"reports":4311,"anyway":4312,"##ir":4313,"liberal":4314,"der":4315,"vehicle":4316,"thick":4317,"dry":4318,"drug":4319,"faced":4320,"largely":4321,"facility":4322,"theme":4323,"holds":4324,"creation":4325,"strange":4326,"colonel":4327,"##mi":4328,"revolution":4329,"bell":4330,"politics":4331,"turns":4332,"silent":4333,"rail":4334,"relief":4335,"independence":4336,"combat":4337,"shape":4338,"write":4339,"determined":4340,"sales":4341,"learned":4342,"4th":4343,"finger":4344,"oxford":4345,"providing":4346,"1937":4347,"heritage":4348,"fiction":4349,"situated":4350,"designated":4351,"allowing":4352,"distribution":4353,"hosted":4354,"##est":4355,"sight":4356,"interview":4357,"estimated":4358,"reduced":4359,"##ria":4360,"toronto":4361,"footballer":4362,"keeping":4363,"guys":4364,"damn":4365,"claim":4366,"motion":4367,"sport":4368,"sixth":4369,"stayed":4370,"##ze":4371,"en":4372,"rear":4373,"receive":4374,"handed":4375,"twelve":4376,"dress":4377,"audience":4378,"granted":4379,"brazil":4380,"##well":4381,"spirit":4382,"##ated":4383,"noticed":4384,"etc":4385,"olympic":4386,"representative":4387,"eric":4388,"tight":4389,"trouble":4390,"reviews":4391,"drink":4392,"vampire":4393,"missing":4394,"roles":4395,"ranked":4396,"newly":4397,"household":4398,"finals":4399,"wave":4400,"critics":4401,"##ee":4402,"phase":4403,"massachusetts":4404,"pilot":4405,"unlike":4406,"philadelphia":4407,"bright":4408,"guns":4409,"crown":4410,"organizations":4411,"roof":4412,"42":4413,"respectively":4414,"clearly":4415,"tongue":4416,"marked":4417,"circle":4418,"fox":4419,"korea":4420,"bronze":4421,"brian":4422,"expanded":4423,"sexual":4424,"supply":4425,"yourself":4426,"inspired":4427,"labour":4428,"fc":4429,"##ah":4430,"reference":4431,"vision":4432,"draft":4433,"connection":4434,"brand":4435,"reasons":4436,"1935":4437,"classic":4438,"driving":4439,"trip":4440,"jesus":4441,"cells":4442,"entry":4443,"1920":4444,"neither":4445,"trail":4446,"claims":4447,"atlantic":4448,"orders":4449,"labor":4450,"nose":4451,"afraid":4452,"identified":4453,"intelligence":4454,"calls":4455,"cancer":4456,"attacked":4457,"passing":4458,"stephen":4459,"positions":4460,"imperial":4461,"grey":4462,"jason":4463,"39":4464,"sunday":4465,"48":4466,"swedish":4467,"avoid":4468,"extra":4469,"uncle":4470,"message":4471,"covers":4472,"allows":4473,"surprise":4474,"materials":4475,"fame":4476,"hunter":4477,"##ji":4478,"1930":4479,"citizens":4480,"figures":4481,"davis":4482,"environmental":4483,"confirmed":4484,"shit":4485,"titles":4486,"di":4487,"performing":4488,"difference":4489,"acts":4490,"attacks":4491,"##ov":4492,"existing":4493,"votes":4494,"opportunity":4495,"nor":4496,"shop":4497,"entirely":4498,"trains":4499,"opposite":4500,"pakistan":4501,"##pa":4502,"develop":4503,"resulted":4504,"representatives":4505,"actions":4506,"reality":4507,"pressed":4508,"##ish":4509,"barely":4510,"wine":4511,"conversation":4512,"faculty":4513,"northwest":4514,"ends":4515,"documentary":4516,"nuclear":4517,"stock":4518,"grace":4519,"sets":4520,"eat":4521,"alternative":4522,"##ps":4523,"bag":4524,"resulting":4525,"creating":4526,"surprised":4527,"cemetery":4528,"1919":4529,"drop":4530,"finding":4531,"sarah":4532,"cricket":4533,"streets":4534,"tradition":4535,"ride":4536,"1933":4537,"exhibition":4538,"target":4539,"ear":4540,"explained":4541,"rain":4542,"composer":4543,"injury":4544,"apartment":4545,"municipal":4546,"educational":4547,"occupied":4548,"netherlands":4549,"clean":4550,"billion":4551,"constitution":4552,"learn":4553,"1914":4554,"maximum":4555,"classical":4556,"francis":4557,"lose":4558,"opposition":4559,"jose":4560,"ontario":4561,"bear":4562,"core":4563,"hills":4564,"rolled":4565,"ending":4566,"drawn":4567,"permanent":4568,"fun":4569,"##tes":4570,"##lla":4571,"lewis":4572,"sites":4573,"chamber":4574,"ryan":4575,"##way":4576,"scoring":4577,"height":4578,"1934":4579,"##house":4580,"lyrics":4581,"staring":4582,"55":4583,"officials":4584,"1917":4585,"snow":4586,"oldest":4587,"##tic":4588,"orange":4589,"##ger":4590,"qualified":4591,"interior":4592,"apparently":4593,"succeeded":4594,"thousand":4595,"dinner":4596,"lights":4597,"existence":4598,"fans":4599,"heavily":4600,"41":4601,"greatest":4602,"conservative":4603,"send":4604,"bowl":4605,"plus":4606,"enter":4607,"catch":4608,"##un":4609,"economy":4610,"duty":4611,"1929":4612,"speech":4613,"authorities":4614,"princess":4615,"performances":4616,"versions":4617,"shall":4618,"graduate":4619,"pictures":4620,"effective":4621,"remembered":4622,"poetry":4623,"desk":4624,"crossed":4625,"starring":4626,"starts":4627,"passenger":4628,"sharp":4629,"##ant":4630,"acres":4631,"ass":4632,"weather":4633,"falling":4634,"rank":4635,"fund":4636,"supporting":4637,"check":4638,"adult":4639,"publishing":4640,"heads":4641,"cm":4642,"southeast":4643,"lane":4644,"##burg":4645,"application":4646,"bc":4647,"##ura":4648,"les":4649,"condition":4650,"transfer":4651,"prevent":4652,"display":4653,"ex":4654,"regions":4655,"earl":4656,"federation":4657,"cool":4658,"relatively":4659,"answered":4660,"besides":4661,"1928":4662,"obtained":4663,"portion":4664,"##town":4665,"mix":4666,"##ding":4667,"reaction":4668,"liked":4669,"dean":4670,"express":4671,"peak":4672,"1932":4673,"##tte":4674,"counter":4675,"religion":4676,"chain":4677,"rare":4678,"miller":4679,"convention":4680,"aid":4681,"lie":4682,"vehicles":4683,"mobile":4684,"perform":4685,"squad":4686,"wonder":4687,"lying":4688,"crazy":4689,"sword":4690,"##ping":4691,"attempted":4692,"centuries":4693,"weren":4694,"philosophy":4695,"category":4696,"##ize":4697,"anna":4698,"interested":4699,"47":4700,"sweden":4701,"wolf":4702,"frequently":4703,"abandoned":4704,"kg":4705,"literary":4706,"alliance":4707,"task":4708,"entitled":4709,"##ay":4710,"threw":4711,"promotion":4712,"factory":4713,"tiny":4714,"soccer":4715,"visited":4716,"matt":4717,"fm":4718,"achieved":4719,"52":4720,"defence":4721,"internal":4722,"persian":4723,"43":4724,"methods":4725,"##ging":4726,"arrested":4727,"otherwise":4728,"cambridge":4729,"programming":4730,"villages":4731,"elementary":4732,"districts":4733,"rooms":4734,"criminal":4735,"conflict":4736,"worry":4737,"trained":4738,"1931":4739,"attempts":4740,"waited":4741,"signal":4742,"bird":4743,"truck":4744,"subsequent":4745,"programme":4746,"##ol":4747,"ad":4748,"49":4749,"communist":4750,"details":4751,"faith":4752,"sector":4753,"patrick":4754,"carrying":4755,"laugh":4756,"##ss":4757,"controlled":4758,"korean":4759,"showing":4760,"origin":4761,"fuel":4762,"evil":4763,"1927":4764,"##ent":4765,"brief":4766,"identity":4767,"darkness":4768,"address":4769,"pool":4770,"missed":4771,"publication":4772,"web":4773,"planet":4774,"ian":4775,"anne":4776,"wings":4777,"invited":4778,"##tt":4779,"briefly":4780,"standards":4781,"kissed":4782,"##be":4783,"ideas":4784,"climate":4785,"causing":4786,"walter":4787,"worse":4788,"albert":4789,"articles":4790,"winners":4791,"desire":4792,"aged":4793,"northeast":4794,"dangerous":4795,"gate":4796,"doubt":4797,"1922":4798,"wooden":4799,"multi":4800,"##ky":4801,"poet":4802,"rising":4803,"funding":4804,"46":4805,"communications":4806,"communication":4807,"violence":4808,"copies":4809,"prepared":4810,"ford":4811,"investigation":4812,"skills":4813,"1924":4814,"pulling":4815,"electronic":4816,"##ak":4817,"##ial":4818,"##han":4819,"containing":4820,"ultimately":4821,"offices":4822,"singing":4823,"understanding":4824,"restaurant":4825,"tomorrow":4826,"fashion":4827,"christ":4828,"ward":4829,"da":4830,"pope":4831,"stands":4832,"5th":4833,"flow":4834,"studios":4835,"aired":4836,"commissioned":4837,"contained":4838,"exist":4839,"fresh":4840,"americans":4841,"##per":4842,"wrestling":4843,"approved":4844,"kid":4845,"employed":4846,"respect":4847,"suit":4848,"1925":4849,"angel":4850,"asking":4851,"increasing":4852,"frame":4853,"angry":4854,"selling":4855,"1950s":4856,"thin":4857,"finds":4858,"##nd":4859,"temperature":4860,"statement":4861,"ali":4862,"explain":4863,"inhabitants":4864,"towns":4865,"extensive":4866,"narrow":4867,"51":4868,"jane":4869,"flowers":4870,"images":4871,"promise":4872,"somewhere":4873,"object":4874,"fly":4875,"closely":4876,"##ls":4877,"1912":4878,"bureau":4879,"cape":4880,"1926":4881,"weekly":4882,"presidential":4883,"legislative":4884,"1921":4885,"##ai":4886,"##au":4887,"launch":4888,"founding":4889,"##ny":4890,"978":4891,"##ring":4892,"artillery":4893,"strike":4894,"un":4895,"institutions":4896,"roll":4897,"writers":4898,"landing":4899,"chose":4900,"kevin":4901,"anymore":4902,"pp":4903,"##ut":4904,"attorney":4905,"fit":4906,"dan":4907,"billboard":4908,"receiving":4909,"agricultural":4910,"breaking":4911,"sought":4912,"dave":4913,"admitted":4914,"lands":4915,"mexican":4916,"##bury":4917,"charlie":4918,"specifically":4919,"hole":4920,"iv":4921,"howard":4922,"credit":4923,"moscow":4924,"roads":4925,"accident":4926,"1923":4927,"proved":4928,"wear":4929,"struck":4930,"hey":4931,"guards":4932,"stuff":4933,"slid":4934,"expansion":4935,"1915":4936,"cat":4937,"anthony":4938,"##kin":4939,"melbourne":4940,"opposed":4941,"sub":4942,"southwest":4943,"architect":4944,"failure":4945,"plane":4946,"1916":4947,"##ron":4948,"map":4949,"camera":4950,"tank":4951,"listen":4952,"regarding":4953,"wet":4954,"introduction":4955,"metropolitan":4956,"link":4957,"ep":4958,"fighter":4959,"inch":4960,"grown":4961,"gene":4962,"anger":4963,"fixed":4964,"buy":4965,"dvd":4966,"khan":4967,"domestic":4968,"worldwide":4969,"chapel":4970,"mill":4971,"functions":4972,"examples":4973,"##head":4974,"developing":4975,"1910":4976,"turkey":4977,"hits":4978,"pocket":4979,"antonio":4980,"papers":4981,"grow":4982,"unless":4983,"circuit":4984,"18th":4985,"concerned":4986,"attached":4987,"journalist":4988,"selection":4989,"journey":4990,"converted":4991,"provincial":4992,"painted":4993,"hearing":4994,"aren":4995,"bands":4996,"negative":4997,"aside":4998,"wondered":4999,"knight":5000,"lap":5001,"survey":5002,"ma":5003,"##ow":5004,"noise":5005,"billy":5006,"##ium":5007,"shooting":5008,"guide":5009,"bedroom":5010,"priest":5011,"resistance":5012,"motor":5013,"homes":5014,"sounded":5015,"giant":5016,"##mer":5017,"150":5018,"scenes":5019,"equal":5020,"comic":5021,"patients":5022,"hidden":5023,"solid":5024,"actual":5025,"bringing":5026,"afternoon":5027,"touched":5028,"funds":5029,"wedding":5030,"consisted":5031,"marie":5032,"canal":5033,"sr":5034,"kim":5035,"treaty":5036,"turkish":5037,"recognition":5038,"residence":5039,"cathedral":5040,"broad":5041,"knees":5042,"incident":5043,"shaped":5044,"fired":5045,"norwegian":5046,"handle":5047,"cheek":5048,"contest":5049,"represent":5050,"##pe":5051,"representing":5052,"beauty":5053,"##sen":5054,"birds":5055,"advantage":5056,"emergency":5057,"wrapped":5058,"drawing":5059,"notice":5060,"pink":5061,"broadcasting":5062,"##ong":5063,"somehow":5064,"bachelor":5065,"seventh":5066,"collected":5067,"registered":5068,"establishment":5069,"alan":5070,"assumed":5071,"chemical":5072,"personnel":5073,"roger":5074,"retirement":5075,"jeff":5076,"portuguese":5077,"wore":5078,"tied":5079,"device":5080,"threat":5081,"progress":5082,"advance":5083,"##ised":5084,"banks":5085,"hired":5086,"manchester":5087,"nfl":5088,"teachers":5089,"structures":5090,"forever":5091,"##bo":5092,"tennis":5093,"helping":5094,"saturday":5095,"sale":5096,"applications":5097,"junction":5098,"hip":5099,"incorporated":5100,"neighborhood":5101,"dressed":5102,"ceremony":5103,"##ds":5104,"influenced":5105,"hers":5106,"visual":5107,"stairs":5108,"decades":5109,"inner":5110,"kansas":5111,"hung":5112,"hoped":5113,"gain":5114,"scheduled":5115,"downtown":5116,"engaged":5117,"austria":5118,"clock":5119,"norway":5120,"certainly":5121,"pale":5122,"protected":5123,"1913":5124,"victor":5125,"employees":5126,"plate":5127,"putting":5128,"surrounded":5129,"##ists":5130,"finishing":5131,"blues":5132,"tropical":5133,"##ries":5134,"minnesota":5135,"consider":5136,"philippines":5137,"accept":5138,"54":5139,"retrieved":5140,"1900":5141,"concern":5142,"anderson":5143,"properties":5144,"institution":5145,"gordon":5146,"successfully":5147,"vietnam":5148,"##dy":5149,"backing":5150,"outstanding":5151,"muslim":5152,"crossing":5153,"folk":5154,"producing":5155,"usual":5156,"demand":5157,"occurs":5158,"observed":5159,"lawyer":5160,"educated":5161,"##ana":5162,"kelly":5163,"string":5164,"pleasure":5165,"budget":5166,"items":5167,"quietly":5168,"colorado":5169,"philip":5170,"typical":5171,"##worth":5172,"derived":5173,"600":5174,"survived":5175,"asks":5176,"mental":5177,"##ide":5178,"56":5179,"jake":5180,"jews":5181,"distinguished":5182,"ltd":5183,"1911":5184,"sri":5185,"extremely":5186,"53":5187,"athletic":5188,"loud":5189,"thousands":5190,"worried":5191,"shadow":5192,"transportation":5193,"horses":5194,"weapon":5195,"arena":5196,"importance":5197,"users":5198,"tim":5199,"objects":5200,"contributed":5201,"dragon":5202,"douglas":5203,"aware":5204,"senator":5205,"johnny":5206,"jordan":5207,"sisters":5208,"engines":5209,"flag":5210,"investment":5211,"samuel":5212,"shock":5213,"capable":5214,"clark":5215,"row":5216,"wheel":5217,"refers":5218,"session":5219,"familiar":5220,"biggest":5221,"wins":5222,"hate":5223,"maintained":5224,"drove":5225,"hamilton":5226,"request":5227,"expressed":5228,"injured":5229,"underground":5230,"churches":5231,"walker":5232,"wars":5233,"tunnel":5234,"passes":5235,"stupid":5236,"agriculture":5237,"softly":5238,"cabinet":5239,"regarded":5240,"joining":5241,"indiana":5242,"##ea":5243,"##ms":5244,"push":5245,"dates":5246,"spend":5247,"behavior":5248,"woods":5249,"protein":5250,"gently":5251,"chase":5252,"morgan":5253,"mention":5254,"burning":5255,"wake":5256,"combination":5257,"occur":5258,"mirror":5259,"leads":5260,"jimmy":5261,"indeed":5262,"impossible":5263,"singapore":5264,"paintings":5265,"covering":5266,"##nes":5267,"soldier":5268,"locations":5269,"attendance":5270,"sell":5271,"historian":5272,"wisconsin":5273,"invasion":5274,"argued":5275,"painter":5276,"diego":5277,"changing":5278,"egypt":5279,"##don":5280,"experienced":5281,"inches":5282,"##ku":5283,"missouri":5284,"vol":5285,"grounds":5286,"spoken":5287,"switzerland":5288,"##gan":5289,"reform":5290,"rolling":5291,"ha":5292,"forget":5293,"massive":5294,"resigned":5295,"burned":5296,"allen":5297,"tennessee":5298,"locked":5299,"values":5300,"improved":5301,"##mo":5302,"wounded":5303,"universe":5304,"sick":5305,"dating":5306,"facing":5307,"pack":5308,"purchase":5309,"user":5310,"##pur":5311,"moments":5312,"##ul":5313,"merged":5314,"anniversary":5315,"1908":5316,"coal":5317,"brick":5318,"understood":5319,"causes":5320,"dynasty":5321,"queensland":5322,"establish":5323,"stores":5324,"crisis":5325,"promote":5326,"hoping":5327,"views":5328,"cards":5329,"referee":5330,"extension":5331,"##si":5332,"raise":5333,"arizona":5334,"improve":5335,"colonial":5336,"formal":5337,"charged":5338,"##rt":5339,"palm":5340,"lucky":5341,"hide":5342,"rescue":5343,"faces":5344,"95":5345,"feelings":5346,"candidates":5347,"juan":5348,"##ell":5349,"goods":5350,"6th":5351,"courses":5352,"weekend":5353,"59":5354,"luke":5355,"cash":5356,"fallen":5357,"##om":5358,"delivered":5359,"affected":5360,"installed":5361,"carefully":5362,"tries":5363,"swiss":5364,"hollywood":5365,"costs":5366,"lincoln":5367,"responsibility":5368,"##he":5369,"shore":5370,"file":5371,"proper":5372,"normally":5373,"maryland":5374,"assistance":5375,"jump":5376,"constant":5377,"offering":5378,"friendly":5379,"waters":5380,"persons":5381,"realize":5382,"contain":5383,"trophy":5384,"800":5385,"partnership":5386,"factor":5387,"58":5388,"musicians":5389,"cry":5390,"bound":5391,"oregon":5392,"indicated":5393,"hero":5394,"houston":5395,"medium":5396,"##ure":5397,"consisting":5398,"somewhat":5399,"##ara":5400,"57":5401,"cycle":5402,"##che":5403,"beer":5404,"moore":5405,"frederick":5406,"gotten":5407,"eleven":5408,"worst":5409,"weak":5410,"approached":5411,"arranged":5412,"chin":5413,"loan":5414,"universal":5415,"bond":5416,"fifteen":5417,"pattern":5418,"disappeared":5419,"##ney":5420,"translated":5421,"##zed":5422,"lip":5423,"arab":5424,"capture":5425,"interests":5426,"insurance":5427,"##chi":5428,"shifted":5429,"cave":5430,"prix":5431,"warning":5432,"sections":5433,"courts":5434,"coat":5435,"plot":5436,"smell":5437,"feed":5438,"golf":5439,"favorite":5440,"maintain":5441,"knife":5442,"vs":5443,"voted":5444,"degrees":5445,"finance":5446,"quebec":5447,"opinion":5448,"translation":5449,"manner":5450,"ruled":5451,"operate":5452,"productions":5453,"choose":5454,"musician":5455,"discovery":5456,"confused":5457,"tired":5458,"separated":5459,"stream":5460,"techniques":5461,"committed":5462,"attend":5463,"ranking":5464,"kings":5465,"throw":5466,"passengers":5467,"measure":5468,"horror":5469,"fan":5470,"mining":5471,"sand":5472,"danger":5473,"salt":5474,"calm":5475,"decade":5476,"dam":5477,"require":5478,"runner":5479,"##ik":5480,"rush":5481,"associate":5482,"greece":5483,"##ker":5484,"rivers":5485,"consecutive":5486,"matthew":5487,"##ski":5488,"sighed":5489,"sq":5490,"documents":5491,"steam":5492,"edited":5493,"closing":5494,"tie":5495,"accused":5496,"1905":5497,"##ini":5498,"islamic":5499,"distributed":5500,"directors":5501,"organisation":5502,"bruce":5503,"7th":5504,"breathing":5505,"mad":5506,"lit":5507,"arrival":5508,"concrete":5509,"taste":5510,"08":5511,"composition":5512,"shaking":5513,"faster":5514,"amateur":5515,"adjacent":5516,"stating":5517,"1906":5518,"twin":5519,"flew":5520,"##ran":5521,"tokyo":5522,"publications":5523,"##tone":5524,"obviously":5525,"ridge":5526,"storage":5527,"1907":5528,"carl":5529,"pages":5530,"concluded":5531,"desert":5532,"driven":5533,"universities":5534,"ages":5535,"terminal":5536,"sequence":5537,"borough":5538,"250":5539,"constituency":5540,"creative":5541,"cousin":5542,"economics":5543,"dreams":5544,"margaret":5545,"notably":5546,"reduce":5547,"montreal":5548,"mode":5549,"17th":5550,"ears":5551,"saved":5552,"jan":5553,"vocal":5554,"##ica":5555,"1909":5556,"andy":5557,"##jo":5558,"riding":5559,"roughly":5560,"threatened":5561,"##ise":5562,"meters":5563,"meanwhile":5564,"landed":5565,"compete":5566,"repeated":5567,"grass":5568,"czech":5569,"regularly":5570,"charges":5571,"tea":5572,"sudden":5573,"appeal":5574,"##ung":5575,"solution":5576,"describes":5577,"pierre":5578,"classification":5579,"glad":5580,"parking":5581,"##ning":5582,"belt":5583,"physics":5584,"99":5585,"rachel":5586,"add":5587,"hungarian":5588,"participate":5589,"expedition":5590,"damaged":5591,"gift":5592,"childhood":5593,"85":5594,"fifty":5595,"##red":5596,"mathematics":5597,"jumped":5598,"letting":5599,"defensive":5600,"mph":5601,"##ux":5602,"##gh":5603,"testing":5604,"##hip":5605,"hundreds":5606,"shoot":5607,"owners":5608,"matters":5609,"smoke":5610,"israeli":5611,"kentucky":5612,"dancing":5613,"mounted":5614,"grandfather":5615,"emma":5616,"designs":5617,"profit":5618,"argentina":5619,"##gs":5620,"truly":5621,"li":5622,"lawrence":5623,"cole":5624,"begun":5625,"detroit":5626,"willing":5627,"branches":5628,"smiling":5629,"decide":5630,"miami":5631,"enjoyed":5632,"recordings":5633,"##dale":5634,"poverty":5635,"ethnic":5636,"gay":5637,"##bi":5638,"gary":5639,"arabic":5640,"09":5641,"accompanied":5642,"##one":5643,"##ons":5644,"fishing":5645,"determine":5646,"residential":5647,"acid":5648,"##ary":5649,"alice":5650,"returns":5651,"starred":5652,"mail":5653,"##ang":5654,"jonathan":5655,"strategy":5656,"##ue":5657,"net":5658,"forty":5659,"cook":5660,"businesses":5661,"equivalent":5662,"commonwealth":5663,"distinct":5664,"ill":5665,"##cy":5666,"seriously":5667,"##ors":5668,"##ped":5669,"shift":5670,"harris":5671,"replace":5672,"rio":5673,"imagine":5674,"formula":5675,"ensure":5676,"##ber":5677,"additionally":5678,"scheme":5679,"conservation":5680,"occasionally":5681,"purposes":5682,"feels":5683,"favor":5684,"##and":5685,"##ore":5686,"1930s":5687,"contrast":5688,"hanging":5689,"hunt":5690,"movies":5691,"1904":5692,"instruments":5693,"victims":5694,"danish":5695,"christopher":5696,"busy":5697,"demon":5698,"sugar":5699,"earliest":5700,"colony":5701,"studying":5702,"balance":5703,"duties":5704,"##ks":5705,"belgium":5706,"slipped":5707,"carter":5708,"05":5709,"visible":5710,"stages":5711,"iraq":5712,"fifa":5713,"##im":5714,"commune":5715,"forming":5716,"zero":5717,"07":5718,"continuing":5719,"talked":5720,"counties":5721,"legend":5722,"bathroom":5723,"option":5724,"tail":5725,"clay":5726,"daughters":5727,"afterwards":5728,"severe":5729,"jaw":5730,"visitors":5731,"##ded":5732,"devices":5733,"aviation":5734,"russell":5735,"kate":5736,"##vi":5737,"entering":5738,"subjects":5739,"##ino":5740,"temporary":5741,"swimming":5742,"forth":5743,"smooth":5744,"ghost":5745,"audio":5746,"bush":5747,"operates":5748,"rocks":5749,"movements":5750,"signs":5751,"eddie":5752,"##tz":5753,"ann":5754,"voices":5755,"honorary":5756,"06":5757,"memories":5758,"dallas":5759,"pure":5760,"measures":5761,"racial":5762,"promised":5763,"66":5764,"harvard":5765,"ceo":5766,"16th":5767,"parliamentary":5768,"indicate":5769,"benefit":5770,"flesh":5771,"dublin":5772,"louisiana":5773,"1902":5774,"1901":5775,"patient":5776,"sleeping":5777,"1903":5778,"membership":5779,"coastal":5780,"medieval":5781,"wanting":5782,"element":5783,"scholars":5784,"rice":5785,"62":5786,"limit":5787,"survive":5788,"makeup":5789,"rating":5790,"definitely":5791,"collaboration":5792,"obvious":5793,"##tan":5794,"boss":5795,"ms":5796,"baron":5797,"birthday":5798,"linked":5799,"soil":5800,"diocese":5801,"##lan":5802,"ncaa":5803,"##mann":5804,"offensive":5805,"shell":5806,"shouldn":5807,"waist":5808,"##tus":5809,"plain":5810,"ross":5811,"organ":5812,"resolution":5813,"manufacturing":5814,"adding":5815,"relative":5816,"kennedy":5817,"98":5818,"whilst":5819,"moth":5820,"marketing":5821,"gardens":5822,"crash":5823,"72":5824,"heading":5825,"partners":5826,"credited":5827,"carlos":5828,"moves":5829,"cable":5830,"##zi":5831,"marshall":5832,"##out":5833,"depending":5834,"bottle":5835,"represents":5836,"rejected":5837,"responded":5838,"existed":5839,"04":5840,"jobs":5841,"denmark":5842,"lock":5843,"##ating":5844,"treated":5845,"graham":5846,"routes":5847,"talent":5848,"commissioner":5849,"drugs":5850,"secure":5851,"tests":5852,"reign":5853,"restored":5854,"photography":5855,"##gi":5856,"contributions":5857,"oklahoma":5858,"designer":5859,"disc":5860,"grin":5861,"seattle":5862,"robin":5863,"paused":5864,"atlanta":5865,"unusual":5866,"##gate":5867,"praised":5868,"las":5869,"laughing":5870,"satellite":5871,"hungary":5872,"visiting":5873,"##sky":5874,"interesting":5875,"factors":5876,"deck":5877,"poems":5878,"norman":5879,"##water":5880,"stuck":5881,"speaker":5882,"rifle":5883,"domain":5884,"premiered":5885,"##her":5886,"dc":5887,"comics":5888,"actors":5889,"01":5890,"reputation":5891,"eliminated":5892,"8th":5893,"ceiling":5894,"prisoners":5895,"script":5896,"##nce":5897,"leather":5898,"austin":5899,"mississippi":5900,"rapidly":5901,"admiral":5902,"parallel":5903,"charlotte":5904,"guilty":5905,"tools":5906,"gender":5907,"divisions":5908,"fruit":5909,"##bs":5910,"laboratory":5911,"nelson":5912,"fantasy":5913,"marry":5914,"rapid":5915,"aunt":5916,"tribe":5917,"requirements":5918,"aspects":5919,"suicide":5920,"amongst":5921,"adams":5922,"bone":5923,"ukraine":5924,"abc":5925,"kick":5926,"sees":5927,"edinburgh":5928,"clothing":5929,"column":5930,"rough":5931,"gods":5932,"hunting":5933,"broadway":5934,"gathered":5935,"concerns":5936,"##ek":5937,"spending":5938,"ty":5939,"12th":5940,"snapped":5941,"requires":5942,"solar":5943,"bones":5944,"cavalry":5945,"##tta":5946,"iowa":5947,"drinking":5948,"waste":5949,"index":5950,"franklin":5951,"charity":5952,"thompson":5953,"stewart":5954,"tip":5955,"flash":5956,"landscape":5957,"friday":5958,"enjoy":5959,"singh":5960,"poem":5961,"listening":5962,"##back":5963,"eighth":5964,"fred":5965,"differences":5966,"adapted":5967,"bomb":5968,"ukrainian":5969,"surgery":5970,"corporate":5971,"masters":5972,"anywhere":5973,"##more":5974,"waves":5975,"odd":5976,"sean":5977,"portugal":5978,"orleans":5979,"dick":5980,"debate":5981,"kent":5982,"eating":5983,"puerto":5984,"cleared":5985,"96":5986,"expect":5987,"cinema":5988,"97":5989,"guitarist":5990,"blocks":5991,"electrical":5992,"agree":5993,"involving":5994,"depth":5995,"dying":5996,"panel":5997,"struggle":5998,"##ged":5999,"peninsula":6000,"adults":6001,"novels":6002,"emerged":6003,"vienna":6004,"metro":6005,"debuted":6006,"shoes":6007,"tamil":6008,"songwriter":6009,"meets":6010,"prove":6011,"beating":6012,"instance":6013,"heaven":6014,"scared":6015,"sending":6016,"marks":6017,"artistic":6018,"passage":6019,"superior":6020,"03":6021,"significantly":6022,"shopping":6023,"##tive":6024,"retained":6025,"##izing":6026,"malaysia":6027,"technique":6028,"cheeks":6029,"##ola":6030,"warren":6031,"maintenance":6032,"destroy":6033,"extreme":6034,"allied":6035,"120":6036,"appearing":6037,"##yn":6038,"fill":6039,"advice":6040,"alabama":6041,"qualifying":6042,"policies":6043,"cleveland":6044,"hat":6045,"battery":6046,"smart":6047,"authors":6048,"10th":6049,"soundtrack":6050,"acted":6051,"dated":6052,"lb":6053,"glance":6054,"equipped":6055,"coalition":6056,"funny":6057,"outer":6058,"ambassador":6059,"roy":6060,"possibility":6061,"couples":6062,"campbell":6063,"dna":6064,"loose":6065,"ethan":6066,"supplies":6067,"1898":6068,"gonna":6069,"88":6070,"monster":6071,"##res":6072,"shake":6073,"agents":6074,"frequency":6075,"springs":6076,"dogs":6077,"practices":6078,"61":6079,"gang":6080,"plastic":6081,"easier":6082,"suggests":6083,"gulf":6084,"blade":6085,"exposed":6086,"colors":6087,"industries":6088,"markets":6089,"pan":6090,"nervous":6091,"electoral":6092,"charts":6093,"legislation":6094,"ownership":6095,"##idae":6096,"mac":6097,"appointment":6098,"shield":6099,"copy":6100,"assault":6101,"socialist":6102,"abbey":6103,"monument":6104,"license":6105,"throne":6106,"employment":6107,"jay":6108,"93":6109,"replacement":6110,"charter":6111,"cloud":6112,"powered":6113,"suffering":6114,"accounts":6115,"oak":6116,"connecticut":6117,"strongly":6118,"wright":6119,"colour":6120,"crystal":6121,"13th":6122,"context":6123,"welsh":6124,"networks":6125,"voiced":6126,"gabriel":6127,"jerry":6128,"##cing":6129,"forehead":6130,"mp":6131,"##ens":6132,"manage":6133,"schedule":6134,"totally":6135,"remix":6136,"##ii":6137,"forests":6138,"occupation":6139,"print":6140,"nicholas":6141,"brazilian":6142,"strategic":6143,"vampires":6144,"engineers":6145,"76":6146,"roots":6147,"seek":6148,"correct":6149,"instrumental":6150,"und":6151,"alfred":6152,"backed":6153,"hop":6154,"##des":6155,"stanley":6156,"robinson":6157,"traveled":6158,"wayne":6159,"welcome":6160,"austrian":6161,"achieve":6162,"67":6163,"exit":6164,"rates":6165,"1899":6166,"strip":6167,"whereas":6168,"##cs":6169,"sing":6170,"deeply":6171,"adventure":6172,"bobby":6173,"rick":6174,"jamie":6175,"careful":6176,"components":6177,"cap":6178,"useful":6179,"personality":6180,"knee":6181,"##shi":6182,"pushing":6183,"hosts":6184,"02":6185,"protest":6186,"ca":6187,"ottoman":6188,"symphony":6189,"##sis":6190,"63":6191,"boundary":6192,"1890":6193,"processes":6194,"considering":6195,"considerable":6196,"tons":6197,"##work":6198,"##ft":6199,"##nia":6200,"cooper":6201,"trading":6202,"dear":6203,"conduct":6204,"91":6205,"illegal":6206,"apple":6207,"revolutionary":6208,"holiday":6209,"definition":6210,"harder":6211,"##van":6212,"jacob":6213,"circumstances":6214,"destruction":6215,"##lle":6216,"popularity":6217,"grip":6218,"classified":6219,"liverpool":6220,"donald":6221,"baltimore":6222,"flows":6223,"seeking":6224,"honour":6225,"approval":6226,"92":6227,"mechanical":6228,"till":6229,"happening":6230,"statue":6231,"critic":6232,"increasingly":6233,"immediate":6234,"describe":6235,"commerce":6236,"stare":6237,"##ster":6238,"indonesia":6239,"meat":6240,"rounds":6241,"boats":6242,"baker":6243,"orthodox":6244,"depression":6245,"formally":6246,"worn":6247,"naked":6248,"claire":6249,"muttered":6250,"sentence":6251,"11th":6252,"emily":6253,"document":6254,"77":6255,"criticism":6256,"wished":6257,"vessel":6258,"spiritual":6259,"bent":6260,"virgin":6261,"parker":6262,"minimum":6263,"murray":6264,"lunch":6265,"danny":6266,"printed":6267,"compilation":6268,"keyboards":6269,"false":6270,"blow":6271,"belonged":6272,"68":6273,"raising":6274,"78":6275,"cutting":6276,"##board":6277,"pittsburgh":6278,"##up":6279,"9th":6280,"shadows":6281,"81":6282,"hated":6283,"indigenous":6284,"jon":6285,"15th":6286,"barry":6287,"scholar":6288,"ah":6289,"##zer":6290,"oliver":6291,"##gy":6292,"stick":6293,"susan":6294,"meetings":6295,"attracted":6296,"spell":6297,"romantic":6298,"##ver":6299,"ye":6300,"1895":6301,"photo":6302,"demanded":6303,"customers":6304,"##ac":6305,"1896":6306,"logan":6307,"revival":6308,"keys":6309,"modified":6310,"commanded":6311,"jeans":6312,"##ious":6313,"upset":6314,"raw":6315,"phil":6316,"detective":6317,"hiding":6318,"resident":6319,"vincent":6320,"##bly":6321,"experiences":6322,"diamond":6323,"defeating":6324,"coverage":6325,"lucas":6326,"external":6327,"parks":6328,"franchise":6329,"helen":6330,"bible":6331,"successor":6332,"percussion":6333,"celebrated":6334,"il":6335,"lift":6336,"profile":6337,"clan":6338,"romania":6339,"##ied":6340,"mills":6341,"##su":6342,"nobody":6343,"achievement":6344,"shrugged":6345,"fault":6346,"1897":6347,"rhythm":6348,"initiative":6349,"breakfast":6350,"carbon":6351,"700":6352,"69":6353,"lasted":6354,"violent":6355,"74":6356,"wound":6357,"ken":6358,"killer":6359,"gradually":6360,"filmed":6361,"°c":6362,"dollars":6363,"processing":6364,"94":6365,"remove":6366,"criticized":6367,"guests":6368,"sang":6369,"chemistry":6370,"##vin":6371,"legislature":6372,"disney":6373,"##bridge":6374,"uniform":6375,"escaped":6376,"integrated":6377,"proposal":6378,"purple":6379,"denied":6380,"liquid":6381,"karl":6382,"influential":6383,"morris":6384,"nights":6385,"stones":6386,"intense":6387,"experimental":6388,"twisted":6389,"71":6390,"84":6391,"##ld":6392,"pace":6393,"nazi":6394,"mitchell":6395,"ny":6396,"blind":6397,"reporter":6398,"newspapers":6399,"14th":6400,"centers":6401,"burn":6402,"basin":6403,"forgotten":6404,"surviving":6405,"filed":6406,"collections":6407,"monastery":6408,"losses":6409,"manual":6410,"couch":6411,"description":6412,"appropriate":6413,"merely":6414,"tag":6415,"missions":6416,"sebastian":6417,"restoration":6418,"replacing":6419,"triple":6420,"73":6421,"elder":6422,"julia":6423,"warriors":6424,"benjamin":6425,"julian":6426,"convinced":6427,"stronger":6428,"amazing":6429,"declined":6430,"versus":6431,"merchant":6432,"happens":6433,"output":6434,"finland":6435,"bare":6436,"barbara":6437,"absence":6438,"ignored":6439,"dawn":6440,"injuries":6441,"##port":6442,"producers":6443,"##ram":6444,"82":6445,"luis":6446,"##ities":6447,"kw":6448,"admit":6449,"expensive":6450,"electricity":6451,"nba":6452,"exception":6453,"symbol":6454,"##ving":6455,"ladies":6456,"shower":6457,"sheriff":6458,"characteristics":6459,"##je":6460,"aimed":6461,"button":6462,"ratio":6463,"effectively":6464,"summit":6465,"angle":6466,"jury":6467,"bears":6468,"foster":6469,"vessels":6470,"pants":6471,"executed":6472,"evans":6473,"dozen":6474,"advertising":6475,"kicked":6476,"patrol":6477,"1889":6478,"competitions":6479,"lifetime":6480,"principles":6481,"athletics":6482,"##logy":6483,"birmingham":6484,"sponsored":6485,"89":6486,"rob":6487,"nomination":6488,"1893":6489,"acoustic":6490,"##sm":6491,"creature":6492,"longest":6493,"##tra":6494,"credits":6495,"harbor":6496,"dust":6497,"josh":6498,"##so":6499,"territories":6500,"milk":6501,"infrastructure":6502,"completion":6503,"thailand":6504,"indians":6505,"leon":6506,"archbishop":6507,"##sy":6508,"assist":6509,"pitch":6510,"blake":6511,"arrangement":6512,"girlfriend":6513,"serbian":6514,"operational":6515,"hence":6516,"sad":6517,"scent":6518,"fur":6519,"dj":6520,"sessions":6521,"hp":6522,"refer":6523,"rarely":6524,"##ora":6525,"exists":6526,"1892":6527,"##ten":6528,"scientists":6529,"dirty":6530,"penalty":6531,"burst":6532,"portrait":6533,"seed":6534,"79":6535,"pole":6536,"limits":6537,"rival":6538,"1894":6539,"stable":6540,"alpha":6541,"grave":6542,"constitutional":6543,"alcohol":6544,"arrest":6545,"flower":6546,"mystery":6547,"devil":6548,"architectural":6549,"relationships":6550,"greatly":6551,"habitat":6552,"##istic":6553,"larry":6554,"progressive":6555,"remote":6556,"cotton":6557,"##ics":6558,"##ok":6559,"preserved":6560,"reaches":6561,"##ming":6562,"cited":6563,"86":6564,"vast":6565,"scholarship":6566,"decisions":6567,"cbs":6568,"joy":6569,"teach":6570,"1885":6571,"editions":6572,"knocked":6573,"eve":6574,"searching":6575,"partly":6576,"participation":6577,"gap":6578,"animated":6579,"fate":6580,"excellent":6581,"##ett":6582,"na":6583,"87":6584,"alternate":6585,"saints":6586,"youngest":6587,"##ily":6588,"climbed":6589,"##ita":6590,"##tors":6591,"suggest":6592,"##ct":6593,"discussion":6594,"staying":6595,"choir":6596,"lakes":6597,"jacket":6598,"revenue":6599,"nevertheless":6600,"peaked":6601,"instrument":6602,"wondering":6603,"annually":6604,"managing":6605,"neil":6606,"1891":6607,"signing":6608,"terry":6609,"##ice":6610,"apply":6611,"clinical":6612,"brooklyn":6613,"aim":6614,"catherine":6615,"fuck":6616,"farmers":6617,"figured":6618,"ninth":6619,"pride":6620,"hugh":6621,"evolution":6622,"ordinary":6623,"involvement":6624,"comfortable":6625,"shouted":6626,"tech":6627,"encouraged":6628,"taiwan":6629,"representation":6630,"sharing":6631,"##lia":6632,"##em":6633,"panic":6634,"exact":6635,"cargo":6636,"competing":6637,"fat":6638,"cried":6639,"83":6640,"1920s":6641,"occasions":6642,"pa":6643,"cabin":6644,"borders":6645,"utah":6646,"marcus":6647,"##isation":6648,"badly":6649,"muscles":6650,"##ance":6651,"victorian":6652,"transition":6653,"warner":6654,"bet":6655,"permission":6656,"##rin":6657,"slave":6658,"terrible":6659,"similarly":6660,"shares":6661,"seth":6662,"uefa":6663,"possession":6664,"medals":6665,"benefits":6666,"colleges":6667,"lowered":6668,"perfectly":6669,"mall":6670,"transit":6671,"##ye":6672,"##kar":6673,"publisher":6674,"##ened":6675,"harrison":6676,"deaths":6677,"elevation":6678,"##ae":6679,"asleep":6680,"machines":6681,"sigh":6682,"ash":6683,"hardly":6684,"argument":6685,"occasion":6686,"parent":6687,"leo":6688,"decline":6689,"1888":6690,"contribution":6691,"##ua":6692,"concentration":6693,"1000":6694,"opportunities":6695,"hispanic":6696,"guardian":6697,"extent":6698,"emotions":6699,"hips":6700,"mason":6701,"volumes":6702,"bloody":6703,"controversy":6704,"diameter":6705,"steady":6706,"mistake":6707,"phoenix":6708,"identify":6709,"violin":6710,"##sk":6711,"departure":6712,"richmond":6713,"spin":6714,"funeral":6715,"enemies":6716,"1864":6717,"gear":6718,"literally":6719,"connor":6720,"random":6721,"sergeant":6722,"grab":6723,"confusion":6724,"1865":6725,"transmission":6726,"informed":6727,"op":6728,"leaning":6729,"sacred":6730,"suspended":6731,"thinks":6732,"gates":6733,"portland":6734,"luck":6735,"agencies":6736,"yours":6737,"hull":6738,"expert":6739,"muscle":6740,"layer":6741,"practical":6742,"sculpture":6743,"jerusalem":6744,"latest":6745,"lloyd":6746,"statistics":6747,"deeper":6748,"recommended":6749,"warrior":6750,"arkansas":6751,"mess":6752,"supports":6753,"greg":6754,"eagle":6755,"1880":6756,"recovered":6757,"rated":6758,"concerts":6759,"rushed":6760,"##ano":6761,"stops":6762,"eggs":6763,"files":6764,"premiere":6765,"keith":6766,"##vo":6767,"delhi":6768,"turner":6769,"pit":6770,"affair":6771,"belief":6772,"paint":6773,"##zing":6774,"mate":6775,"##ach":6776,"##ev":6777,"victim":6778,"##ology":6779,"withdrew":6780,"bonus":6781,"styles":6782,"fled":6783,"##ud":6784,"glasgow":6785,"technologies":6786,"funded":6787,"nbc":6788,"adaptation":6789,"##ata":6790,"portrayed":6791,"cooperation":6792,"supporters":6793,"judges":6794,"bernard":6795,"justin":6796,"hallway":6797,"ralph":6798,"##ick":6799,"graduating":6800,"controversial":6801,"distant":6802,"continental":6803,"spider":6804,"bite":6805,"##ho":6806,"recognize":6807,"intention":6808,"mixing":6809,"##ese":6810,"egyptian":6811,"bow":6812,"tourism":6813,"suppose":6814,"claiming":6815,"tiger":6816,"dominated":6817,"participants":6818,"vi":6819,"##ru":6820,"nurse":6821,"partially":6822,"tape":6823,"##rum":6824,"psychology":6825,"##rn":6826,"essential":6827,"touring":6828,"duo":6829,"voting":6830,"civilian":6831,"emotional":6832,"channels":6833,"##king":6834,"apparent":6835,"hebrew":6836,"1887":6837,"tommy":6838,"carrier":6839,"intersection":6840,"beast":6841,"hudson":6842,"##gar":6843,"##zo":6844,"lab":6845,"nova":6846,"bench":6847,"discuss":6848,"costa":6849,"##ered":6850,"detailed":6851,"behalf":6852,"drivers":6853,"unfortunately":6854,"obtain":6855,"##lis":6856,"rocky":6857,"##dae":6858,"siege":6859,"friendship":6860,"honey":6861,"##rian":6862,"1861":6863,"amy":6864,"hang":6865,"posted":6866,"governments":6867,"collins":6868,"respond":6869,"wildlife":6870,"preferred":6871,"operator":6872,"##po":6873,"laura":6874,"pregnant":6875,"videos":6876,"dennis":6877,"suspected":6878,"boots":6879,"instantly":6880,"weird":6881,"automatic":6882,"businessman":6883,"alleged":6884,"placing":6885,"throwing":6886,"ph":6887,"mood":6888,"1862":6889,"perry":6890,"venue":6891,"jet":6892,"remainder":6893,"##lli":6894,"##ci":6895,"passion":6896,"biological":6897,"boyfriend":6898,"1863":6899,"dirt":6900,"buffalo":6901,"ron":6902,"segment":6903,"fa":6904,"abuse":6905,"##era":6906,"genre":6907,"thrown":6908,"stroke":6909,"colored":6910,"stress":6911,"exercise":6912,"displayed":6913,"##gen":6914,"struggled":6915,"##tti":6916,"abroad":6917,"dramatic":6918,"wonderful":6919,"thereafter":6920,"madrid":6921,"component":6922,"widespread":6923,"##sed":6924,"tale":6925,"citizen":6926,"todd":6927,"monday":6928,"1886":6929,"vancouver":6930,"overseas":6931,"forcing":6932,"crying":6933,"descent":6934,"##ris":6935,"discussed":6936,"substantial":6937,"ranks":6938,"regime":6939,"1870":6940,"provinces":6941,"switch":6942,"drum":6943,"zane":6944,"ted":6945,"tribes":6946,"proof":6947,"lp":6948,"cream":6949,"researchers":6950,"volunteer":6951,"manor":6952,"silk":6953,"milan":6954,"donated":6955,"allies":6956,"venture":6957,"principle":6958,"delivery":6959,"enterprise":6960,"##ves":6961,"##ans":6962,"bars":6963,"traditionally":6964,"witch":6965,"reminded":6966,"copper":6967,"##uk":6968,"pete":6969,"inter":6970,"links":6971,"colin":6972,"grinned":6973,"elsewhere":6974,"competitive":6975,"frequent":6976,"##oy":6977,"scream":6978,"##hu":6979,"tension":6980,"texts":6981,"submarine":6982,"finnish":6983,"defending":6984,"defend":6985,"pat":6986,"detail":6987,"1884":6988,"affiliated":6989,"stuart":6990,"themes":6991,"villa":6992,"periods":6993,"tool":6994,"belgian":6995,"ruling":6996,"crimes":6997,"answers":6998,"folded":6999,"licensed":7000,"resort":7001,"demolished":7002,"hans":7003,"lucy":7004,"1881":7005,"lion":7006,"traded":7007,"photographs":7008,"writes":7009,"craig":7010,"##fa":7011,"trials":7012,"generated":7013,"beth":7014,"noble":7015,"debt":7016,"percentage":7017,"yorkshire":7018,"erected":7019,"ss":7020,"viewed":7021,"grades":7022,"confidence":7023,"ceased":7024,"islam":7025,"telephone":7026,"retail":7027,"##ible":7028,"chile":7029,"m²":7030,"roberts":7031,"sixteen":7032,"##ich":7033,"commented":7034,"hampshire":7035,"innocent":7036,"dual":7037,"pounds":7038,"checked":7039,"regulations":7040,"afghanistan":7041,"sung":7042,"rico":7043,"liberty":7044,"assets":7045,"bigger":7046,"options":7047,"angels":7048,"relegated":7049,"tribute":7050,"wells":7051,"attending":7052,"leaf":7053,"##yan":7054,"butler":7055,"romanian":7056,"forum":7057,"monthly":7058,"lisa":7059,"patterns":7060,"gmina":7061,"##tory":7062,"madison":7063,"hurricane":7064,"rev":7065,"##ians":7066,"bristol":7067,"##ula":7068,"elite":7069,"valuable":7070,"disaster":7071,"democracy":7072,"awareness":7073,"germans":7074,"freyja":7075,"##ins":7076,"loop":7077,"absolutely":7078,"paying":7079,"populations":7080,"maine":7081,"sole":7082,"prayer":7083,"spencer":7084,"releases":7085,"doorway":7086,"bull":7087,"##ani":7088,"lover":7089,"midnight":7090,"conclusion":7091,"##sson":7092,"thirteen":7093,"lily":7094,"mediterranean":7095,"##lt":7096,"nhl":7097,"proud":7098,"sample":7099,"##hill":7100,"drummer":7101,"guinea":7102,"##ova":7103,"murphy":7104,"climb":7105,"##ston":7106,"instant":7107,"attributed":7108,"horn":7109,"ain":7110,"railways":7111,"steven":7112,"##ao":7113,"autumn":7114,"ferry":7115,"opponent":7116,"root":7117,"traveling":7118,"secured":7119,"corridor":7120,"stretched":7121,"tales":7122,"sheet":7123,"trinity":7124,"cattle":7125,"helps":7126,"indicates":7127,"manhattan":7128,"murdered":7129,"fitted":7130,"1882":7131,"gentle":7132,"grandmother":7133,"mines":7134,"shocked":7135,"vegas":7136,"produces":7137,"##light":7138,"caribbean":7139,"##ou":7140,"belong":7141,"continuous":7142,"desperate":7143,"drunk":7144,"historically":7145,"trio":7146,"waved":7147,"raf":7148,"dealing":7149,"nathan":7150,"bat":7151,"murmured":7152,"interrupted":7153,"residing":7154,"scientist":7155,"pioneer":7156,"harold":7157,"aaron":7158,"##net":7159,"delta":7160,"attempting":7161,"minority":7162,"mini":7163,"believes":7164,"chorus":7165,"tend":7166,"lots":7167,"eyed":7168,"indoor":7169,"load":7170,"shots":7171,"updated":7172,"jail":7173,"##llo":7174,"concerning":7175,"connecting":7176,"wealth":7177,"##ved":7178,"slaves":7179,"arrive":7180,"rangers":7181,"sufficient":7182,"rebuilt":7183,"##wick":7184,"cardinal":7185,"flood":7186,"muhammad":7187,"whenever":7188,"relation":7189,"runners":7190,"moral":7191,"repair":7192,"viewers":7193,"arriving":7194,"revenge":7195,"punk":7196,"assisted":7197,"bath":7198,"fairly":7199,"breathe":7200,"lists":7201,"innings":7202,"illustrated":7203,"whisper":7204,"nearest":7205,"voters":7206,"clinton":7207,"ties":7208,"ultimate":7209,"screamed":7210,"beijing":7211,"lions":7212,"andre":7213,"fictional":7214,"gathering":7215,"comfort":7216,"radar":7217,"suitable":7218,"dismissed":7219,"hms":7220,"ban":7221,"pine":7222,"wrist":7223,"atmosphere":7224,"voivodeship":7225,"bid":7226,"timber":7227,"##ned":7228,"##nan":7229,"giants":7230,"##ane":7231,"cameron":7232,"recovery":7233,"uss":7234,"identical":7235,"categories":7236,"switched":7237,"serbia":7238,"laughter":7239,"noah":7240,"ensemble":7241,"therapy":7242,"peoples":7243,"touching":7244,"##off":7245,"locally":7246,"pearl":7247,"platforms":7248,"everywhere":7249,"ballet":7250,"tables":7251,"lanka":7252,"herbert":7253,"outdoor":7254,"toured":7255,"derek":7256,"1883":7257,"spaces":7258,"contested":7259,"swept":7260,"1878":7261,"exclusive":7262,"slight":7263,"connections":7264,"##dra":7265,"winds":7266,"prisoner":7267,"collective":7268,"bangladesh":7269,"tube":7270,"publicly":7271,"wealthy":7272,"thai":7273,"##ys":7274,"isolated":7275,"select":7276,"##ric":7277,"insisted":7278,"pen":7279,"fortune":7280,"ticket":7281,"spotted":7282,"reportedly":7283,"animation":7284,"enforcement":7285,"tanks":7286,"110":7287,"decides":7288,"wider":7289,"lowest":7290,"owen":7291,"##time":7292,"nod":7293,"hitting":7294,"##hn":7295,"gregory":7296,"furthermore":7297,"magazines":7298,"fighters":7299,"solutions":7300,"##ery":7301,"pointing":7302,"requested":7303,"peru":7304,"reed":7305,"chancellor":7306,"knights":7307,"mask":7308,"worker":7309,"eldest":7310,"flames":7311,"reduction":7312,"1860":7313,"volunteers":7314,"##tis":7315,"reporting":7316,"##hl":7317,"wire":7318,"advisory":7319,"endemic":7320,"origins":7321,"settlers":7322,"pursue":7323,"knock":7324,"consumer":7325,"1876":7326,"eu":7327,"compound":7328,"creatures":7329,"mansion":7330,"sentenced":7331,"ivan":7332,"deployed":7333,"guitars":7334,"frowned":7335,"involves":7336,"mechanism":7337,"kilometers":7338,"perspective":7339,"shops":7340,"maps":7341,"terminus":7342,"duncan":7343,"alien":7344,"fist":7345,"bridges":7346,"##pers":7347,"heroes":7348,"fed":7349,"derby":7350,"swallowed":7351,"##ros":7352,"patent":7353,"sara":7354,"illness":7355,"characterized":7356,"adventures":7357,"slide":7358,"hawaii":7359,"jurisdiction":7360,"##op":7361,"organised":7362,"##side":7363,"adelaide":7364,"walks":7365,"biology":7366,"se":7367,"##ties":7368,"rogers":7369,"swing":7370,"tightly":7371,"boundaries":7372,"##rie":7373,"prepare":7374,"implementation":7375,"stolen":7376,"##sha":7377,"certified":7378,"colombia":7379,"edwards":7380,"garage":7381,"##mm":7382,"recalled":7383,"##ball":7384,"rage":7385,"harm":7386,"nigeria":7387,"breast":7388,"##ren":7389,"furniture":7390,"pupils":7391,"settle":7392,"##lus":7393,"cuba":7394,"balls":7395,"client":7396,"alaska":7397,"21st":7398,"linear":7399,"thrust":7400,"celebration":7401,"latino":7402,"genetic":7403,"terror":7404,"##cia":7405,"##ening":7406,"lightning":7407,"fee":7408,"witness":7409,"lodge":7410,"establishing":7411,"skull":7412,"##ique":7413,"earning":7414,"hood":7415,"##ei":7416,"rebellion":7417,"wang":7418,"sporting":7419,"warned":7420,"missile":7421,"devoted":7422,"activist":7423,"porch":7424,"worship":7425,"fourteen":7426,"package":7427,"1871":7428,"decorated":7429,"##shire":7430,"housed":7431,"##ock":7432,"chess":7433,"sailed":7434,"doctors":7435,"oscar":7436,"joan":7437,"treat":7438,"garcia":7439,"harbour":7440,"jeremy":7441,"##ire":7442,"traditions":7443,"dominant":7444,"jacques":7445,"##gon":7446,"##wan":7447,"relocated":7448,"1879":7449,"amendment":7450,"sized":7451,"companion":7452,"simultaneously":7453,"volleyball":7454,"spun":7455,"acre":7456,"increases":7457,"stopping":7458,"loves":7459,"belongs":7460,"affect":7461,"drafted":7462,"tossed":7463,"scout":7464,"battles":7465,"1875":7466,"filming":7467,"shoved":7468,"munich":7469,"tenure":7470,"vertical":7471,"romance":7472,"pc":7473,"##cher":7474,"argue":7475,"##ical":7476,"craft":7477,"ranging":7478,"www":7479,"opens":7480,"honest":7481,"tyler":7482,"yesterday":7483,"virtual":7484,"##let":7485,"muslims":7486,"reveal":7487,"snake":7488,"immigrants":7489,"radical":7490,"screaming":7491,"speakers":7492,"firing":7493,"saving":7494,"belonging":7495,"ease":7496,"lighting":7497,"prefecture":7498,"blame":7499,"farmer":7500,"hungry":7501,"grows":7502,"rubbed":7503,"beam":7504,"sur":7505,"subsidiary":7506,"##cha":7507,"armenian":7508,"sao":7509,"dropping":7510,"conventional":7511,"##fer":7512,"microsoft":7513,"reply":7514,"qualify":7515,"spots":7516,"1867":7517,"sweat":7518,"festivals":7519,"##ken":7520,"immigration":7521,"physician":7522,"discover":7523,"exposure":7524,"sandy":7525,"explanation":7526,"isaac":7527,"implemented":7528,"##fish":7529,"hart":7530,"initiated":7531,"connect":7532,"stakes":7533,"presents":7534,"heights":7535,"householder":7536,"pleased":7537,"tourist":7538,"regardless":7539,"slip":7540,"closest":7541,"##ction":7542,"surely":7543,"sultan":7544,"brings":7545,"riley":7546,"preparation":7547,"aboard":7548,"slammed":7549,"baptist":7550,"experiment":7551,"ongoing":7552,"interstate":7553,"organic":7554,"playoffs":7555,"##ika":7556,"1877":7557,"130":7558,"##tar":7559,"hindu":7560,"error":7561,"tours":7562,"tier":7563,"plenty":7564,"arrangements":7565,"talks":7566,"trapped":7567,"excited":7568,"sank":7569,"ho":7570,"athens":7571,"1872":7572,"denver":7573,"welfare":7574,"suburb":7575,"athletes":7576,"trick":7577,"diverse":7578,"belly":7579,"exclusively":7580,"yelled":7581,"1868":7582,"##med":7583,"conversion":7584,"##ette":7585,"1874":7586,"internationally":7587,"computers":7588,"conductor":7589,"abilities":7590,"sensitive":7591,"hello":7592,"dispute":7593,"measured":7594,"globe":7595,"rocket":7596,"prices":7597,"amsterdam":7598,"flights":7599,"tigers":7600,"inn":7601,"municipalities":7602,"emotion":7603,"references":7604,"3d":7605,"##mus":7606,"explains":7607,"airlines":7608,"manufactured":7609,"pm":7610,"archaeological":7611,"1873":7612,"interpretation":7613,"devon":7614,"comment":7615,"##ites":7616,"settlements":7617,"kissing":7618,"absolute":7619,"improvement":7620,"suite":7621,"impressed":7622,"barcelona":7623,"sullivan":7624,"jefferson":7625,"towers":7626,"jesse":7627,"julie":7628,"##tin":7629,"##lu":7630,"grandson":7631,"hi":7632,"gauge":7633,"regard":7634,"rings":7635,"interviews":7636,"trace":7637,"raymond":7638,"thumb":7639,"departments":7640,"burns":7641,"serial":7642,"bulgarian":7643,"scores":7644,"demonstrated":7645,"##ix":7646,"1866":7647,"kyle":7648,"alberta":7649,"underneath":7650,"romanized":7651,"##ward":7652,"relieved":7653,"acquisition":7654,"phrase":7655,"cliff":7656,"reveals":7657,"han":7658,"cuts":7659,"merger":7660,"custom":7661,"##dar":7662,"nee":7663,"gilbert":7664,"graduation":7665,"##nts":7666,"assessment":7667,"cafe":7668,"difficulty":7669,"demands":7670,"swung":7671,"democrat":7672,"jennifer":7673,"commons":7674,"1940s":7675,"grove":7676,"##yo":7677,"completing":7678,"focuses":7679,"sum":7680,"substitute":7681,"bearing":7682,"stretch":7683,"reception":7684,"##py":7685,"reflected":7686,"essentially":7687,"destination":7688,"pairs":7689,"##ched":7690,"survival":7691,"resource":7692,"##bach":7693,"promoting":7694,"doubles":7695,"messages":7696,"tear":7697,"##down":7698,"##fully":7699,"parade":7700,"florence":7701,"harvey":7702,"incumbent":7703,"partial":7704,"framework":7705,"900":7706,"pedro":7707,"frozen":7708,"procedure":7709,"olivia":7710,"controls":7711,"##mic":7712,"shelter":7713,"personally":7714,"temperatures":7715,"##od":7716,"brisbane":7717,"tested":7718,"sits":7719,"marble":7720,"comprehensive":7721,"oxygen":7722,"leonard":7723,"##kov":7724,"inaugural":7725,"iranian":7726,"referring":7727,"quarters":7728,"attitude":7729,"##ivity":7730,"mainstream":7731,"lined":7732,"mars":7733,"dakota":7734,"norfolk":7735,"unsuccessful":7736,"##°":7737,"explosion":7738,"helicopter":7739,"congressional":7740,"##sing":7741,"inspector":7742,"bitch":7743,"seal":7744,"departed":7745,"divine":7746,"##ters":7747,"coaching":7748,"examination":7749,"punishment":7750,"manufacturer":7751,"sink":7752,"columns":7753,"unincorporated":7754,"signals":7755,"nevada":7756,"squeezed":7757,"dylan":7758,"dining":7759,"photos":7760,"martial":7761,"manuel":7762,"eighteen":7763,"elevator":7764,"brushed":7765,"plates":7766,"ministers":7767,"ivy":7768,"congregation":7769,"##len":7770,"slept":7771,"specialized":7772,"taxes":7773,"curve":7774,"restricted":7775,"negotiations":7776,"likes":7777,"statistical":7778,"arnold":7779,"inspiration":7780,"execution":7781,"bold":7782,"intermediate":7783,"significance":7784,"margin":7785,"ruler":7786,"wheels":7787,"gothic":7788,"intellectual":7789,"dependent":7790,"listened":7791,"eligible":7792,"buses":7793,"widow":7794,"syria":7795,"earn":7796,"cincinnati":7797,"collapsed":7798,"recipient":7799,"secrets":7800,"accessible":7801,"philippine":7802,"maritime":7803,"goddess":7804,"clerk":7805,"surrender":7806,"breaks":7807,"playoff":7808,"database":7809,"##ified":7810,"##lon":7811,"ideal":7812,"beetle":7813,"aspect":7814,"soap":7815,"regulation":7816,"strings":7817,"expand":7818,"anglo":7819,"shorter":7820,"crosses":7821,"retreat":7822,"tough":7823,"coins":7824,"wallace":7825,"directions":7826,"pressing":7827,"##oon":7828,"shipping":7829,"locomotives":7830,"comparison":7831,"topics":7832,"nephew":7833,"##mes":7834,"distinction":7835,"honors":7836,"travelled":7837,"sierra":7838,"ibn":7839,"##over":7840,"fortress":7841,"sa":7842,"recognised":7843,"carved":7844,"1869":7845,"clients":7846,"##dan":7847,"intent":7848,"##mar":7849,"coaches":7850,"describing":7851,"bread":7852,"##ington":7853,"beaten":7854,"northwestern":7855,"##ona":7856,"merit":7857,"youtube":7858,"collapse":7859,"challenges":7860,"em":7861,"historians":7862,"objective":7863,"submitted":7864,"virus":7865,"attacking":7866,"drake":7867,"assume":7868,"##ere":7869,"diseases":7870,"marc":7871,"stem":7872,"leeds":7873,"##cus":7874,"##ab":7875,"farming":7876,"glasses":7877,"##lock":7878,"visits":7879,"nowhere":7880,"fellowship":7881,"relevant":7882,"carries":7883,"restaurants":7884,"experiments":7885,"101":7886,"constantly":7887,"bases":7888,"targets":7889,"shah":7890,"tenth":7891,"opponents":7892,"verse":7893,"territorial":7894,"##ira":7895,"writings":7896,"corruption":7897,"##hs":7898,"instruction":7899,"inherited":7900,"reverse":7901,"emphasis":7902,"##vic":7903,"employee":7904,"arch":7905,"keeps":7906,"rabbi":7907,"watson":7908,"payment":7909,"uh":7910,"##ala":7911,"nancy":7912,"##tre":7913,"venice":7914,"fastest":7915,"sexy":7916,"banned":7917,"adrian":7918,"properly":7919,"ruth":7920,"touchdown":7921,"dollar":7922,"boards":7923,"metre":7924,"circles":7925,"edges":7926,"favour":7927,"comments":7928,"ok":7929,"travels":7930,"liberation":7931,"scattered":7932,"firmly":7933,"##ular":7934,"holland":7935,"permitted":7936,"diesel":7937,"kenya":7938,"den":7939,"originated":7940,"##ral":7941,"demons":7942,"resumed":7943,"dragged":7944,"rider":7945,"##rus":7946,"servant":7947,"blinked":7948,"extend":7949,"torn":7950,"##ias":7951,"##sey":7952,"input":7953,"meal":7954,"everybody":7955,"cylinder":7956,"kinds":7957,"camps":7958,"##fe":7959,"bullet":7960,"logic":7961,"##wn":7962,"croatian":7963,"evolved":7964,"healthy":7965,"fool":7966,"chocolate":7967,"wise":7968,"preserve":7969,"pradesh":7970,"##ess":7971,"respective":7972,"1850":7973,"##ew":7974,"chicken":7975,"artificial":7976,"gross":7977,"corresponding":7978,"convicted":7979,"cage":7980,"caroline":7981,"dialogue":7982,"##dor":7983,"narrative":7984,"stranger":7985,"mario":7986,"br":7987,"christianity":7988,"failing":7989,"trent":7990,"commanding":7991,"buddhist":7992,"1848":7993,"maurice":7994,"focusing":7995,"yale":7996,"bike":7997,"altitude":7998,"##ering":7999,"mouse":8000,"revised":8001,"##sley":8002,"veteran":8003,"##ig":8004,"pulls":8005,"theology":8006,"crashed":8007,"campaigns":8008,"legion":8009,"##ability":8010,"drag":8011,"excellence":8012,"customer":8013,"cancelled":8014,"intensity":8015,"excuse":8016,"##lar":8017,"liga":8018,"participating":8019,"contributing":8020,"printing":8021,"##burn":8022,"variable":8023,"##rk":8024,"curious":8025,"bin":8026,"legacy":8027,"renaissance":8028,"##my":8029,"symptoms":8030,"binding":8031,"vocalist":8032,"dancer":8033,"##nie":8034,"grammar":8035,"gospel":8036,"democrats":8037,"ya":8038,"enters":8039,"sc":8040,"diplomatic":8041,"hitler":8042,"##ser":8043,"clouds":8044,"mathematical":8045,"quit":8046,"defended":8047,"oriented":8048,"##heim":8049,"fundamental":8050,"hardware":8051,"impressive":8052,"equally":8053,"convince":8054,"confederate":8055,"guilt":8056,"chuck":8057,"sliding":8058,"##ware":8059,"magnetic":8060,"narrowed":8061,"petersburg":8062,"bulgaria":8063,"otto":8064,"phd":8065,"skill":8066,"##ama":8067,"reader":8068,"hopes":8069,"pitcher":8070,"reservoir":8071,"hearts":8072,"automatically":8073,"expecting":8074,"mysterious":8075,"bennett":8076,"extensively":8077,"imagined":8078,"seeds":8079,"monitor":8080,"fix":8081,"##ative":8082,"journalism":8083,"struggling":8084,"signature":8085,"ranch":8086,"encounter":8087,"photographer":8088,"observation":8089,"protests":8090,"##pin":8091,"influences":8092,"##hr":8093,"calendar":8094,"##all":8095,"cruz":8096,"croatia":8097,"locomotive":8098,"hughes":8099,"naturally":8100,"shakespeare":8101,"basement":8102,"hook":8103,"uncredited":8104,"faded":8105,"theories":8106,"approaches":8107,"dare":8108,"phillips":8109,"filling":8110,"fury":8111,"obama":8112,"##ain":8113,"efficient":8114,"arc":8115,"deliver":8116,"min":8117,"raid":8118,"breeding":8119,"inducted":8120,"leagues":8121,"efficiency":8122,"axis":8123,"montana":8124,"eagles":8125,"##ked":8126,"supplied":8127,"instructions":8128,"karen":8129,"picking":8130,"indicating":8131,"trap":8132,"anchor":8133,"practically":8134,"christians":8135,"tomb":8136,"vary":8137,"occasional":8138,"electronics":8139,"lords":8140,"readers":8141,"newcastle":8142,"faint":8143,"innovation":8144,"collect":8145,"situations":8146,"engagement":8147,"160":8148,"claude":8149,"mixture":8150,"##feld":8151,"peer":8152,"tissue":8153,"logo":8154,"lean":8155,"##ration":8156,"°f":8157,"floors":8158,"##ven":8159,"architects":8160,"reducing":8161,"##our":8162,"##ments":8163,"rope":8164,"1859":8165,"ottawa":8166,"##har":8167,"samples":8168,"banking":8169,"declaration":8170,"proteins":8171,"resignation":8172,"francois":8173,"saudi":8174,"advocate":8175,"exhibited":8176,"armor":8177,"twins":8178,"divorce":8179,"##ras":8180,"abraham":8181,"reviewed":8182,"jo":8183,"temporarily":8184,"matrix":8185,"physically":8186,"pulse":8187,"curled":8188,"##ena":8189,"difficulties":8190,"bengal":8191,"usage":8192,"##ban":8193,"annie":8194,"riders":8195,"certificate":8196,"##pi":8197,"holes":8198,"warsaw":8199,"distinctive":8200,"jessica":8201,"##mon":8202,"mutual":8203,"1857":8204,"customs":8205,"circular":8206,"eugene":8207,"removal":8208,"loaded":8209,"mere":8210,"vulnerable":8211,"depicted":8212,"generations":8213,"dame":8214,"heir":8215,"enormous":8216,"lightly":8217,"climbing":8218,"pitched":8219,"lessons":8220,"pilots":8221,"nepal":8222,"ram":8223,"google":8224,"preparing":8225,"brad":8226,"louise":8227,"renowned":8228,"##₂":8229,"liam":8230,"##ably":8231,"plaza":8232,"shaw":8233,"sophie":8234,"brilliant":8235,"bills":8236,"##bar":8237,"##nik":8238,"fucking":8239,"mainland":8240,"server":8241,"pleasant":8242,"seized":8243,"veterans":8244,"jerked":8245,"fail":8246,"beta":8247,"brush":8248,"radiation":8249,"stored":8250,"warmth":8251,"southeastern":8252,"nate":8253,"sin":8254,"raced":8255,"berkeley":8256,"joke":8257,"athlete":8258,"designation":8259,"trunk":8260,"##low":8261,"roland":8262,"qualification":8263,"archives":8264,"heels":8265,"artwork":8266,"receives":8267,"judicial":8268,"reserves":8269,"##bed":8270,"woke":8271,"installation":8272,"abu":8273,"floating":8274,"fake":8275,"lesser":8276,"excitement":8277,"interface":8278,"concentrated":8279,"addressed":8280,"characteristic":8281,"amanda":8282,"saxophone":8283,"monk":8284,"auto":8285,"##bus":8286,"releasing":8287,"egg":8288,"dies":8289,"interaction":8290,"defender":8291,"ce":8292,"outbreak":8293,"glory":8294,"loving":8295,"##bert":8296,"sequel":8297,"consciousness":8298,"http":8299,"awake":8300,"ski":8301,"enrolled":8302,"##ress":8303,"handling":8304,"rookie":8305,"brow":8306,"somebody":8307,"biography":8308,"warfare":8309,"amounts":8310,"contracts":8311,"presentation":8312,"fabric":8313,"dissolved":8314,"challenged":8315,"meter":8316,"psychological":8317,"lt":8318,"elevated":8319,"rally":8320,"accurate":8321,"##tha":8322,"hospitals":8323,"undergraduate":8324,"specialist":8325,"venezuela":8326,"exhibit":8327,"shed":8328,"nursing":8329,"protestant":8330,"fluid":8331,"structural":8332,"footage":8333,"jared":8334,"consistent":8335,"prey":8336,"##ska":8337,"succession":8338,"reflect":8339,"exile":8340,"lebanon":8341,"wiped":8342,"suspect":8343,"shanghai":8344,"resting":8345,"integration":8346,"preservation":8347,"marvel":8348,"variant":8349,"pirates":8350,"sheep":8351,"rounded":8352,"capita":8353,"sailing":8354,"colonies":8355,"manuscript":8356,"deemed":8357,"variations":8358,"clarke":8359,"functional":8360,"emerging":8361,"boxing":8362,"relaxed":8363,"curse":8364,"azerbaijan":8365,"heavyweight":8366,"nickname":8367,"editorial":8368,"rang":8369,"grid":8370,"tightened":8371,"earthquake":8372,"flashed":8373,"miguel":8374,"rushing":8375,"##ches":8376,"improvements":8377,"boxes":8378,"brooks":8379,"180":8380,"consumption":8381,"molecular":8382,"felix":8383,"societies":8384,"repeatedly":8385,"variation":8386,"aids":8387,"civic":8388,"graphics":8389,"professionals":8390,"realm":8391,"autonomous":8392,"receiver":8393,"delayed":8394,"workshop":8395,"militia":8396,"chairs":8397,"trump":8398,"canyon":8399,"##point":8400,"harsh":8401,"extending":8402,"lovely":8403,"happiness":8404,"##jan":8405,"stake":8406,"eyebrows":8407,"embassy":8408,"wellington":8409,"hannah":8410,"##ella":8411,"sony":8412,"corners":8413,"bishops":8414,"swear":8415,"cloth":8416,"contents":8417,"xi":8418,"namely":8419,"commenced":8420,"1854":8421,"stanford":8422,"nashville":8423,"courage":8424,"graphic":8425,"commitment":8426,"garrison":8427,"##bin":8428,"hamlet":8429,"clearing":8430,"rebels":8431,"attraction":8432,"literacy":8433,"cooking":8434,"ruins":8435,"temples":8436,"jenny":8437,"humanity":8438,"celebrate":8439,"hasn":8440,"freight":8441,"sixty":8442,"rebel":8443,"bastard":8444,"##art":8445,"newton":8446,"##ada":8447,"deer":8448,"##ges":8449,"##ching":8450,"smiles":8451,"delaware":8452,"singers":8453,"##ets":8454,"approaching":8455,"assists":8456,"flame":8457,"##ph":8458,"boulevard":8459,"barrel":8460,"planted":8461,"##ome":8462,"pursuit":8463,"##sia":8464,"consequences":8465,"posts":8466,"shallow":8467,"invitation":8468,"rode":8469,"depot":8470,"ernest":8471,"kane":8472,"rod":8473,"concepts":8474,"preston":8475,"topic":8476,"chambers":8477,"striking":8478,"blast":8479,"arrives":8480,"descendants":8481,"montgomery":8482,"ranges":8483,"worlds":8484,"##lay":8485,"##ari":8486,"span":8487,"chaos":8488,"praise":8489,"##ag":8490,"fewer":8491,"1855":8492,"sanctuary":8493,"mud":8494,"fbi":8495,"##ions":8496,"programmes":8497,"maintaining":8498,"unity":8499,"harper":8500,"bore":8501,"handsome":8502,"closure":8503,"tournaments":8504,"thunder":8505,"nebraska":8506,"linda":8507,"facade":8508,"puts":8509,"satisfied":8510,"argentine":8511,"dale":8512,"cork":8513,"dome":8514,"panama":8515,"##yl":8516,"1858":8517,"tasks":8518,"experts":8519,"##ates":8520,"feeding":8521,"equation":8522,"##las":8523,"##ida":8524,"##tu":8525,"engage":8526,"bryan":8527,"##ax":8528,"um":8529,"quartet":8530,"melody":8531,"disbanded":8532,"sheffield":8533,"blocked":8534,"gasped":8535,"delay":8536,"kisses":8537,"maggie":8538,"connects":8539,"##non":8540,"sts":8541,"poured":8542,"creator":8543,"publishers":8544,"##we":8545,"guided":8546,"ellis":8547,"extinct":8548,"hug":8549,"gaining":8550,"##ord":8551,"complicated":8552,"##bility":8553,"poll":8554,"clenched":8555,"investigate":8556,"##use":8557,"thereby":8558,"quantum":8559,"spine":8560,"cdp":8561,"humor":8562,"kills":8563,"administered":8564,"semifinals":8565,"##du":8566,"encountered":8567,"ignore":8568,"##bu":8569,"commentary":8570,"##maker":8571,"bother":8572,"roosevelt":8573,"140":8574,"plains":8575,"halfway":8576,"flowing":8577,"cultures":8578,"crack":8579,"imprisoned":8580,"neighboring":8581,"airline":8582,"##ses":8583,"##view":8584,"##mate":8585,"##ec":8586,"gather":8587,"wolves":8588,"marathon":8589,"transformed":8590,"##ill":8591,"cruise":8592,"organisations":8593,"carol":8594,"punch":8595,"exhibitions":8596,"numbered":8597,"alarm":8598,"ratings":8599,"daddy":8600,"silently":8601,"##stein":8602,"queens":8603,"colours":8604,"impression":8605,"guidance":8606,"liu":8607,"tactical":8608,"##rat":8609,"marshal":8610,"della":8611,"arrow":8612,"##ings":8613,"rested":8614,"feared":8615,"tender":8616,"owns":8617,"bitter":8618,"advisor":8619,"escort":8620,"##ides":8621,"spare":8622,"farms":8623,"grants":8624,"##ene":8625,"dragons":8626,"encourage":8627,"colleagues":8628,"cameras":8629,"##und":8630,"sucked":8631,"pile":8632,"spirits":8633,"prague":8634,"statements":8635,"suspension":8636,"landmark":8637,"fence":8638,"torture":8639,"recreation":8640,"bags":8641,"permanently":8642,"survivors":8643,"pond":8644,"spy":8645,"predecessor":8646,"bombing":8647,"coup":8648,"##og":8649,"protecting":8650,"transformation":8651,"glow":8652,"##lands":8653,"##book":8654,"dug":8655,"priests":8656,"andrea":8657,"feat":8658,"barn":8659,"jumping":8660,"##chen":8661,"##ologist":8662,"##con":8663,"casualties":8664,"stern":8665,"auckland":8666,"pipe":8667,"serie":8668,"revealing":8669,"ba":8670,"##bel":8671,"trevor":8672,"mercy":8673,"spectrum":8674,"yang":8675,"consist":8676,"governing":8677,"collaborated":8678,"possessed":8679,"epic":8680,"comprises":8681,"blew":8682,"shane":8683,"##ack":8684,"lopez":8685,"honored":8686,"magical":8687,"sacrifice":8688,"judgment":8689,"perceived":8690,"hammer":8691,"mtv":8692,"baronet":8693,"tune":8694,"das":8695,"missionary":8696,"sheets":8697,"350":8698,"neutral":8699,"oral":8700,"threatening":8701,"attractive":8702,"shade":8703,"aims":8704,"seminary":8705,"##master":8706,"estates":8707,"1856":8708,"michel":8709,"wounds":8710,"refugees":8711,"manufacturers":8712,"##nic":8713,"mercury":8714,"syndrome":8715,"porter":8716,"##iya":8717,"##din":8718,"hamburg":8719,"identification":8720,"upstairs":8721,"purse":8722,"widened":8723,"pause":8724,"cared":8725,"breathed":8726,"affiliate":8727,"santiago":8728,"prevented":8729,"celtic":8730,"fisher":8731,"125":8732,"recruited":8733,"byzantine":8734,"reconstruction":8735,"farther":8736,"##mp":8737,"diet":8738,"sake":8739,"au":8740,"spite":8741,"sensation":8742,"##ert":8743,"blank":8744,"separation":8745,"105":8746,"##hon":8747,"vladimir":8748,"armies":8749,"anime":8750,"##lie":8751,"accommodate":8752,"orbit":8753,"cult":8754,"sofia":8755,"archive":8756,"##ify":8757,"##box":8758,"founders":8759,"sustained":8760,"disorder":8761,"honours":8762,"northeastern":8763,"mia":8764,"crops":8765,"violet":8766,"threats":8767,"blanket":8768,"fires":8769,"canton":8770,"followers":8771,"southwestern":8772,"prototype":8773,"voyage":8774,"assignment":8775,"altered":8776,"moderate":8777,"protocol":8778,"pistol":8779,"##eo":8780,"questioned":8781,"brass":8782,"lifting":8783,"1852":8784,"math":8785,"authored":8786,"##ual":8787,"doug":8788,"dimensional":8789,"dynamic":8790,"##san":8791,"1851":8792,"pronounced":8793,"grateful":8794,"quest":8795,"uncomfortable":8796,"boom":8797,"presidency":8798,"stevens":8799,"relating":8800,"politicians":8801,"chen":8802,"barrier":8803,"quinn":8804,"diana":8805,"mosque":8806,"tribal":8807,"cheese":8808,"palmer":8809,"portions":8810,"sometime":8811,"chester":8812,"treasure":8813,"wu":8814,"bend":8815,"download":8816,"millions":8817,"reforms":8818,"registration":8819,"##osa":8820,"consequently":8821,"monitoring":8822,"ate":8823,"preliminary":8824,"brandon":8825,"invented":8826,"ps":8827,"eaten":8828,"exterior":8829,"intervention":8830,"ports":8831,"documented":8832,"log":8833,"displays":8834,"lecture":8835,"sally":8836,"favourite":8837,"##itz":8838,"vermont":8839,"lo":8840,"invisible":8841,"isle":8842,"breed":8843,"##ator":8844,"journalists":8845,"relay":8846,"speaks":8847,"backward":8848,"explore":8849,"midfielder":8850,"actively":8851,"stefan":8852,"procedures":8853,"cannon":8854,"blond":8855,"kenneth":8856,"centered":8857,"servants":8858,"chains":8859,"libraries":8860,"malcolm":8861,"essex":8862,"henri":8863,"slavery":8864,"##hal":8865,"facts":8866,"fairy":8867,"coached":8868,"cassie":8869,"cats":8870,"washed":8871,"cop":8872,"##fi":8873,"announcement":8874,"item":8875,"2000s":8876,"vinyl":8877,"activated":8878,"marco":8879,"frontier":8880,"growled":8881,"curriculum":8882,"##das":8883,"loyal":8884,"accomplished":8885,"leslie":8886,"ritual":8887,"kenny":8888,"##00":8889,"vii":8890,"napoleon":8891,"hollow":8892,"hybrid":8893,"jungle":8894,"stationed":8895,"friedrich":8896,"counted":8897,"##ulated":8898,"platinum":8899,"theatrical":8900,"seated":8901,"col":8902,"rubber":8903,"glen":8904,"1840":8905,"diversity":8906,"healing":8907,"extends":8908,"id":8909,"provisions":8910,"administrator":8911,"columbus":8912,"##oe":8913,"tributary":8914,"te":8915,"assured":8916,"org":8917,"##uous":8918,"prestigious":8919,"examined":8920,"lectures":8921,"grammy":8922,"ronald":8923,"associations":8924,"bailey":8925,"allan":8926,"essays":8927,"flute":8928,"believing":8929,"consultant":8930,"proceedings":8931,"travelling":8932,"1853":8933,"kit":8934,"kerala":8935,"yugoslavia":8936,"buddy":8937,"methodist":8938,"##ith":8939,"burial":8940,"centres":8941,"batman":8942,"##nda":8943,"discontinued":8944,"bo":8945,"dock":8946,"stockholm":8947,"lungs":8948,"severely":8949,"##nk":8950,"citing":8951,"manga":8952,"##ugh":8953,"steal":8954,"mumbai":8955,"iraqi":8956,"robot":8957,"celebrity":8958,"bride":8959,"broadcasts":8960,"abolished":8961,"pot":8962,"joel":8963,"overhead":8964,"franz":8965,"packed":8966,"reconnaissance":8967,"johann":8968,"acknowledged":8969,"introduce":8970,"handled":8971,"doctorate":8972,"developments":8973,"drinks":8974,"alley":8975,"palestine":8976,"##nis":8977,"##aki":8978,"proceeded":8979,"recover":8980,"bradley":8981,"grain":8982,"patch":8983,"afford":8984,"infection":8985,"nationalist":8986,"legendary":8987,"##ath":8988,"interchange":8989,"virtually":8990,"gen":8991,"gravity":8992,"exploration":8993,"amber":8994,"vital":8995,"wishes":8996,"powell":8997,"doctrine":8998,"elbow":8999,"screenplay":9000,"##bird":9001,"contribute":9002,"indonesian":9003,"pet":9004,"creates":9005,"##com":9006,"enzyme":9007,"kylie":9008,"discipline":9009,"drops":9010,"manila":9011,"hunger":9012,"##ien":9013,"layers":9014,"suffer":9015,"fever":9016,"bits":9017,"monica":9018,"keyboard":9019,"manages":9020,"##hood":9021,"searched":9022,"appeals":9023,"##bad":9024,"testament":9025,"grande":9026,"reid":9027,"##war":9028,"beliefs":9029,"congo":9030,"##ification":9031,"##dia":9032,"si":9033,"requiring":9034,"##via":9035,"casey":9036,"1849":9037,"regret":9038,"streak":9039,"rape":9040,"depends":9041,"syrian":9042,"sprint":9043,"pound":9044,"tourists":9045,"upcoming":9046,"pub":9047,"##xi":9048,"tense":9049,"##els":9050,"practiced":9051,"echo":9052,"nationwide":9053,"guild":9054,"motorcycle":9055,"liz":9056,"##zar":9057,"chiefs":9058,"desired":9059,"elena":9060,"bye":9061,"precious":9062,"absorbed":9063,"relatives":9064,"booth":9065,"pianist":9066,"##mal":9067,"citizenship":9068,"exhausted":9069,"wilhelm":9070,"##ceae":9071,"##hed":9072,"noting":9073,"quarterback":9074,"urge":9075,"hectares":9076,"##gue":9077,"ace":9078,"holly":9079,"##tal":9080,"blonde":9081,"davies":9082,"parked":9083,"sustainable":9084,"stepping":9085,"twentieth":9086,"airfield":9087,"galaxy":9088,"nest":9089,"chip":9090,"##nell":9091,"tan":9092,"shaft":9093,"paulo":9094,"requirement":9095,"##zy":9096,"paradise":9097,"tobacco":9098,"trans":9099,"renewed":9100,"vietnamese":9101,"##cker":9102,"##ju":9103,"suggesting":9104,"catching":9105,"holmes":9106,"enjoying":9107,"md":9108,"trips":9109,"colt":9110,"holder":9111,"butterfly":9112,"nerve":9113,"reformed":9114,"cherry":9115,"bowling":9116,"trailer":9117,"carriage":9118,"goodbye":9119,"appreciate":9120,"toy":9121,"joshua":9122,"interactive":9123,"enabled":9124,"involve":9125,"##kan":9126,"collar":9127,"determination":9128,"bunch":9129,"facebook":9130,"recall":9131,"shorts":9132,"superintendent":9133,"episcopal":9134,"frustration":9135,"giovanni":9136,"nineteenth":9137,"laser":9138,"privately":9139,"array":9140,"circulation":9141,"##ovic":9142,"armstrong":9143,"deals":9144,"painful":9145,"permit":9146,"discrimination":9147,"##wi":9148,"aires":9149,"retiring":9150,"cottage":9151,"ni":9152,"##sta":9153,"horizon":9154,"ellen":9155,"jamaica":9156,"ripped":9157,"fernando":9158,"chapters":9159,"playstation":9160,"patron":9161,"lecturer":9162,"navigation":9163,"behaviour":9164,"genes":9165,"georgian":9166,"export":9167,"solomon":9168,"rivals":9169,"swift":9170,"seventeen":9171,"rodriguez":9172,"princeton":9173,"independently":9174,"sox":9175,"1847":9176,"arguing":9177,"entity":9178,"casting":9179,"hank":9180,"criteria":9181,"oakland":9182,"geographic":9183,"milwaukee":9184,"reflection":9185,"expanding":9186,"conquest":9187,"dubbed":9188,"##tv":9189,"halt":9190,"brave":9191,"brunswick":9192,"doi":9193,"arched":9194,"curtis":9195,"divorced":9196,"predominantly":9197,"somerset":9198,"streams":9199,"ugly":9200,"zoo":9201,"horrible":9202,"curved":9203,"buenos":9204,"fierce":9205,"dictionary":9206,"vector":9207,"theological":9208,"unions":9209,"handful":9210,"stability":9211,"chan":9212,"punjab":9213,"segments":9214,"##lly":9215,"altar":9216,"ignoring":9217,"gesture":9218,"monsters":9219,"pastor":9220,"##stone":9221,"thighs":9222,"unexpected":9223,"operators":9224,"abruptly":9225,"coin":9226,"compiled":9227,"associates":9228,"improving":9229,"migration":9230,"pin":9231,"##ose":9232,"compact":9233,"collegiate":9234,"reserved":9235,"##urs":9236,"quarterfinals":9237,"roster":9238,"restore":9239,"assembled":9240,"hurry":9241,"oval":9242,"##cies":9243,"1846":9244,"flags":9245,"martha":9246,"##del":9247,"victories":9248,"sharply":9249,"##rated":9250,"argues":9251,"deadly":9252,"neo":9253,"drawings":9254,"symbols":9255,"performer":9256,"##iel":9257,"griffin":9258,"restrictions":9259,"editing":9260,"andrews":9261,"java":9262,"journals":9263,"arabia":9264,"compositions":9265,"dee":9266,"pierce":9267,"removing":9268,"hindi":9269,"casino":9270,"runway":9271,"civilians":9272,"minds":9273,"nasa":9274,"hotels":9275,"##zation":9276,"refuge":9277,"rent":9278,"retain":9279,"potentially":9280,"conferences":9281,"suburban":9282,"conducting":9283,"##tto":9284,"##tions":9285,"##tle":9286,"descended":9287,"massacre":9288,"##cal":9289,"ammunition":9290,"terrain":9291,"fork":9292,"souls":9293,"counts":9294,"chelsea":9295,"durham":9296,"drives":9297,"cab":9298,"##bank":9299,"perth":9300,"realizing":9301,"palestinian":9302,"finn":9303,"simpson":9304,"##dal":9305,"betty":9306,"##ule":9307,"moreover":9308,"particles":9309,"cardinals":9310,"tent":9311,"evaluation":9312,"extraordinary":9313,"##oid":9314,"inscription":9315,"##works":9316,"wednesday":9317,"chloe":9318,"maintains":9319,"panels":9320,"ashley":9321,"trucks":9322,"##nation":9323,"cluster":9324,"sunlight":9325,"strikes":9326,"zhang":9327,"##wing":9328,"dialect":9329,"canon":9330,"##ap":9331,"tucked":9332,"##ws":9333,"collecting":9334,"##mas":9335,"##can":9336,"##sville":9337,"maker":9338,"quoted":9339,"evan":9340,"franco":9341,"aria":9342,"buying":9343,"cleaning":9344,"eva":9345,"closet":9346,"provision":9347,"apollo":9348,"clinic":9349,"rat":9350,"##ez":9351,"necessarily":9352,"ac":9353,"##gle":9354,"##ising":9355,"venues":9356,"flipped":9357,"cent":9358,"spreading":9359,"trustees":9360,"checking":9361,"authorized":9362,"##sco":9363,"disappointed":9364,"##ado":9365,"notion":9366,"duration":9367,"trumpet":9368,"hesitated":9369,"topped":9370,"brussels":9371,"rolls":9372,"theoretical":9373,"hint":9374,"define":9375,"aggressive":9376,"repeat":9377,"wash":9378,"peaceful":9379,"optical":9380,"width":9381,"allegedly":9382,"mcdonald":9383,"strict":9384,"copyright":9385,"##illa":9386,"investors":9387,"mar":9388,"jam":9389,"witnesses":9390,"sounding":9391,"miranda":9392,"michelle":9393,"privacy":9394,"hugo":9395,"harmony":9396,"##pp":9397,"valid":9398,"lynn":9399,"glared":9400,"nina":9401,"102":9402,"headquartered":9403,"diving":9404,"boarding":9405,"gibson":9406,"##ncy":9407,"albanian":9408,"marsh":9409,"routine":9410,"dealt":9411,"enhanced":9412,"er":9413,"intelligent":9414,"substance":9415,"targeted":9416,"enlisted":9417,"discovers":9418,"spinning":9419,"observations":9420,"pissed":9421,"smoking":9422,"rebecca":9423,"capitol":9424,"visa":9425,"varied":9426,"costume":9427,"seemingly":9428,"indies":9429,"compensation":9430,"surgeon":9431,"thursday":9432,"arsenal":9433,"westminster":9434,"suburbs":9435,"rid":9436,"anglican":9437,"##ridge":9438,"knots":9439,"foods":9440,"alumni":9441,"lighter":9442,"fraser":9443,"whoever":9444,"portal":9445,"scandal":9446,"##ray":9447,"gavin":9448,"advised":9449,"instructor":9450,"flooding":9451,"terrorist":9452,"##ale":9453,"teenage":9454,"interim":9455,"senses":9456,"duck":9457,"teen":9458,"thesis":9459,"abby":9460,"eager":9461,"overcome":9462,"##ile":9463,"newport":9464,"glenn":9465,"rises":9466,"shame":9467,"##cc":9468,"prompted":9469,"priority":9470,"forgot":9471,"bomber":9472,"nicolas":9473,"protective":9474,"360":9475,"cartoon":9476,"katherine":9477,"breeze":9478,"lonely":9479,"trusted":9480,"henderson":9481,"richardson":9482,"relax":9483,"banner":9484,"candy":9485,"palms":9486,"remarkable":9487,"##rio":9488,"legends":9489,"cricketer":9490,"essay":9491,"ordained":9492,"edmund":9493,"rifles":9494,"trigger":9495,"##uri":9496,"##away":9497,"sail":9498,"alert":9499,"1830":9500,"audiences":9501,"penn":9502,"sussex":9503,"siblings":9504,"pursued":9505,"indianapolis":9506,"resist":9507,"rosa":9508,"consequence":9509,"succeed":9510,"avoided":9511,"1845":9512,"##ulation":9513,"inland":9514,"##tie":9515,"##nna":9516,"counsel":9517,"profession":9518,"chronicle":9519,"hurried":9520,"##una":9521,"eyebrow":9522,"eventual":9523,"bleeding":9524,"innovative":9525,"cure":9526,"##dom":9527,"committees":9528,"accounting":9529,"con":9530,"scope":9531,"hardy":9532,"heather":9533,"tenor":9534,"gut":9535,"herald":9536,"codes":9537,"tore":9538,"scales":9539,"wagon":9540,"##oo":9541,"luxury":9542,"tin":9543,"prefer":9544,"fountain":9545,"triangle":9546,"bonds":9547,"darling":9548,"convoy":9549,"dried":9550,"traced":9551,"beings":9552,"troy":9553,"accidentally":9554,"slam":9555,"findings":9556,"smelled":9557,"joey":9558,"lawyers":9559,"outcome":9560,"steep":9561,"bosnia":9562,"configuration":9563,"shifting":9564,"toll":9565,"brook":9566,"performers":9567,"lobby":9568,"philosophical":9569,"construct":9570,"shrine":9571,"aggregate":9572,"boot":9573,"cox":9574,"phenomenon":9575,"savage":9576,"insane":9577,"solely":9578,"reynolds":9579,"lifestyle":9580,"##ima":9581,"nationally":9582,"holdings":9583,"consideration":9584,"enable":9585,"edgar":9586,"mo":9587,"mama":9588,"##tein":9589,"fights":9590,"relegation":9591,"chances":9592,"atomic":9593,"hub":9594,"conjunction":9595,"awkward":9596,"reactions":9597,"currency":9598,"finale":9599,"kumar":9600,"underwent":9601,"steering":9602,"elaborate":9603,"gifts":9604,"comprising":9605,"melissa":9606,"veins":9607,"reasonable":9608,"sunshine":9609,"chi":9610,"solve":9611,"trails":9612,"inhabited":9613,"elimination":9614,"ethics":9615,"huh":9616,"ana":9617,"molly":9618,"consent":9619,"apartments":9620,"layout":9621,"marines":9622,"##ces":9623,"hunters":9624,"bulk":9625,"##oma":9626,"hometown":9627,"##wall":9628,"##mont":9629,"cracked":9630,"reads":9631,"neighbouring":9632,"withdrawn":9633,"admission":9634,"wingspan":9635,"damned":9636,"anthology":9637,"lancashire":9638,"brands":9639,"batting":9640,"forgive":9641,"cuban":9642,"awful":9643,"##lyn":9644,"104":9645,"dimensions":9646,"imagination":9647,"##ade":9648,"dante":9649,"##ship":9650,"tracking":9651,"desperately":9652,"goalkeeper":9653,"##yne":9654,"groaned":9655,"workshops":9656,"confident":9657,"burton":9658,"gerald":9659,"milton":9660,"circus":9661,"uncertain":9662,"slope":9663,"copenhagen":9664,"sophia":9665,"fog":9666,"philosopher":9667,"portraits":9668,"accent":9669,"cycling":9670,"varying":9671,"gripped":9672,"larvae":9673,"garrett":9674,"specified":9675,"scotia":9676,"mature":9677,"luther":9678,"kurt":9679,"rap":9680,"##kes":9681,"aerial":9682,"750":9683,"ferdinand":9684,"heated":9685,"es":9686,"transported":9687,"##shan":9688,"safely":9689,"nonetheless":9690,"##orn":9691,"##gal":9692,"motors":9693,"demanding":9694,"##sburg":9695,"startled":9696,"##brook":9697,"ally":9698,"generate":9699,"caps":9700,"ghana":9701,"stained":9702,"demo":9703,"mentions":9704,"beds":9705,"ap":9706,"afterward":9707,"diary":9708,"##bling":9709,"utility":9710,"##iro":9711,"richards":9712,"1837":9713,"conspiracy":9714,"conscious":9715,"shining":9716,"footsteps":9717,"observer":9718,"cyprus":9719,"urged":9720,"loyalty":9721,"developer":9722,"probability":9723,"olive":9724,"upgraded":9725,"gym":9726,"miracle":9727,"insects":9728,"graves":9729,"1844":9730,"ourselves":9731,"hydrogen":9732,"amazon":9733,"katie":9734,"tickets":9735,"poets":9736,"##pm":9737,"planes":9738,"##pan":9739,"prevention":9740,"witnessed":9741,"dense":9742,"jin":9743,"randy":9744,"tang":9745,"warehouse":9746,"monroe":9747,"bang":9748,"archived":9749,"elderly":9750,"investigations":9751,"alec":9752,"granite":9753,"mineral":9754,"conflicts":9755,"controlling":9756,"aboriginal":9757,"carlo":9758,"##zu":9759,"mechanics":9760,"stan":9761,"stark":9762,"rhode":9763,"skirt":9764,"est":9765,"##berry":9766,"bombs":9767,"respected":9768,"##horn":9769,"imposed":9770,"limestone":9771,"deny":9772,"nominee":9773,"memphis":9774,"grabbing":9775,"disabled":9776,"##als":9777,"amusement":9778,"aa":9779,"frankfurt":9780,"corn":9781,"referendum":9782,"varies":9783,"slowed":9784,"disk":9785,"firms":9786,"unconscious":9787,"incredible":9788,"clue":9789,"sue":9790,"##zhou":9791,"twist":9792,"##cio":9793,"joins":9794,"idaho":9795,"chad":9796,"developers":9797,"computing":9798,"destroyer":9799,"103":9800,"mortal":9801,"tucker":9802,"kingston":9803,"choices":9804,"yu":9805,"carson":9806,"1800":9807,"os":9808,"whitney":9809,"geneva":9810,"pretend":9811,"dimension":9812,"staged":9813,"plateau":9814,"maya":9815,"##une":9816,"freestyle":9817,"##bc":9818,"rovers":9819,"hiv":9820,"##ids":9821,"tristan":9822,"classroom":9823,"prospect":9824,"##hus":9825,"honestly":9826,"diploma":9827,"lied":9828,"thermal":9829,"auxiliary":9830,"feast":9831,"unlikely":9832,"iata":9833,"##tel":9834,"morocco":9835,"pounding":9836,"treasury":9837,"lithuania":9838,"considerably":9839,"1841":9840,"dish":9841,"1812":9842,"geological":9843,"matching":9844,"stumbled":9845,"destroying":9846,"marched":9847,"brien":9848,"advances":9849,"cake":9850,"nicole":9851,"belle":9852,"settling":9853,"measuring":9854,"directing":9855,"##mie":9856,"tuesday":9857,"bassist":9858,"capabilities":9859,"stunned":9860,"fraud":9861,"torpedo":9862,"##list":9863,"##phone":9864,"anton":9865,"wisdom":9866,"surveillance":9867,"ruined":9868,"##ulate":9869,"lawsuit":9870,"healthcare":9871,"theorem":9872,"halls":9873,"trend":9874,"aka":9875,"horizontal":9876,"dozens":9877,"acquire":9878,"lasting":9879,"swim":9880,"hawk":9881,"gorgeous":9882,"fees":9883,"vicinity":9884,"decrease":9885,"adoption":9886,"tactics":9887,"##ography":9888,"pakistani":9889,"##ole":9890,"draws":9891,"##hall":9892,"willie":9893,"burke":9894,"heath":9895,"algorithm":9896,"integral":9897,"powder":9898,"elliott":9899,"brigadier":9900,"jackie":9901,"tate":9902,"varieties":9903,"darker":9904,"##cho":9905,"lately":9906,"cigarette":9907,"specimens":9908,"adds":9909,"##ree":9910,"##ensis":9911,"##inger":9912,"exploded":9913,"finalist":9914,"cia":9915,"murders":9916,"wilderness":9917,"arguments":9918,"nicknamed":9919,"acceptance":9920,"onwards":9921,"manufacture":9922,"robertson":9923,"jets":9924,"tampa":9925,"enterprises":9926,"blog":9927,"loudly":9928,"composers":9929,"nominations":9930,"1838":9931,"ai":9932,"malta":9933,"inquiry":9934,"automobile":9935,"hosting":9936,"viii":9937,"rays":9938,"tilted":9939,"grief":9940,"museums":9941,"strategies":9942,"furious":9943,"euro":9944,"equality":9945,"cohen":9946,"poison":9947,"surrey":9948,"wireless":9949,"governed":9950,"ridiculous":9951,"moses":9952,"##esh":9953,"##room":9954,"vanished":9955,"##ito":9956,"barnes":9957,"attract":9958,"morrison":9959,"istanbul":9960,"##iness":9961,"absent":9962,"rotation":9963,"petition":9964,"janet":9965,"##logical":9966,"satisfaction":9967,"custody":9968,"deliberately":9969,"observatory":9970,"comedian":9971,"surfaces":9972,"pinyin":9973,"novelist":9974,"strictly":9975,"canterbury":9976,"oslo":9977,"monks":9978,"embrace":9979,"ibm":9980,"jealous":9981,"photograph":9982,"continent":9983,"dorothy":9984,"marina":9985,"doc":9986,"excess":9987,"holden":9988,"allegations":9989,"explaining":9990,"stack":9991,"avoiding":9992,"lance":9993,"storyline":9994,"majesty":9995,"poorly":9996,"spike":9997,"dos":9998,"bradford":9999,"raven":10000,"travis":10001,"classics":10002,"proven":10003,"voltage":10004,"pillow":10005,"fists":10006,"butt":10007,"1842":10008,"interpreted":10009,"##car":10010,"1839":10011,"gage":10012,"telegraph":10013,"lens":10014,"promising":10015,"expelled":10016,"casual":10017,"collector":10018,"zones":10019,"##min":10020,"silly":10021,"nintendo":10022,"##kh":10023,"##bra":10024,"downstairs":10025,"chef":10026,"suspicious":10027,"afl":10028,"flies":10029,"vacant":10030,"uganda":10031,"pregnancy":10032,"condemned":10033,"lutheran":10034,"estimates":10035,"cheap":10036,"decree":10037,"saxon":10038,"proximity":10039,"stripped":10040,"idiot":10041,"deposits":10042,"contrary":10043,"presenter":10044,"magnus":10045,"glacier":10046,"im":10047,"offense":10048,"edwin":10049,"##ori":10050,"upright":10051,"##long":10052,"bolt":10053,"##ois":10054,"toss":10055,"geographical":10056,"##izes":10057,"environments":10058,"delicate":10059,"marking":10060,"abstract":10061,"xavier":10062,"nails":10063,"windsor":10064,"plantation":10065,"occurring":10066,"equity":10067,"saskatchewan":10068,"fears":10069,"drifted":10070,"sequences":10071,"vegetation":10072,"revolt":10073,"##stic":10074,"1843":10075,"sooner":10076,"fusion":10077,"opposing":10078,"nato":10079,"skating":10080,"1836":10081,"secretly":10082,"ruin":10083,"lease":10084,"##oc":10085,"edit":10086,"##nne":10087,"flora":10088,"anxiety":10089,"ruby":10090,"##ological":10091,"##mia":10092,"tel":10093,"bout":10094,"taxi":10095,"emmy":10096,"frost":10097,"rainbow":10098,"compounds":10099,"foundations":10100,"rainfall":10101,"assassination":10102,"nightmare":10103,"dominican":10104,"##win":10105,"achievements":10106,"deserve":10107,"orlando":10108,"intact":10109,"armenia":10110,"##nte":10111,"calgary":10112,"valentine":10113,"106":10114,"marion":10115,"proclaimed":10116,"theodore":10117,"bells":10118,"courtyard":10119,"thigh":10120,"gonzalez":10121,"console":10122,"troop":10123,"minimal":10124,"monte":10125,"everyday":10126,"##ence":10127,"##if":10128,"supporter":10129,"terrorism":10130,"buck":10131,"openly":10132,"presbyterian":10133,"activists":10134,"carpet":10135,"##iers":10136,"rubbing":10137,"uprising":10138,"##yi":10139,"cute":10140,"conceived":10141,"legally":10142,"##cht":10143,"millennium":10144,"cello":10145,"velocity":10146,"ji":10147,"rescued":10148,"cardiff":10149,"1835":10150,"rex":10151,"concentrate":10152,"senators":10153,"beard":10154,"rendered":10155,"glowing":10156,"battalions":10157,"scouts":10158,"competitors":10159,"sculptor":10160,"catalogue":10161,"arctic":10162,"ion":10163,"raja":10164,"bicycle":10165,"wow":10166,"glancing":10167,"lawn":10168,"##woman":10169,"gentleman":10170,"lighthouse":10171,"publish":10172,"predicted":10173,"calculated":10174,"##val":10175,"variants":10176,"##gne":10177,"strain":10178,"##ui":10179,"winston":10180,"deceased":10181,"##nus":10182,"touchdowns":10183,"brady":10184,"caleb":10185,"sinking":10186,"echoed":10187,"crush":10188,"hon":10189,"blessed":10190,"protagonist":10191,"hayes":10192,"endangered":10193,"magnitude":10194,"editors":10195,"##tine":10196,"estimate":10197,"responsibilities":10198,"##mel":10199,"backup":10200,"laying":10201,"consumed":10202,"sealed":10203,"zurich":10204,"lovers":10205,"frustrated":10206,"##eau":10207,"ahmed":10208,"kicking":10209,"mit":10210,"treasurer":10211,"1832":10212,"biblical":10213,"refuse":10214,"terrified":10215,"pump":10216,"agrees":10217,"genuine":10218,"imprisonment":10219,"refuses":10220,"plymouth":10221,"##hen":10222,"lou":10223,"##nen":10224,"tara":10225,"trembling":10226,"antarctic":10227,"ton":10228,"learns":10229,"##tas":10230,"crap":10231,"crucial":10232,"faction":10233,"atop":10234,"##borough":10235,"wrap":10236,"lancaster":10237,"odds":10238,"hopkins":10239,"erik":10240,"lyon":10241,"##eon":10242,"bros":10243,"##ode":10244,"snap":10245,"locality":10246,"tips":10247,"empress":10248,"crowned":10249,"cal":10250,"acclaimed":10251,"chuckled":10252,"##ory":10253,"clara":10254,"sends":10255,"mild":10256,"towel":10257,"##fl":10258,"##day":10259,"##а":10260,"wishing":10261,"assuming":10262,"interviewed":10263,"##bal":10264,"##die":10265,"interactions":10266,"eden":10267,"cups":10268,"helena":10269,"##lf":10270,"indie":10271,"beck":10272,"##fire":10273,"batteries":10274,"filipino":10275,"wizard":10276,"parted":10277,"##lam":10278,"traces":10279,"##born":10280,"rows":10281,"idol":10282,"albany":10283,"delegates":10284,"##ees":10285,"##sar":10286,"discussions":10287,"##ex":10288,"notre":10289,"instructed":10290,"belgrade":10291,"highways":10292,"suggestion":10293,"lauren":10294,"possess":10295,"orientation":10296,"alexandria":10297,"abdul":10298,"beats":10299,"salary":10300,"reunion":10301,"ludwig":10302,"alright":10303,"wagner":10304,"intimate":10305,"pockets":10306,"slovenia":10307,"hugged":10308,"brighton":10309,"merchants":10310,"cruel":10311,"stole":10312,"trek":10313,"slopes":10314,"repairs":10315,"enrollment":10316,"politically":10317,"underlying":10318,"promotional":10319,"counting":10320,"boeing":10321,"##bb":10322,"isabella":10323,"naming":10324,"##и":10325,"keen":10326,"bacteria":10327,"listing":10328,"separately":10329,"belfast":10330,"ussr":10331,"450":10332,"lithuanian":10333,"anybody":10334,"ribs":10335,"sphere":10336,"martinez":10337,"cock":10338,"embarrassed":10339,"proposals":10340,"fragments":10341,"nationals":10342,"##fs":10343,"##wski":10344,"premises":10345,"fin":10346,"1500":10347,"alpine":10348,"matched":10349,"freely":10350,"bounded":10351,"jace":10352,"sleeve":10353,"##af":10354,"gaming":10355,"pier":10356,"populated":10357,"evident":10358,"##like":10359,"frances":10360,"flooded":10361,"##dle":10362,"frightened":10363,"pour":10364,"trainer":10365,"framed":10366,"visitor":10367,"challenging":10368,"pig":10369,"wickets":10370,"##fold":10371,"infected":10372,"email":10373,"##pes":10374,"arose":10375,"##aw":10376,"reward":10377,"ecuador":10378,"oblast":10379,"vale":10380,"ch":10381,"shuttle":10382,"##usa":10383,"bach":10384,"rankings":10385,"forbidden":10386,"cornwall":10387,"accordance":10388,"salem":10389,"consumers":10390,"bruno":10391,"fantastic":10392,"toes":10393,"machinery":10394,"resolved":10395,"julius":10396,"remembering":10397,"propaganda":10398,"iceland":10399,"bombardment":10400,"tide":10401,"contacts":10402,"wives":10403,"##rah":10404,"concerto":10405,"macdonald":10406,"albania":10407,"implement":10408,"daisy":10409,"tapped":10410,"sudan":10411,"helmet":10412,"angela":10413,"mistress":10414,"##lic":10415,"crop":10416,"sunk":10417,"finest":10418,"##craft":10419,"hostile":10420,"##ute":10421,"##tsu":10422,"boxer":10423,"fr":10424,"paths":10425,"adjusted":10426,"habit":10427,"ballot":10428,"supervision":10429,"soprano":10430,"##zen":10431,"bullets":10432,"wicked":10433,"sunset":10434,"regiments":10435,"disappear":10436,"lamp":10437,"performs":10438,"app":10439,"##gia":10440,"##oa":10441,"rabbit":10442,"digging":10443,"incidents":10444,"entries":10445,"##cion":10446,"dishes":10447,"##oi":10448,"introducing":10449,"##ati":10450,"##fied":10451,"freshman":10452,"slot":10453,"jill":10454,"tackles":10455,"baroque":10456,"backs":10457,"##iest":10458,"lone":10459,"sponsor":10460,"destiny":10461,"altogether":10462,"convert":10463,"##aro":10464,"consensus":10465,"shapes":10466,"demonstration":10467,"basically":10468,"feminist":10469,"auction":10470,"artifacts":10471,"##bing":10472,"strongest":10473,"twitter":10474,"halifax":10475,"2019":10476,"allmusic":10477,"mighty":10478,"smallest":10479,"precise":10480,"alexandra":10481,"viola":10482,"##los":10483,"##ille":10484,"manuscripts":10485,"##illo":10486,"dancers":10487,"ari":10488,"managers":10489,"monuments":10490,"blades":10491,"barracks":10492,"springfield":10493,"maiden":10494,"consolidated":10495,"electron":10496,"##end":10497,"berry":10498,"airing":10499,"wheat":10500,"nobel":10501,"inclusion":10502,"blair":10503,"payments":10504,"geography":10505,"bee":10506,"cc":10507,"eleanor":10508,"react":10509,"##hurst":10510,"afc":10511,"manitoba":10512,"##yu":10513,"su":10514,"lineup":10515,"fitness":10516,"recreational":10517,"investments":10518,"airborne":10519,"disappointment":10520,"##dis":10521,"edmonton":10522,"viewing":10523,"##row":10524,"renovation":10525,"##cast":10526,"infant":10527,"bankruptcy":10528,"roses":10529,"aftermath":10530,"pavilion":10531,"##yer":10532,"carpenter":10533,"withdrawal":10534,"ladder":10535,"##hy":10536,"discussing":10537,"popped":10538,"reliable":10539,"agreements":10540,"rochester":10541,"##abad":10542,"curves":10543,"bombers":10544,"220":10545,"rao":10546,"reverend":10547,"decreased":10548,"choosing":10549,"107":10550,"stiff":10551,"consulting":10552,"naples":10553,"crawford":10554,"tracy":10555,"ka":10556,"ribbon":10557,"cops":10558,"##lee":10559,"crushed":10560,"deciding":10561,"unified":10562,"teenager":10563,"accepting":10564,"flagship":10565,"explorer":10566,"poles":10567,"sanchez":10568,"inspection":10569,"revived":10570,"skilled":10571,"induced":10572,"exchanged":10573,"flee":10574,"locals":10575,"tragedy":10576,"swallow":10577,"loading":10578,"hanna":10579,"demonstrate":10580,"##ela":10581,"salvador":10582,"flown":10583,"contestants":10584,"civilization":10585,"##ines":10586,"wanna":10587,"rhodes":10588,"fletcher":10589,"hector":10590,"knocking":10591,"considers":10592,"##ough":10593,"nash":10594,"mechanisms":10595,"sensed":10596,"mentally":10597,"walt":10598,"unclear":10599,"##eus":10600,"renovated":10601,"madame":10602,"##cks":10603,"crews":10604,"governmental":10605,"##hin":10606,"undertaken":10607,"monkey":10608,"##ben":10609,"##ato":10610,"fatal":10611,"armored":10612,"copa":10613,"caves":10614,"governance":10615,"grasp":10616,"perception":10617,"certification":10618,"froze":10619,"damp":10620,"tugged":10621,"wyoming":10622,"##rg":10623,"##ero":10624,"newman":10625,"##lor":10626,"nerves":10627,"curiosity":10628,"graph":10629,"115":10630,"##ami":10631,"withdraw":10632,"tunnels":10633,"dull":10634,"meredith":10635,"moss":10636,"exhibits":10637,"neighbors":10638,"communicate":10639,"accuracy":10640,"explored":10641,"raiders":10642,"republicans":10643,"secular":10644,"kat":10645,"superman":10646,"penny":10647,"criticised":10648,"##tch":10649,"freed":10650,"update":10651,"conviction":10652,"wade":10653,"ham":10654,"likewise":10655,"delegation":10656,"gotta":10657,"doll":10658,"promises":10659,"technological":10660,"myth":10661,"nationality":10662,"resolve":10663,"convent":10664,"##mark":10665,"sharon":10666,"dig":10667,"sip":10668,"coordinator":10669,"entrepreneur":10670,"fold":10671,"##dine":10672,"capability":10673,"councillor":10674,"synonym":10675,"blown":10676,"swan":10677,"cursed":10678,"1815":10679,"jonas":10680,"haired":10681,"sofa":10682,"canvas":10683,"keeper":10684,"rivalry":10685,"##hart":10686,"rapper":10687,"speedway":10688,"swords":10689,"postal":10690,"maxwell":10691,"estonia":10692,"potter":10693,"recurring":10694,"##nn":10695,"##ave":10696,"errors":10697,"##oni":10698,"cognitive":10699,"1834":10700,"##²":10701,"claws":10702,"nadu":10703,"roberto":10704,"bce":10705,"wrestler":10706,"ellie":10707,"##ations":10708,"infinite":10709,"ink":10710,"##tia":10711,"presumably":10712,"finite":10713,"staircase":10714,"108":10715,"noel":10716,"patricia":10717,"nacional":10718,"##cation":10719,"chill":10720,"eternal":10721,"tu":10722,"preventing":10723,"prussia":10724,"fossil":10725,"limbs":10726,"##logist":10727,"ernst":10728,"frog":10729,"perez":10730,"rene":10731,"##ace":10732,"pizza":10733,"prussian":10734,"##ios":10735,"##vy":10736,"molecules":10737,"regulatory":10738,"answering":10739,"opinions":10740,"sworn":10741,"lengths":10742,"supposedly":10743,"hypothesis":10744,"upward":10745,"habitats":10746,"seating":10747,"ancestors":10748,"drank":10749,"yield":10750,"hd":10751,"synthesis":10752,"researcher":10753,"modest":10754,"##var":10755,"mothers":10756,"peered":10757,"voluntary":10758,"homeland":10759,"##the":10760,"acclaim":10761,"##igan":10762,"static":10763,"valve":10764,"luxembourg":10765,"alto":10766,"carroll":10767,"fe":10768,"receptor":10769,"norton":10770,"ambulance":10771,"##tian":10772,"johnston":10773,"catholics":10774,"depicting":10775,"jointly":10776,"elephant":10777,"gloria":10778,"mentor":10779,"badge":10780,"ahmad":10781,"distinguish":10782,"remarked":10783,"councils":10784,"precisely":10785,"allison":10786,"advancing":10787,"detection":10788,"crowded":10789,"##10":10790,"cooperative":10791,"ankle":10792,"mercedes":10793,"dagger":10794,"surrendered":10795,"pollution":10796,"commit":10797,"subway":10798,"jeffrey":10799,"lesson":10800,"sculptures":10801,"provider":10802,"##fication":10803,"membrane":10804,"timothy":10805,"rectangular":10806,"fiscal":10807,"heating":10808,"teammate":10809,"basket":10810,"particle":10811,"anonymous":10812,"deployment":10813,"##ple":10814,"missiles":10815,"courthouse":10816,"proportion":10817,"shoe":10818,"sec":10819,"##ller":10820,"complaints":10821,"forbes":10822,"blacks":10823,"abandon":10824,"remind":10825,"sizes":10826,"overwhelming":10827,"autobiography":10828,"natalie":10829,"##awa":10830,"risks":10831,"contestant":10832,"countryside":10833,"babies":10834,"scorer":10835,"invaded":10836,"enclosed":10837,"proceed":10838,"hurling":10839,"disorders":10840,"##cu":10841,"reflecting":10842,"continuously":10843,"cruiser":10844,"graduates":10845,"freeway":10846,"investigated":10847,"ore":10848,"deserved":10849,"maid":10850,"blocking":10851,"phillip":10852,"jorge":10853,"shakes":10854,"dove":10855,"mann":10856,"variables":10857,"lacked":10858,"burden":10859,"accompanying":10860,"que":10861,"consistently":10862,"organizing":10863,"provisional":10864,"complained":10865,"endless":10866,"##rm":10867,"tubes":10868,"juice":10869,"georges":10870,"krishna":10871,"mick":10872,"labels":10873,"thriller":10874,"##uch":10875,"laps":10876,"arcade":10877,"sage":10878,"snail":10879,"##table":10880,"shannon":10881,"fi":10882,"laurence":10883,"seoul":10884,"vacation":10885,"presenting":10886,"hire":10887,"churchill":10888,"surprisingly":10889,"prohibited":10890,"savannah":10891,"technically":10892,"##oli":10893,"170":10894,"##lessly":10895,"testimony":10896,"suited":10897,"speeds":10898,"toys":10899,"romans":10900,"mlb":10901,"flowering":10902,"measurement":10903,"talented":10904,"kay":10905,"settings":10906,"charleston":10907,"expectations":10908,"shattered":10909,"achieving":10910,"triumph":10911,"ceremonies":10912,"portsmouth":10913,"lanes":10914,"mandatory":10915,"loser":10916,"stretching":10917,"cologne":10918,"realizes":10919,"seventy":10920,"cornell":10921,"careers":10922,"webb":10923,"##ulating":10924,"americas":10925,"budapest":10926,"ava":10927,"suspicion":10928,"##ison":10929,"yo":10930,"conrad":10931,"##hai":10932,"sterling":10933,"jessie":10934,"rector":10935,"##az":10936,"1831":10937,"transform":10938,"organize":10939,"loans":10940,"christine":10941,"volcanic":10942,"warrant":10943,"slender":10944,"summers":10945,"subfamily":10946,"newer":10947,"danced":10948,"dynamics":10949,"rhine":10950,"proceeds":10951,"heinrich":10952,"gastropod":10953,"commands":10954,"sings":10955,"facilitate":10956,"easter":10957,"ra":10958,"positioned":10959,"responses":10960,"expense":10961,"fruits":10962,"yanked":10963,"imported":10964,"25th":10965,"velvet":10966,"vic":10967,"primitive":10968,"tribune":10969,"baldwin":10970,"neighbourhood":10971,"donna":10972,"rip":10973,"hay":10974,"pr":10975,"##uro":10976,"1814":10977,"espn":10978,"welcomed":10979,"##aria":10980,"qualifier":10981,"glare":10982,"highland":10983,"timing":10984,"##cted":10985,"shells":10986,"eased":10987,"geometry":10988,"louder":10989,"exciting":10990,"slovakia":10991,"##sion":10992,"##iz":10993,"##lot":10994,"savings":10995,"prairie":10996,"##ques":10997,"marching":10998,"rafael":10999,"tonnes":11000,"##lled":11001,"curtain":11002,"preceding":11003,"shy":11004,"heal":11005,"greene":11006,"worthy":11007,"##pot":11008,"detachment":11009,"bury":11010,"sherman":11011,"##eck":11012,"reinforced":11013,"seeks":11014,"bottles":11015,"contracted":11016,"duchess":11017,"outfit":11018,"walsh":11019,"##sc":11020,"mickey":11021,"##ase":11022,"geoffrey":11023,"archer":11024,"squeeze":11025,"dawson":11026,"eliminate":11027,"invention":11028,"##enberg":11029,"neal":11030,"##eth":11031,"stance":11032,"dealer":11033,"coral":11034,"maple":11035,"retire":11036,"polo":11037,"simplified":11038,"##ht":11039,"1833":11040,"hid":11041,"watts":11042,"backwards":11043,"jules":11044,"##oke":11045,"genesis":11046,"mt":11047,"frames":11048,"rebounds":11049,"burma":11050,"woodland":11051,"moist":11052,"santos":11053,"whispers":11054,"drained":11055,"subspecies":11056,"##aa":11057,"streaming":11058,"ulster":11059,"burnt":11060,"correspondence":11061,"maternal":11062,"gerard":11063,"denis":11064,"stealing":11065,"##load":11066,"genius":11067,"duchy":11068,"##oria":11069,"inaugurated":11070,"momentum":11071,"suits":11072,"placement":11073,"sovereign":11074,"clause":11075,"thames":11076,"##hara":11077,"confederation":11078,"reservation":11079,"sketch":11080,"yankees":11081,"lets":11082,"rotten":11083,"charm":11084,"hal":11085,"verses":11086,"ultra":11087,"commercially":11088,"dot":11089,"salon":11090,"citation":11091,"adopt":11092,"winnipeg":11093,"mist":11094,"allocated":11095,"cairo":11096,"##boy":11097,"jenkins":11098,"interference":11099,"objectives":11100,"##wind":11101,"1820":11102,"portfolio":11103,"armoured":11104,"sectors":11105,"##eh":11106,"initiatives":11107,"##world":11108,"integrity":11109,"exercises":11110,"robe":11111,"tap":11112,"ab":11113,"gazed":11114,"##tones":11115,"distracted":11116,"rulers":11117,"111":11118,"favorable":11119,"jerome":11120,"tended":11121,"cart":11122,"factories":11123,"##eri":11124,"diplomat":11125,"valued":11126,"gravel":11127,"charitable":11128,"##try":11129,"calvin":11130,"exploring":11131,"chang":11132,"shepherd":11133,"terrace":11134,"pdf":11135,"pupil":11136,"##ural":11137,"reflects":11138,"ups":11139,"##rch":11140,"governors":11141,"shelf":11142,"depths":11143,"##nberg":11144,"trailed":11145,"crest":11146,"tackle":11147,"##nian":11148,"##ats":11149,"hatred":11150,"##kai":11151,"clare":11152,"makers":11153,"ethiopia":11154,"longtime":11155,"detected":11156,"embedded":11157,"lacking":11158,"slapped":11159,"rely":11160,"thomson":11161,"anticipation":11162,"iso":11163,"morton":11164,"successive":11165,"agnes":11166,"screenwriter":11167,"straightened":11168,"philippe":11169,"playwright":11170,"haunted":11171,"licence":11172,"iris":11173,"intentions":11174,"sutton":11175,"112":11176,"logical":11177,"correctly":11178,"##weight":11179,"branded":11180,"licked":11181,"tipped":11182,"silva":11183,"ricky":11184,"narrator":11185,"requests":11186,"##ents":11187,"greeted":11188,"supernatural":11189,"cow":11190,"##wald":11191,"lung":11192,"refusing":11193,"employer":11194,"strait":11195,"gaelic":11196,"liner":11197,"##piece":11198,"zoe":11199,"sabha":11200,"##mba":11201,"driveway":11202,"harvest":11203,"prints":11204,"bates":11205,"reluctantly":11206,"threshold":11207,"algebra":11208,"ira":11209,"wherever":11210,"coupled":11211,"240":11212,"assumption":11213,"picks":11214,"##air":11215,"designers":11216,"raids":11217,"gentlemen":11218,"##ean":11219,"roller":11220,"blowing":11221,"leipzig":11222,"locks":11223,"screw":11224,"dressing":11225,"strand":11226,"##lings":11227,"scar":11228,"dwarf":11229,"depicts":11230,"##nu":11231,"nods":11232,"##mine":11233,"differ":11234,"boris":11235,"##eur":11236,"yuan":11237,"flip":11238,"##gie":11239,"mob":11240,"invested":11241,"questioning":11242,"applying":11243,"##ture":11244,"shout":11245,"##sel":11246,"gameplay":11247,"blamed":11248,"illustrations":11249,"bothered":11250,"weakness":11251,"rehabilitation":11252,"##of":11253,"##zes":11254,"envelope":11255,"rumors":11256,"miners":11257,"leicester":11258,"subtle":11259,"kerry":11260,"##ico":11261,"ferguson":11262,"##fu":11263,"premiership":11264,"ne":11265,"##cat":11266,"bengali":11267,"prof":11268,"catches":11269,"remnants":11270,"dana":11271,"##rily":11272,"shouting":11273,"presidents":11274,"baltic":11275,"ought":11276,"ghosts":11277,"dances":11278,"sailors":11279,"shirley":11280,"fancy":11281,"dominic":11282,"##bie":11283,"madonna":11284,"##rick":11285,"bark":11286,"buttons":11287,"gymnasium":11288,"ashes":11289,"liver":11290,"toby":11291,"oath":11292,"providence":11293,"doyle":11294,"evangelical":11295,"nixon":11296,"cement":11297,"carnegie":11298,"embarked":11299,"hatch":11300,"surroundings":11301,"guarantee":11302,"needing":11303,"pirate":11304,"essence":11305,"##bee":11306,"filter":11307,"crane":11308,"hammond":11309,"projected":11310,"immune":11311,"percy":11312,"twelfth":11313,"##ult":11314,"regent":11315,"doctoral":11316,"damon":11317,"mikhail":11318,"##ichi":11319,"lu":11320,"critically":11321,"elect":11322,"realised":11323,"abortion":11324,"acute":11325,"screening":11326,"mythology":11327,"steadily":11328,"##fc":11329,"frown":11330,"nottingham":11331,"kirk":11332,"wa":11333,"minneapolis":11334,"##rra":11335,"module":11336,"algeria":11337,"mc":11338,"nautical":11339,"encounters":11340,"surprising":11341,"statues":11342,"availability":11343,"shirts":11344,"pie":11345,"alma":11346,"brows":11347,"munster":11348,"mack":11349,"soup":11350,"crater":11351,"tornado":11352,"sanskrit":11353,"cedar":11354,"explosive":11355,"bordered":11356,"dixon":11357,"planets":11358,"stamp":11359,"exam":11360,"happily":11361,"##bble":11362,"carriers":11363,"kidnapped":11364,"##vis":11365,"accommodation":11366,"emigrated":11367,"##met":11368,"knockout":11369,"correspondent":11370,"violation":11371,"profits":11372,"peaks":11373,"lang":11374,"specimen":11375,"agenda":11376,"ancestry":11377,"pottery":11378,"spelling":11379,"equations":11380,"obtaining":11381,"ki":11382,"linking":11383,"1825":11384,"debris":11385,"asylum":11386,"##20":11387,"buddhism":11388,"teddy":11389,"##ants":11390,"gazette":11391,"##nger":11392,"##sse":11393,"dental":11394,"eligibility":11395,"utc":11396,"fathers":11397,"averaged":11398,"zimbabwe":11399,"francesco":11400,"coloured":11401,"hissed":11402,"translator":11403,"lynch":11404,"mandate":11405,"humanities":11406,"mackenzie":11407,"uniforms":11408,"lin":11409,"##iana":11410,"##gio":11411,"asset":11412,"mhz":11413,"fitting":11414,"samantha":11415,"genera":11416,"wei":11417,"rim":11418,"beloved":11419,"shark":11420,"riot":11421,"entities":11422,"expressions":11423,"indo":11424,"carmen":11425,"slipping":11426,"owing":11427,"abbot":11428,"neighbor":11429,"sidney":11430,"##av":11431,"rats":11432,"recommendations":11433,"encouraging":11434,"squadrons":11435,"anticipated":11436,"commanders":11437,"conquered":11438,"##oto":11439,"donations":11440,"diagnosed":11441,"##mond":11442,"divide":11443,"##iva":11444,"guessed":11445,"decoration":11446,"vernon":11447,"auditorium":11448,"revelation":11449,"conversations":11450,"##kers":11451,"##power":11452,"herzegovina":11453,"dash":11454,"alike":11455,"protested":11456,"lateral":11457,"herman":11458,"accredited":11459,"mg":11460,"##gent":11461,"freeman":11462,"mel":11463,"fiji":11464,"crow":11465,"crimson":11466,"##rine":11467,"livestock":11468,"##pped":11469,"humanitarian":11470,"bored":11471,"oz":11472,"whip":11473,"##lene":11474,"##ali":11475,"legitimate":11476,"alter":11477,"grinning":11478,"spelled":11479,"anxious":11480,"oriental":11481,"wesley":11482,"##nin":11483,"##hole":11484,"carnival":11485,"controller":11486,"detect":11487,"##ssa":11488,"bowed":11489,"educator":11490,"kosovo":11491,"macedonia":11492,"##sin":11493,"occupy":11494,"mastering":11495,"stephanie":11496,"janeiro":11497,"para":11498,"unaware":11499,"nurses":11500,"noon":11501,"135":11502,"cam":11503,"hopefully":11504,"ranger":11505,"combine":11506,"sociology":11507,"polar":11508,"rica":11509,"##eer":11510,"neill":11511,"##sman":11512,"holocaust":11513,"##ip":11514,"doubled":11515,"lust":11516,"1828":11517,"109":11518,"decent":11519,"cooling":11520,"unveiled":11521,"##card":11522,"1829":11523,"nsw":11524,"homer":11525,"chapman":11526,"meyer":11527,"##gin":11528,"dive":11529,"mae":11530,"reagan":11531,"expertise":11532,"##gled":11533,"darwin":11534,"brooke":11535,"sided":11536,"prosecution":11537,"investigating":11538,"comprised":11539,"petroleum":11540,"genres":11541,"reluctant":11542,"differently":11543,"trilogy":11544,"johns":11545,"vegetables":11546,"corpse":11547,"highlighted":11548,"lounge":11549,"pension":11550,"unsuccessfully":11551,"elegant":11552,"aided":11553,"ivory":11554,"beatles":11555,"amelia":11556,"cain":11557,"dubai":11558,"sunny":11559,"immigrant":11560,"babe":11561,"click":11562,"##nder":11563,"underwater":11564,"pepper":11565,"combining":11566,"mumbled":11567,"atlas":11568,"horns":11569,"accessed":11570,"ballad":11571,"physicians":11572,"homeless":11573,"gestured":11574,"rpm":11575,"freak":11576,"louisville":11577,"corporations":11578,"patriots":11579,"prizes":11580,"rational":11581,"warn":11582,"modes":11583,"decorative":11584,"overnight":11585,"din":11586,"troubled":11587,"phantom":11588,"##ort":11589,"monarch":11590,"sheer":11591,"##dorf":11592,"generals":11593,"guidelines":11594,"organs":11595,"addresses":11596,"##zon":11597,"enhance":11598,"curling":11599,"parishes":11600,"cord":11601,"##kie":11602,"linux":11603,"caesar":11604,"deutsche":11605,"bavaria":11606,"##bia":11607,"coleman":11608,"cyclone":11609,"##eria":11610,"bacon":11611,"petty":11612,"##yama":11613,"##old":11614,"hampton":11615,"diagnosis":11616,"1824":11617,"throws":11618,"complexity":11619,"rita":11620,"disputed":11621,"##₃":11622,"pablo":11623,"##sch":11624,"marketed":11625,"trafficking":11626,"##ulus":11627,"examine":11628,"plague":11629,"formats":11630,"##oh":11631,"vault":11632,"faithful":11633,"##bourne":11634,"webster":11635,"##ox":11636,"highlights":11637,"##ient":11638,"##ann":11639,"phones":11640,"vacuum":11641,"sandwich":11642,"modeling":11643,"##gated":11644,"bolivia":11645,"clergy":11646,"qualities":11647,"isabel":11648,"##nas":11649,"##ars":11650,"wears":11651,"screams":11652,"reunited":11653,"annoyed":11654,"bra":11655,"##ancy":11656,"##rate":11657,"differential":11658,"transmitter":11659,"tattoo":11660,"container":11661,"poker":11662,"##och":11663,"excessive":11664,"resides":11665,"cowboys":11666,"##tum":11667,"augustus":11668,"trash":11669,"providers":11670,"statute":11671,"retreated":11672,"balcony":11673,"reversed":11674,"void":11675,"storey":11676,"preceded":11677,"masses":11678,"leap":11679,"laughs":11680,"neighborhoods":11681,"wards":11682,"schemes":11683,"falcon":11684,"santo":11685,"battlefield":11686,"pad":11687,"ronnie":11688,"thread":11689,"lesbian":11690,"venus":11691,"##dian":11692,"beg":11693,"sandstone":11694,"daylight":11695,"punched":11696,"gwen":11697,"analog":11698,"stroked":11699,"wwe":11700,"acceptable":11701,"measurements":11702,"dec":11703,"toxic":11704,"##kel":11705,"adequate":11706,"surgical":11707,"economist":11708,"parameters":11709,"varsity":11710,"##sberg":11711,"quantity":11712,"ella":11713,"##chy":11714,"##rton":11715,"countess":11716,"generating":11717,"precision":11718,"diamonds":11719,"expressway":11720,"ga":11721,"##ı":11722,"1821":11723,"uruguay":11724,"talents":11725,"galleries":11726,"expenses":11727,"scanned":11728,"colleague":11729,"outlets":11730,"ryder":11731,"lucien":11732,"##ila":11733,"paramount":11734,"##bon":11735,"syracuse":11736,"dim":11737,"fangs":11738,"gown":11739,"sweep":11740,"##sie":11741,"toyota":11742,"missionaries":11743,"websites":11744,"##nsis":11745,"sentences":11746,"adviser":11747,"val":11748,"trademark":11749,"spells":11750,"##plane":11751,"patience":11752,"starter":11753,"slim":11754,"##borg":11755,"toe":11756,"incredibly":11757,"shoots":11758,"elliot":11759,"nobility":11760,"##wyn":11761,"cowboy":11762,"endorsed":11763,"gardner":11764,"tendency":11765,"persuaded":11766,"organisms":11767,"emissions":11768,"kazakhstan":11769,"amused":11770,"boring":11771,"chips":11772,"themed":11773,"##hand":11774,"llc":11775,"constantinople":11776,"chasing":11777,"systematic":11778,"guatemala":11779,"borrowed":11780,"erin":11781,"carey":11782,"##hard":11783,"highlands":11784,"struggles":11785,"1810":11786,"##ifying":11787,"##ced":11788,"wong":11789,"exceptions":11790,"develops":11791,"enlarged":11792,"kindergarten":11793,"castro":11794,"##ern":11795,"##rina":11796,"leigh":11797,"zombie":11798,"juvenile":11799,"##most":11800,"consul":11801,"##nar":11802,"sailor":11803,"hyde":11804,"clarence":11805,"intensive":11806,"pinned":11807,"nasty":11808,"useless":11809,"jung":11810,"clayton":11811,"stuffed":11812,"exceptional":11813,"ix":11814,"apostolic":11815,"230":11816,"transactions":11817,"##dge":11818,"exempt":11819,"swinging":11820,"cove":11821,"religions":11822,"##ash":11823,"shields":11824,"dairy":11825,"bypass":11826,"190":11827,"pursuing":11828,"bug":11829,"joyce":11830,"bombay":11831,"chassis":11832,"southampton":11833,"chat":11834,"interact":11835,"redesignated":11836,"##pen":11837,"nascar":11838,"pray":11839,"salmon":11840,"rigid":11841,"regained":11842,"malaysian":11843,"grim":11844,"publicity":11845,"constituted":11846,"capturing":11847,"toilet":11848,"delegate":11849,"purely":11850,"tray":11851,"drift":11852,"loosely":11853,"striker":11854,"weakened":11855,"trinidad":11856,"mitch":11857,"itv":11858,"defines":11859,"transmitted":11860,"ming":11861,"scarlet":11862,"nodding":11863,"fitzgerald":11864,"fu":11865,"narrowly":11866,"sp":11867,"tooth":11868,"standings":11869,"virtue":11870,"##₁":11871,"##wara":11872,"##cting":11873,"chateau":11874,"gloves":11875,"lid":11876,"##nel":11877,"hurting":11878,"conservatory":11879,"##pel":11880,"sinclair":11881,"reopened":11882,"sympathy":11883,"nigerian":11884,"strode":11885,"advocated":11886,"optional":11887,"chronic":11888,"discharge":11889,"##rc":11890,"suck":11891,"compatible":11892,"laurel":11893,"stella":11894,"shi":11895,"fails":11896,"wage":11897,"dodge":11898,"128":11899,"informal":11900,"sorts":11901,"levi":11902,"buddha":11903,"villagers":11904,"##aka":11905,"chronicles":11906,"heavier":11907,"summoned":11908,"gateway":11909,"3000":11910,"eleventh":11911,"jewelry":11912,"translations":11913,"accordingly":11914,"seas":11915,"##ency":11916,"fiber":11917,"pyramid":11918,"cubic":11919,"dragging":11920,"##ista":11921,"caring":11922,"##ops":11923,"android":11924,"contacted":11925,"lunar":11926,"##dt":11927,"kai":11928,"lisbon":11929,"patted":11930,"1826":11931,"sacramento":11932,"theft":11933,"madagascar":11934,"subtropical":11935,"disputes":11936,"ta":11937,"holidays":11938,"piper":11939,"willow":11940,"mare":11941,"cane":11942,"itunes":11943,"newfoundland":11944,"benny":11945,"companions":11946,"dong":11947,"raj":11948,"observe":11949,"roar":11950,"charming":11951,"plaque":11952,"tibetan":11953,"fossils":11954,"enacted":11955,"manning":11956,"bubble":11957,"tina":11958,"tanzania":11959,"##eda":11960,"##hir":11961,"funk":11962,"swamp":11963,"deputies":11964,"cloak":11965,"ufc":11966,"scenario":11967,"par":11968,"scratch":11969,"metals":11970,"anthem":11971,"guru":11972,"engaging":11973,"specially":11974,"##boat":11975,"dialects":11976,"nineteen":11977,"cecil":11978,"duet":11979,"disability":11980,"messenger":11981,"unofficial":11982,"##lies":11983,"defunct":11984,"eds":11985,"moonlight":11986,"drainage":11987,"surname":11988,"puzzle":11989,"honda":11990,"switching":11991,"conservatives":11992,"mammals":11993,"knox":11994,"broadcaster":11995,"sidewalk":11996,"cope":11997,"##ried":11998,"benson":11999,"princes":12000,"peterson":12001,"##sal":12002,"bedford":12003,"sharks":12004,"eli":12005,"wreck":12006,"alberto":12007,"gasp":12008,"archaeology":12009,"lgbt":12010,"teaches":12011,"securities":12012,"madness":12013,"compromise":12014,"waving":12015,"coordination":12016,"davidson":12017,"visions":12018,"leased":12019,"possibilities":12020,"eighty":12021,"jun":12022,"fernandez":12023,"enthusiasm":12024,"assassin":12025,"sponsorship":12026,"reviewer":12027,"kingdoms":12028,"estonian":12029,"laboratories":12030,"##fy":12031,"##nal":12032,"applies":12033,"verb":12034,"celebrations":12035,"##zzo":12036,"rowing":12037,"lightweight":12038,"sadness":12039,"submit":12040,"mvp":12041,"balanced":12042,"dude":12043,"##vas":12044,"explicitly":12045,"metric":12046,"magnificent":12047,"mound":12048,"brett":12049,"mohammad":12050,"mistakes":12051,"irregular":12052,"##hing":12053,"##ass":12054,"sanders":12055,"betrayed":12056,"shipped":12057,"surge":12058,"##enburg":12059,"reporters":12060,"termed":12061,"georg":12062,"pity":12063,"verbal":12064,"bulls":12065,"abbreviated":12066,"enabling":12067,"appealed":12068,"##are":12069,"##atic":12070,"sicily":12071,"sting":12072,"heel":12073,"sweetheart":12074,"bart":12075,"spacecraft":12076,"brutal":12077,"monarchy":12078,"##tter":12079,"aberdeen":12080,"cameo":12081,"diane":12082,"##ub":12083,"survivor":12084,"clyde":12085,"##aries":12086,"complaint":12087,"##makers":12088,"clarinet":12089,"delicious":12090,"chilean":12091,"karnataka":12092,"coordinates":12093,"1818":12094,"panties":12095,"##rst":12096,"pretending":12097,"ar":12098,"dramatically":12099,"kiev":12100,"bella":12101,"tends":12102,"distances":12103,"113":12104,"catalog":12105,"launching":12106,"instances":12107,"telecommunications":12108,"portable":12109,"lindsay":12110,"vatican":12111,"##eim":12112,"angles":12113,"aliens":12114,"marker":12115,"stint":12116,"screens":12117,"bolton":12118,"##rne":12119,"judy":12120,"wool":12121,"benedict":12122,"plasma":12123,"europa":12124,"spark":12125,"imaging":12126,"filmmaker":12127,"swiftly":12128,"##een":12129,"contributor":12130,"##nor":12131,"opted":12132,"stamps":12133,"apologize":12134,"financing":12135,"butter":12136,"gideon":12137,"sophisticated":12138,"alignment":12139,"avery":12140,"chemicals":12141,"yearly":12142,"speculation":12143,"prominence":12144,"professionally":12145,"##ils":12146,"immortal":12147,"institutional":12148,"inception":12149,"wrists":12150,"identifying":12151,"tribunal":12152,"derives":12153,"gains":12154,"##wo":12155,"papal":12156,"preference":12157,"linguistic":12158,"vince":12159,"operative":12160,"brewery":12161,"##ont":12162,"unemployment":12163,"boyd":12164,"##ured":12165,"##outs":12166,"albeit":12167,"prophet":12168,"1813":12169,"bi":12170,"##rr":12171,"##face":12172,"##rad":12173,"quarterly":12174,"asteroid":12175,"cleaned":12176,"radius":12177,"temper":12178,"##llen":12179,"telugu":12180,"jerk":12181,"viscount":12182,"menu":12183,"##ote":12184,"glimpse":12185,"##aya":12186,"yacht":12187,"hawaiian":12188,"baden":12189,"##rl":12190,"laptop":12191,"readily":12192,"##gu":12193,"monetary":12194,"offshore":12195,"scots":12196,"watches":12197,"##yang":12198,"##arian":12199,"upgrade":12200,"needle":12201,"xbox":12202,"lea":12203,"encyclopedia":12204,"flank":12205,"fingertips":12206,"##pus":12207,"delight":12208,"teachings":12209,"confirm":12210,"roth":12211,"beaches":12212,"midway":12213,"winters":12214,"##iah":12215,"teasing":12216,"daytime":12217,"beverly":12218,"gambling":12219,"bonnie":12220,"##backs":12221,"regulated":12222,"clement":12223,"hermann":12224,"tricks":12225,"knot":12226,"##shing":12227,"##uring":12228,"##vre":12229,"detached":12230,"ecological":12231,"owed":12232,"specialty":12233,"byron":12234,"inventor":12235,"bats":12236,"stays":12237,"screened":12238,"unesco":12239,"midland":12240,"trim":12241,"affection":12242,"##ander":12243,"##rry":12244,"jess":12245,"thoroughly":12246,"feedback":12247,"##uma":12248,"chennai":12249,"strained":12250,"heartbeat":12251,"wrapping":12252,"overtime":12253,"pleaded":12254,"##sworth":12255,"mon":12256,"leisure":12257,"oclc":12258,"##tate":12259,"##ele":12260,"feathers":12261,"angelo":12262,"thirds":12263,"nuts":12264,"surveys":12265,"clever":12266,"gill":12267,"commentator":12268,"##dos":12269,"darren":12270,"rides":12271,"gibraltar":12272,"##nc":12273,"##mu":12274,"dissolution":12275,"dedication":12276,"shin":12277,"meals":12278,"saddle":12279,"elvis":12280,"reds":12281,"chaired":12282,"taller":12283,"appreciation":12284,"functioning":12285,"niece":12286,"favored":12287,"advocacy":12288,"robbie":12289,"criminals":12290,"suffolk":12291,"yugoslav":12292,"passport":12293,"constable":12294,"congressman":12295,"hastings":12296,"vera":12297,"##rov":12298,"consecrated":12299,"sparks":12300,"ecclesiastical":12301,"confined":12302,"##ovich":12303,"muller":12304,"floyd":12305,"nora":12306,"1822":12307,"paved":12308,"1827":12309,"cumberland":12310,"ned":12311,"saga":12312,"spiral":12313,"##flow":12314,"appreciated":12315,"yi":12316,"collaborative":12317,"treating":12318,"similarities":12319,"feminine":12320,"finishes":12321,"##ib":12322,"jade":12323,"import":12324,"##nse":12325,"##hot":12326,"champagne":12327,"mice":12328,"securing":12329,"celebrities":12330,"helsinki":12331,"attributes":12332,"##gos":12333,"cousins":12334,"phases":12335,"ache":12336,"lucia":12337,"gandhi":12338,"submission":12339,"vicar":12340,"spear":12341,"shine":12342,"tasmania":12343,"biting":12344,"detention":12345,"constitute":12346,"tighter":12347,"seasonal":12348,"##gus":12349,"terrestrial":12350,"matthews":12351,"##oka":12352,"effectiveness":12353,"parody":12354,"philharmonic":12355,"##onic":12356,"1816":12357,"strangers":12358,"encoded":12359,"consortium":12360,"guaranteed":12361,"regards":12362,"shifts":12363,"tortured":12364,"collision":12365,"supervisor":12366,"inform":12367,"broader":12368,"insight":12369,"theaters":12370,"armour":12371,"emeritus":12372,"blink":12373,"incorporates":12374,"mapping":12375,"##50":12376,"##ein":12377,"handball":12378,"flexible":12379,"##nta":12380,"substantially":12381,"generous":12382,"thief":12383,"##own":12384,"carr":12385,"loses":12386,"1793":12387,"prose":12388,"ucla":12389,"romeo":12390,"generic":12391,"metallic":12392,"realization":12393,"damages":12394,"mk":12395,"commissioners":12396,"zach":12397,"default":12398,"##ther":12399,"helicopters":12400,"lengthy":12401,"stems":12402,"spa":12403,"partnered":12404,"spectators":12405,"rogue":12406,"indication":12407,"penalties":12408,"teresa":12409,"1801":12410,"sen":12411,"##tric":12412,"dalton":12413,"##wich":12414,"irving":12415,"photographic":12416,"##vey":12417,"dell":12418,"deaf":12419,"peters":12420,"excluded":12421,"unsure":12422,"##vable":12423,"patterson":12424,"crawled":12425,"##zio":12426,"resided":12427,"whipped":12428,"latvia":12429,"slower":12430,"ecole":12431,"pipes":12432,"employers":12433,"maharashtra":12434,"comparable":12435,"va":12436,"textile":12437,"pageant":12438,"##gel":12439,"alphabet":12440,"binary":12441,"irrigation":12442,"chartered":12443,"choked":12444,"antoine":12445,"offs":12446,"waking":12447,"supplement":12448,"##wen":12449,"quantities":12450,"demolition":12451,"regain":12452,"locate":12453,"urdu":12454,"folks":12455,"alt":12456,"114":12457,"##mc":12458,"scary":12459,"andreas":12460,"whites":12461,"##ava":12462,"classrooms":12463,"mw":12464,"aesthetic":12465,"publishes":12466,"valleys":12467,"guides":12468,"cubs":12469,"johannes":12470,"bryant":12471,"conventions":12472,"affecting":12473,"##itt":12474,"drain":12475,"awesome":12476,"isolation":12477,"prosecutor":12478,"ambitious":12479,"apology":12480,"captive":12481,"downs":12482,"atmospheric":12483,"lorenzo":12484,"aisle":12485,"beef":12486,"foul":12487,"##onia":12488,"kidding":12489,"composite":12490,"disturbed":12491,"illusion":12492,"natives":12493,"##ffer":12494,"emi":12495,"rockets":12496,"riverside":12497,"wartime":12498,"painters":12499,"adolf":12500,"melted":12501,"##ail":12502,"uncertainty":12503,"simulation":12504,"hawks":12505,"progressed":12506,"meantime":12507,"builder":12508,"spray":12509,"breach":12510,"unhappy":12511,"regina":12512,"russians":12513,"##urg":12514,"determining":12515,"##tation":12516,"tram":12517,"1806":12518,"##quin":12519,"aging":12520,"##12":12521,"1823":12522,"garion":12523,"rented":12524,"mister":12525,"diaz":12526,"terminated":12527,"clip":12528,"1817":12529,"depend":12530,"nervously":12531,"disco":12532,"owe":12533,"defenders":12534,"shiva":12535,"notorious":12536,"disbelief":12537,"shiny":12538,"worcester":12539,"##gation":12540,"##yr":12541,"trailing":12542,"undertook":12543,"islander":12544,"belarus":12545,"limitations":12546,"watershed":12547,"fuller":12548,"overlooking":12549,"utilized":12550,"raphael":12551,"1819":12552,"synthetic":12553,"breakdown":12554,"klein":12555,"##nate":12556,"moaned":12557,"memoir":12558,"lamb":12559,"practicing":12560,"##erly":12561,"cellular":12562,"arrows":12563,"exotic":12564,"##graphy":12565,"witches":12566,"117":12567,"charted":12568,"rey":12569,"hut":12570,"hierarchy":12571,"subdivision":12572,"freshwater":12573,"giuseppe":12574,"aloud":12575,"reyes":12576,"qatar":12577,"marty":12578,"sideways":12579,"utterly":12580,"sexually":12581,"jude":12582,"prayers":12583,"mccarthy":12584,"softball":12585,"blend":12586,"damien":12587,"##gging":12588,"##metric":12589,"wholly":12590,"erupted":12591,"lebanese":12592,"negro":12593,"revenues":12594,"tasted":12595,"comparative":12596,"teamed":12597,"transaction":12598,"labeled":12599,"maori":12600,"sovereignty":12601,"parkway":12602,"trauma":12603,"gran":12604,"malay":12605,"121":12606,"advancement":12607,"descendant":12608,"2020":12609,"buzz":12610,"salvation":12611,"inventory":12612,"symbolic":12613,"##making":12614,"antarctica":12615,"mps":12616,"##gas":12617,"##bro":12618,"mohammed":12619,"myanmar":12620,"holt":12621,"submarines":12622,"tones":12623,"##lman":12624,"locker":12625,"patriarch":12626,"bangkok":12627,"emerson":12628,"remarks":12629,"predators":12630,"kin":12631,"afghan":12632,"confession":12633,"norwich":12634,"rental":12635,"emerge":12636,"advantages":12637,"##zel":12638,"rca":12639,"##hold":12640,"shortened":12641,"storms":12642,"aidan":12643,"##matic":12644,"autonomy":12645,"compliance":12646,"##quet":12647,"dudley":12648,"atp":12649,"##osis":12650,"1803":12651,"motto":12652,"documentation":12653,"summary":12654,"professors":12655,"spectacular":12656,"christina":12657,"archdiocese":12658,"flashing":12659,"innocence":12660,"remake":12661,"##dell":12662,"psychic":12663,"reef":12664,"scare":12665,"employ":12666,"rs":12667,"sticks":12668,"meg":12669,"gus":12670,"leans":12671,"##ude":12672,"accompany":12673,"bergen":12674,"tomas":12675,"##iko":12676,"doom":12677,"wages":12678,"pools":12679,"##nch":12680,"##bes":12681,"breasts":12682,"scholarly":12683,"alison":12684,"outline":12685,"brittany":12686,"breakthrough":12687,"willis":12688,"realistic":12689,"##cut":12690,"##boro":12691,"competitor":12692,"##stan":12693,"pike":12694,"picnic":12695,"icon":12696,"designing":12697,"commercials":12698,"washing":12699,"villain":12700,"skiing":12701,"micro":12702,"costumes":12703,"auburn":12704,"halted":12705,"executives":12706,"##hat":12707,"logistics":12708,"cycles":12709,"vowel":12710,"applicable":12711,"barrett":12712,"exclaimed":12713,"eurovision":12714,"eternity":12715,"ramon":12716,"##umi":12717,"##lls":12718,"modifications":12719,"sweeping":12720,"disgust":12721,"##uck":12722,"torch":12723,"aviv":12724,"ensuring":12725,"rude":12726,"dusty":12727,"sonic":12728,"donovan":12729,"outskirts":12730,"cu":12731,"pathway":12732,"##band":12733,"##gun":12734,"##lines":12735,"disciplines":12736,"acids":12737,"cadet":12738,"paired":12739,"##40":12740,"sketches":12741,"##sive":12742,"marriages":12743,"##⁺":12744,"folding":12745,"peers":12746,"slovak":12747,"implies":12748,"admired":12749,"##beck":12750,"1880s":12751,"leopold":12752,"instinct":12753,"attained":12754,"weston":12755,"megan":12756,"horace":12757,"##ination":12758,"dorsal":12759,"ingredients":12760,"evolutionary":12761,"##its":12762,"complications":12763,"deity":12764,"lethal":12765,"brushing":12766,"levy":12767,"deserted":12768,"institutes":12769,"posthumously":12770,"delivering":12771,"telescope":12772,"coronation":12773,"motivated":12774,"rapids":12775,"luc":12776,"flicked":12777,"pays":12778,"volcano":12779,"tanner":12780,"weighed":12781,"##nica":12782,"crowds":12783,"frankie":12784,"gifted":12785,"addressing":12786,"granddaughter":12787,"winding":12788,"##rna":12789,"constantine":12790,"gomez":12791,"##front":12792,"landscapes":12793,"rudolf":12794,"anthropology":12795,"slate":12796,"werewolf":12797,"##lio":12798,"astronomy":12799,"circa":12800,"rouge":12801,"dreaming":12802,"sack":12803,"knelt":12804,"drowned":12805,"naomi":12806,"prolific":12807,"tracked":12808,"freezing":12809,"herb":12810,"##dium":12811,"agony":12812,"randall":12813,"twisting":12814,"wendy":12815,"deposit":12816,"touches":12817,"vein":12818,"wheeler":12819,"##bbled":12820,"##bor":12821,"batted":12822,"retaining":12823,"tire":12824,"presently":12825,"compare":12826,"specification":12827,"daemon":12828,"nigel":12829,"##grave":12830,"merry":12831,"recommendation":12832,"czechoslovakia":12833,"sandra":12834,"ng":12835,"roma":12836,"##sts":12837,"lambert":12838,"inheritance":12839,"sheikh":12840,"winchester":12841,"cries":12842,"examining":12843,"##yle":12844,"comeback":12845,"cuisine":12846,"nave":12847,"##iv":12848,"ko":12849,"retrieve":12850,"tomatoes":12851,"barker":12852,"polished":12853,"defining":12854,"irene":12855,"lantern":12856,"personalities":12857,"begging":12858,"tract":12859,"swore":12860,"1809":12861,"175":12862,"##gic":12863,"omaha":12864,"brotherhood":12865,"##rley":12866,"haiti":12867,"##ots":12868,"exeter":12869,"##ete":12870,"##zia":12871,"steele":12872,"dumb":12873,"pearson":12874,"210":12875,"surveyed":12876,"elisabeth":12877,"trends":12878,"##ef":12879,"fritz":12880,"##rf":12881,"premium":12882,"bugs":12883,"fraction":12884,"calmly":12885,"viking":12886,"##birds":12887,"tug":12888,"inserted":12889,"unusually":12890,"##ield":12891,"confronted":12892,"distress":12893,"crashing":12894,"brent":12895,"turks":12896,"resign":12897,"##olo":12898,"cambodia":12899,"gabe":12900,"sauce":12901,"##kal":12902,"evelyn":12903,"116":12904,"extant":12905,"clusters":12906,"quarry":12907,"teenagers":12908,"luna":12909,"##lers":12910,"##ister":12911,"affiliation":12912,"drill":12913,"##ashi":12914,"panthers":12915,"scenic":12916,"libya":12917,"anita":12918,"strengthen":12919,"inscriptions":12920,"##cated":12921,"lace":12922,"sued":12923,"judith":12924,"riots":12925,"##uted":12926,"mint":12927,"##eta":12928,"preparations":12929,"midst":12930,"dub":12931,"challenger":12932,"##vich":12933,"mock":12934,"cf":12935,"displaced":12936,"wicket":12937,"breaths":12938,"enables":12939,"schmidt":12940,"analyst":12941,"##lum":12942,"ag":12943,"highlight":12944,"automotive":12945,"axe":12946,"josef":12947,"newark":12948,"sufficiently":12949,"resembles":12950,"50th":12951,"##pal":12952,"flushed":12953,"mum":12954,"traits":12955,"##ante":12956,"commodore":12957,"incomplete":12958,"warming":12959,"titular":12960,"ceremonial":12961,"ethical":12962,"118":12963,"celebrating":12964,"eighteenth":12965,"cao":12966,"lima":12967,"medalist":12968,"mobility":12969,"strips":12970,"snakes":12971,"##city":12972,"miniature":12973,"zagreb":12974,"barton":12975,"escapes":12976,"umbrella":12977,"automated":12978,"doubted":12979,"differs":12980,"cooled":12981,"georgetown":12982,"dresden":12983,"cooked":12984,"fade":12985,"wyatt":12986,"rna":12987,"jacobs":12988,"carlton":12989,"abundant":12990,"stereo":12991,"boost":12992,"madras":12993,"inning":12994,"##hia":12995,"spur":12996,"ip":12997,"malayalam":12998,"begged":12999,"osaka":13000,"groan":13001,"escaping":13002,"charging":13003,"dose":13004,"vista":13005,"##aj":13006,"bud":13007,"papa":13008,"communists":13009,"advocates":13010,"edged":13011,"tri":13012,"##cent":13013,"resemble":13014,"peaking":13015,"necklace":13016,"fried":13017,"montenegro":13018,"saxony":13019,"goose":13020,"glances":13021,"stuttgart":13022,"curator":13023,"recruit":13024,"grocery":13025,"sympathetic":13026,"##tting":13027,"##fort":13028,"127":13029,"lotus":13030,"randolph":13031,"ancestor":13032,"##rand":13033,"succeeding":13034,"jupiter":13035,"1798":13036,"macedonian":13037,"##heads":13038,"hiking":13039,"1808":13040,"handing":13041,"fischer":13042,"##itive":13043,"garbage":13044,"node":13045,"##pies":13046,"prone":13047,"singular":13048,"papua":13049,"inclined":13050,"attractions":13051,"italia":13052,"pouring":13053,"motioned":13054,"grandma":13055,"garnered":13056,"jacksonville":13057,"corp":13058,"ego":13059,"ringing":13060,"aluminum":13061,"##hausen":13062,"ordering":13063,"##foot":13064,"drawer":13065,"traders":13066,"synagogue":13067,"##play":13068,"##kawa":13069,"resistant":13070,"wandering":13071,"fragile":13072,"fiona":13073,"teased":13074,"var":13075,"hardcore":13076,"soaked":13077,"jubilee":13078,"decisive":13079,"exposition":13080,"mercer":13081,"poster":13082,"valencia":13083,"hale":13084,"kuwait":13085,"1811":13086,"##ises":13087,"##wr":13088,"##eed":13089,"tavern":13090,"gamma":13091,"122":13092,"johan":13093,"##uer":13094,"airways":13095,"amino":13096,"gil":13097,"##ury":13098,"vocational":13099,"domains":13100,"torres":13101,"##sp":13102,"generator":13103,"folklore":13104,"outcomes":13105,"##keeper":13106,"canberra":13107,"shooter":13108,"fl":13109,"beams":13110,"confrontation":13111,"##lling":13112,"##gram":13113,"feb":13114,"aligned":13115,"forestry":13116,"pipeline":13117,"jax":13118,"motorway":13119,"conception":13120,"decay":13121,"##tos":13122,"coffin":13123,"##cott":13124,"stalin":13125,"1805":13126,"escorted":13127,"minded":13128,"##nam":13129,"sitcom":13130,"purchasing":13131,"twilight":13132,"veronica":13133,"additions":13134,"passive":13135,"tensions":13136,"straw":13137,"123":13138,"frequencies":13139,"1804":13140,"refugee":13141,"cultivation":13142,"##iate":13143,"christie":13144,"clary":13145,"bulletin":13146,"crept":13147,"disposal":13148,"##rich":13149,"##zong":13150,"processor":13151,"crescent":13152,"##rol":13153,"bmw":13154,"emphasized":13155,"whale":13156,"nazis":13157,"aurora":13158,"##eng":13159,"dwelling":13160,"hauled":13161,"sponsors":13162,"toledo":13163,"mega":13164,"ideology":13165,"theatres":13166,"tessa":13167,"cerambycidae":13168,"saves":13169,"turtle":13170,"cone":13171,"suspects":13172,"kara":13173,"rusty":13174,"yelling":13175,"greeks":13176,"mozart":13177,"shades":13178,"cocked":13179,"participant":13180,"##tro":13181,"shire":13182,"spit":13183,"freeze":13184,"necessity":13185,"##cos":13186,"inmates":13187,"nielsen":13188,"councillors":13189,"loaned":13190,"uncommon":13191,"omar":13192,"peasants":13193,"botanical":13194,"offspring":13195,"daniels":13196,"formations":13197,"jokes":13198,"1794":13199,"pioneers":13200,"sigma":13201,"licensing":13202,"##sus":13203,"wheelchair":13204,"polite":13205,"1807":13206,"liquor":13207,"pratt":13208,"trustee":13209,"##uta":13210,"forewings":13211,"balloon":13212,"##zz":13213,"kilometre":13214,"camping":13215,"explicit":13216,"casually":13217,"shawn":13218,"foolish":13219,"teammates":13220,"nm":13221,"hassan":13222,"carrie":13223,"judged":13224,"satisfy":13225,"vanessa":13226,"knives":13227,"selective":13228,"cnn":13229,"flowed":13230,"##lice":13231,"eclipse":13232,"stressed":13233,"eliza":13234,"mathematician":13235,"cease":13236,"cultivated":13237,"##roy":13238,"commissions":13239,"browns":13240,"##ania":13241,"destroyers":13242,"sheridan":13243,"meadow":13244,"##rius":13245,"minerals":13246,"##cial":13247,"downstream":13248,"clash":13249,"gram":13250,"memoirs":13251,"ventures":13252,"baha":13253,"seymour":13254,"archie":13255,"midlands":13256,"edith":13257,"fare":13258,"flynn":13259,"invite":13260,"canceled":13261,"tiles":13262,"stabbed":13263,"boulder":13264,"incorporate":13265,"amended":13266,"camden":13267,"facial":13268,"mollusk":13269,"unreleased":13270,"descriptions":13271,"yoga":13272,"grabs":13273,"550":13274,"raises":13275,"ramp":13276,"shiver":13277,"##rose":13278,"coined":13279,"pioneering":13280,"tunes":13281,"qing":13282,"warwick":13283,"tops":13284,"119":13285,"melanie":13286,"giles":13287,"##rous":13288,"wandered":13289,"##inal":13290,"annexed":13291,"nov":13292,"30th":13293,"unnamed":13294,"##ished":13295,"organizational":13296,"airplane":13297,"normandy":13298,"stoke":13299,"whistle":13300,"blessing":13301,"violations":13302,"chased":13303,"holders":13304,"shotgun":13305,"##ctic":13306,"outlet":13307,"reactor":13308,"##vik":13309,"tires":13310,"tearing":13311,"shores":13312,"fortified":13313,"mascot":13314,"constituencies":13315,"nc":13316,"columnist":13317,"productive":13318,"tibet":13319,"##rta":13320,"lineage":13321,"hooked":13322,"oct":13323,"tapes":13324,"judging":13325,"cody":13326,"##gger":13327,"hansen":13328,"kashmir":13329,"triggered":13330,"##eva":13331,"solved":13332,"cliffs":13333,"##tree":13334,"resisted":13335,"anatomy":13336,"protesters":13337,"transparent":13338,"implied":13339,"##iga":13340,"injection":13341,"mattress":13342,"excluding":13343,"##mbo":13344,"defenses":13345,"helpless":13346,"devotion":13347,"##elli":13348,"growl":13349,"liberals":13350,"weber":13351,"phenomena":13352,"atoms":13353,"plug":13354,"##iff":13355,"mortality":13356,"apprentice":13357,"howe":13358,"convincing":13359,"aaa":13360,"swimmer":13361,"barber":13362,"leone":13363,"promptly":13364,"sodium":13365,"def":13366,"nowadays":13367,"arise":13368,"##oning":13369,"gloucester":13370,"corrected":13371,"dignity":13372,"norm":13373,"erie":13374,"##ders":13375,"elders":13376,"evacuated":13377,"sylvia":13378,"compression":13379,"##yar":13380,"hartford":13381,"pose":13382,"backpack":13383,"reasoning":13384,"accepts":13385,"24th":13386,"wipe":13387,"millimetres":13388,"marcel":13389,"##oda":13390,"dodgers":13391,"albion":13392,"1790":13393,"overwhelmed":13394,"aerospace":13395,"oaks":13396,"1795":13397,"showcase":13398,"acknowledge":13399,"recovering":13400,"nolan":13401,"ashe":13402,"hurts":13403,"geology":13404,"fashioned":13405,"disappearance":13406,"farewell":13407,"swollen":13408,"shrug":13409,"marquis":13410,"wimbledon":13411,"124":13412,"rue":13413,"1792":13414,"commemorate":13415,"reduces":13416,"experiencing":13417,"inevitable":13418,"calcutta":13419,"intel":13420,"##court":13421,"murderer":13422,"sticking":13423,"fisheries":13424,"imagery":13425,"bloom":13426,"280":13427,"brake":13428,"##inus":13429,"gustav":13430,"hesitation":13431,"memorable":13432,"po":13433,"viral":13434,"beans":13435,"accidents":13436,"tunisia":13437,"antenna":13438,"spilled":13439,"consort":13440,"treatments":13441,"aye":13442,"perimeter":13443,"##gard":13444,"donation":13445,"hostage":13446,"migrated":13447,"banker":13448,"addiction":13449,"apex":13450,"lil":13451,"trout":13452,"##ously":13453,"conscience":13454,"##nova":13455,"rams":13456,"sands":13457,"genome":13458,"passionate":13459,"troubles":13460,"##lets":13461,"##set":13462,"amid":13463,"##ibility":13464,"##ret":13465,"higgins":13466,"exceed":13467,"vikings":13468,"##vie":13469,"payne":13470,"##zan":13471,"muscular":13472,"##ste":13473,"defendant":13474,"sucking":13475,"##wal":13476,"ibrahim":13477,"fuselage":13478,"claudia":13479,"vfl":13480,"europeans":13481,"snails":13482,"interval":13483,"##garh":13484,"preparatory":13485,"statewide":13486,"tasked":13487,"lacrosse":13488,"viktor":13489,"##lation":13490,"angola":13491,"##hra":13492,"flint":13493,"implications":13494,"employs":13495,"teens":13496,"patrons":13497,"stall":13498,"weekends":13499,"barriers":13500,"scrambled":13501,"nucleus":13502,"tehran":13503,"jenna":13504,"parsons":13505,"lifelong":13506,"robots":13507,"displacement":13508,"5000":13509,"##bles":13510,"precipitation":13511,"##gt":13512,"knuckles":13513,"clutched":13514,"1802":13515,"marrying":13516,"ecology":13517,"marx":13518,"accusations":13519,"declare":13520,"scars":13521,"kolkata":13522,"mat":13523,"meadows":13524,"bermuda":13525,"skeleton":13526,"finalists":13527,"vintage":13528,"crawl":13529,"coordinate":13530,"affects":13531,"subjected":13532,"orchestral":13533,"mistaken":13534,"##tc":13535,"mirrors":13536,"dipped":13537,"relied":13538,"260":13539,"arches":13540,"candle":13541,"##nick":13542,"incorporating":13543,"wildly":13544,"fond":13545,"basilica":13546,"owl":13547,"fringe":13548,"rituals":13549,"whispering":13550,"stirred":13551,"feud":13552,"tertiary":13553,"slick":13554,"goat":13555,"honorable":13556,"whereby":13557,"skip":13558,"ricardo":13559,"stripes":13560,"parachute":13561,"adjoining":13562,"submerged":13563,"synthesizer":13564,"##gren":13565,"intend":13566,"positively":13567,"ninety":13568,"phi":13569,"beaver":13570,"partition":13571,"fellows":13572,"alexis":13573,"prohibition":13574,"carlisle":13575,"bizarre":13576,"fraternity":13577,"##bre":13578,"doubts":13579,"icy":13580,"cbc":13581,"aquatic":13582,"sneak":13583,"sonny":13584,"combines":13585,"airports":13586,"crude":13587,"supervised":13588,"spatial":13589,"merge":13590,"alfonso":13591,"##bic":13592,"corrupt":13593,"scan":13594,"undergo":13595,"##ams":13596,"disabilities":13597,"colombian":13598,"comparing":13599,"dolphins":13600,"perkins":13601,"##lish":13602,"reprinted":13603,"unanimous":13604,"bounced":13605,"hairs":13606,"underworld":13607,"midwest":13608,"semester":13609,"bucket":13610,"paperback":13611,"miniseries":13612,"coventry":13613,"demise":13614,"##leigh":13615,"demonstrations":13616,"sensor":13617,"rotating":13618,"yan":13619,"##hler":13620,"arrange":13621,"soils":13622,"##idge":13623,"hyderabad":13624,"labs":13625,"##dr":13626,"brakes":13627,"grandchildren":13628,"##nde":13629,"negotiated":13630,"rover":13631,"ferrari":13632,"continuation":13633,"directorate":13634,"augusta":13635,"stevenson":13636,"counterpart":13637,"gore":13638,"##rda":13639,"nursery":13640,"rican":13641,"ave":13642,"collectively":13643,"broadly":13644,"pastoral":13645,"repertoire":13646,"asserted":13647,"discovering":13648,"nordic":13649,"styled":13650,"fiba":13651,"cunningham":13652,"harley":13653,"middlesex":13654,"survives":13655,"tumor":13656,"tempo":13657,"zack":13658,"aiming":13659,"lok":13660,"urgent":13661,"##rade":13662,"##nto":13663,"devils":13664,"##ement":13665,"contractor":13666,"turin":13667,"##wl":13668,"##ool":13669,"bliss":13670,"repaired":13671,"simmons":13672,"moan":13673,"astronomical":13674,"cr":13675,"negotiate":13676,"lyric":13677,"1890s":13678,"lara":13679,"bred":13680,"clad":13681,"angus":13682,"pbs":13683,"##ience":13684,"engineered":13685,"posed":13686,"##lk":13687,"hernandez":13688,"possessions":13689,"elbows":13690,"psychiatric":13691,"strokes":13692,"confluence":13693,"electorate":13694,"lifts":13695,"campuses":13696,"lava":13697,"alps":13698,"##ep":13699,"##ution":13700,"##date":13701,"physicist":13702,"woody":13703,"##page":13704,"##ographic":13705,"##itis":13706,"juliet":13707,"reformation":13708,"sparhawk":13709,"320":13710,"complement":13711,"suppressed":13712,"jewel":13713,"##½":13714,"floated":13715,"##kas":13716,"continuity":13717,"sadly":13718,"##ische":13719,"inability":13720,"melting":13721,"scanning":13722,"paula":13723,"flour":13724,"judaism":13725,"safer":13726,"vague":13727,"##lm":13728,"solving":13729,"curb":13730,"##stown":13731,"financially":13732,"gable":13733,"bees":13734,"expired":13735,"miserable":13736,"cassidy":13737,"dominion":13738,"1789":13739,"cupped":13740,"145":13741,"robbery":13742,"facto":13743,"amos":13744,"warden":13745,"resume":13746,"tallest":13747,"marvin":13748,"ing":13749,"pounded":13750,"usd":13751,"declaring":13752,"gasoline":13753,"##aux":13754,"darkened":13755,"270":13756,"650":13757,"sophomore":13758,"##mere":13759,"erection":13760,"gossip":13761,"televised":13762,"risen":13763,"dial":13764,"##eu":13765,"pillars":13766,"##link":13767,"passages":13768,"profound":13769,"##tina":13770,"arabian":13771,"ashton":13772,"silicon":13773,"nail":13774,"##ead":13775,"##lated":13776,"##wer":13777,"##hardt":13778,"fleming":13779,"firearms":13780,"ducked":13781,"circuits":13782,"blows":13783,"waterloo":13784,"titans":13785,"##lina":13786,"atom":13787,"fireplace":13788,"cheshire":13789,"financed":13790,"activation":13791,"algorithms":13792,"##zzi":13793,"constituent":13794,"catcher":13795,"cherokee":13796,"partnerships":13797,"sexuality":13798,"platoon":13799,"tragic":13800,"vivian":13801,"guarded":13802,"whiskey":13803,"meditation":13804,"poetic":13805,"##late":13806,"##nga":13807,"##ake":13808,"porto":13809,"listeners":13810,"dominance":13811,"kendra":13812,"mona":13813,"chandler":13814,"factions":13815,"22nd":13816,"salisbury":13817,"attitudes":13818,"derivative":13819,"##ido":13820,"##haus":13821,"intake":13822,"paced":13823,"javier":13824,"illustrator":13825,"barrels":13826,"bias":13827,"cockpit":13828,"burnett":13829,"dreamed":13830,"ensuing":13831,"##anda":13832,"receptors":13833,"someday":13834,"hawkins":13835,"mattered":13836,"##lal":13837,"slavic":13838,"1799":13839,"jesuit":13840,"cameroon":13841,"wasted":13842,"tai":13843,"wax":13844,"lowering":13845,"victorious":13846,"freaking":13847,"outright":13848,"hancock":13849,"librarian":13850,"sensing":13851,"bald":13852,"calcium":13853,"myers":13854,"tablet":13855,"announcing":13856,"barack":13857,"shipyard":13858,"pharmaceutical":13859,"##uan":13860,"greenwich":13861,"flush":13862,"medley":13863,"patches":13864,"wolfgang":13865,"pt":13866,"speeches":13867,"acquiring":13868,"exams":13869,"nikolai":13870,"##gg":13871,"hayden":13872,"kannada":13873,"##type":13874,"reilly":13875,"##pt":13876,"waitress":13877,"abdomen":13878,"devastated":13879,"capped":13880,"pseudonym":13881,"pharmacy":13882,"fulfill":13883,"paraguay":13884,"1796":13885,"clicked":13886,"##trom":13887,"archipelago":13888,"syndicated":13889,"##hman":13890,"lumber":13891,"orgasm":13892,"rejection":13893,"clifford":13894,"lorraine":13895,"advent":13896,"mafia":13897,"rodney":13898,"brock":13899,"##ght":13900,"##used":13901,"##elia":13902,"cassette":13903,"chamberlain":13904,"despair":13905,"mongolia":13906,"sensors":13907,"developmental":13908,"upstream":13909,"##eg":13910,"##alis":13911,"spanning":13912,"165":13913,"trombone":13914,"basque":13915,"seeded":13916,"interred":13917,"renewable":13918,"rhys":13919,"leapt":13920,"revision":13921,"molecule":13922,"##ages":13923,"chord":13924,"vicious":13925,"nord":13926,"shivered":13927,"23rd":13928,"arlington":13929,"debts":13930,"corpus":13931,"sunrise":13932,"bays":13933,"blackburn":13934,"centimetres":13935,"##uded":13936,"shuddered":13937,"gm":13938,"strangely":13939,"gripping":13940,"cartoons":13941,"isabelle":13942,"orbital":13943,"##ppa":13944,"seals":13945,"proving":13946,"##lton":13947,"refusal":13948,"strengthened":13949,"bust":13950,"assisting":13951,"baghdad":13952,"batsman":13953,"portrayal":13954,"mara":13955,"pushes":13956,"spears":13957,"og":13958,"##cock":13959,"reside":13960,"nathaniel":13961,"brennan":13962,"1776":13963,"confirmation":13964,"caucus":13965,"##worthy":13966,"markings":13967,"yemen":13968,"nobles":13969,"ku":13970,"lazy":13971,"viewer":13972,"catalan":13973,"encompasses":13974,"sawyer":13975,"##fall":13976,"sparked":13977,"substances":13978,"patents":13979,"braves":13980,"arranger":13981,"evacuation":13982,"sergio":13983,"persuade":13984,"dover":13985,"tolerance":13986,"penguin":13987,"cum":13988,"jockey":13989,"insufficient":13990,"townships":13991,"occupying":13992,"declining":13993,"plural":13994,"processed":13995,"projection":13996,"puppet":13997,"flanders":13998,"introduces":13999,"liability":14000,"##yon":14001,"gymnastics":14002,"antwerp":14003,"taipei":14004,"hobart":14005,"candles":14006,"jeep":14007,"wes":14008,"observers":14009,"126":14010,"chaplain":14011,"bundle":14012,"glorious":14013,"##hine":14014,"hazel":14015,"flung":14016,"sol":14017,"excavations":14018,"dumped":14019,"stares":14020,"sh":14021,"bangalore":14022,"triangular":14023,"icelandic":14024,"intervals":14025,"expressing":14026,"turbine":14027,"##vers":14028,"songwriting":14029,"crafts":14030,"##igo":14031,"jasmine":14032,"ditch":14033,"rite":14034,"##ways":14035,"entertaining":14036,"comply":14037,"sorrow":14038,"wrestlers":14039,"basel":14040,"emirates":14041,"marian":14042,"rivera":14043,"helpful":14044,"##some":14045,"caution":14046,"downward":14047,"networking":14048,"##atory":14049,"##tered":14050,"darted":14051,"genocide":14052,"emergence":14053,"replies":14054,"specializing":14055,"spokesman":14056,"convenient":14057,"unlocked":14058,"fading":14059,"augustine":14060,"concentrations":14061,"resemblance":14062,"elijah":14063,"investigator":14064,"andhra":14065,"##uda":14066,"promotes":14067,"bean":14068,"##rrell":14069,"fleeing":14070,"wan":14071,"simone":14072,"announcer":14073,"##ame":14074,"##bby":14075,"lydia":14076,"weaver":14077,"132":14078,"residency":14079,"modification":14080,"##fest":14081,"stretches":14082,"##ast":14083,"alternatively":14084,"nat":14085,"lowe":14086,"lacks":14087,"##ented":14088,"pam":14089,"tile":14090,"concealed":14091,"inferior":14092,"abdullah":14093,"residences":14094,"tissues":14095,"vengeance":14096,"##ided":14097,"moisture":14098,"peculiar":14099,"groove":14100,"zip":14101,"bologna":14102,"jennings":14103,"ninja":14104,"oversaw":14105,"zombies":14106,"pumping":14107,"batch":14108,"livingston":14109,"emerald":14110,"installations":14111,"1797":14112,"peel":14113,"nitrogen":14114,"rama":14115,"##fying":14116,"##star":14117,"schooling":14118,"strands":14119,"responding":14120,"werner":14121,"##ost":14122,"lime":14123,"casa":14124,"accurately":14125,"targeting":14126,"##rod":14127,"underway":14128,"##uru":14129,"hemisphere":14130,"lester":14131,"##yard":14132,"occupies":14133,"2d":14134,"griffith":14135,"angrily":14136,"reorganized":14137,"##owing":14138,"courtney":14139,"deposited":14140,"##dd":14141,"##30":14142,"estadio":14143,"##ifies":14144,"dunn":14145,"exiled":14146,"##ying":14147,"checks":14148,"##combe":14149,"##о":14150,"##fly":14151,"successes":14152,"unexpectedly":14153,"blu":14154,"assessed":14155,"##flower":14156,"##ه":14157,"observing":14158,"sacked":14159,"spiders":14160,"kn":14161,"##tail":14162,"mu":14163,"nodes":14164,"prosperity":14165,"audrey":14166,"divisional":14167,"155":14168,"broncos":14169,"tangled":14170,"adjust":14171,"feeds":14172,"erosion":14173,"paolo":14174,"surf":14175,"directory":14176,"snatched":14177,"humid":14178,"admiralty":14179,"screwed":14180,"gt":14181,"reddish":14182,"##nese":14183,"modules":14184,"trench":14185,"lamps":14186,"bind":14187,"leah":14188,"bucks":14189,"competes":14190,"##nz":14191,"##form":14192,"transcription":14193,"##uc":14194,"isles":14195,"violently":14196,"clutching":14197,"pga":14198,"cyclist":14199,"inflation":14200,"flats":14201,"ragged":14202,"unnecessary":14203,"##hian":14204,"stubborn":14205,"coordinated":14206,"harriet":14207,"baba":14208,"disqualified":14209,"330":14210,"insect":14211,"wolfe":14212,"##fies":14213,"reinforcements":14214,"rocked":14215,"duel":14216,"winked":14217,"embraced":14218,"bricks":14219,"##raj":14220,"hiatus":14221,"defeats":14222,"pending":14223,"brightly":14224,"jealousy":14225,"##xton":14226,"##hm":14227,"##uki":14228,"lena":14229,"gdp":14230,"colorful":14231,"##dley":14232,"stein":14233,"kidney":14234,"##shu":14235,"underwear":14236,"wanderers":14237,"##haw":14238,"##icus":14239,"guardians":14240,"m³":14241,"roared":14242,"habits":14243,"##wise":14244,"permits":14245,"gp":14246,"uranium":14247,"punished":14248,"disguise":14249,"bundesliga":14250,"elise":14251,"dundee":14252,"erotic":14253,"partisan":14254,"pi":14255,"collectors":14256,"float":14257,"individually":14258,"rendering":14259,"behavioral":14260,"bucharest":14261,"ser":14262,"hare":14263,"valerie":14264,"corporal":14265,"nutrition":14266,"proportional":14267,"##isa":14268,"immense":14269,"##kis":14270,"pavement":14271,"##zie":14272,"##eld":14273,"sutherland":14274,"crouched":14275,"1775":14276,"##lp":14277,"suzuki":14278,"trades":14279,"endurance":14280,"operas":14281,"crosby":14282,"prayed":14283,"priory":14284,"rory":14285,"socially":14286,"##urn":14287,"gujarat":14288,"##pu":14289,"walton":14290,"cube":14291,"pasha":14292,"privilege":14293,"lennon":14294,"floods":14295,"thorne":14296,"waterfall":14297,"nipple":14298,"scouting":14299,"approve":14300,"##lov":14301,"minorities":14302,"voter":14303,"dwight":14304,"extensions":14305,"assure":14306,"ballroom":14307,"slap":14308,"dripping":14309,"privileges":14310,"rejoined":14311,"confessed":14312,"demonstrating":14313,"patriotic":14314,"yell":14315,"investor":14316,"##uth":14317,"pagan":14318,"slumped":14319,"squares":14320,"##cle":14321,"##kins":14322,"confront":14323,"bert":14324,"embarrassment":14325,"##aid":14326,"aston":14327,"urging":14328,"sweater":14329,"starr":14330,"yuri":14331,"brains":14332,"williamson":14333,"commuter":14334,"mortar":14335,"structured":14336,"selfish":14337,"exports":14338,"##jon":14339,"cds":14340,"##him":14341,"unfinished":14342,"##rre":14343,"mortgage":14344,"destinations":14345,"##nagar":14346,"canoe":14347,"solitary":14348,"buchanan":14349,"delays":14350,"magistrate":14351,"fk":14352,"##pling":14353,"motivation":14354,"##lier":14355,"##vier":14356,"recruiting":14357,"assess":14358,"##mouth":14359,"malik":14360,"antique":14361,"1791":14362,"pius":14363,"rahman":14364,"reich":14365,"tub":14366,"zhou":14367,"smashed":14368,"airs":14369,"galway":14370,"xii":14371,"conditioning":14372,"honduras":14373,"discharged":14374,"dexter":14375,"##pf":14376,"lionel":14377,"129":14378,"debates":14379,"lemon":14380,"tiffany":14381,"volunteered":14382,"dom":14383,"dioxide":14384,"procession":14385,"devi":14386,"sic":14387,"tremendous":14388,"advertisements":14389,"colts":14390,"transferring":14391,"verdict":14392,"hanover":14393,"decommissioned":14394,"utter":14395,"relate":14396,"pac":14397,"racism":14398,"##top":14399,"beacon":14400,"limp":14401,"similarity":14402,"terra":14403,"occurrence":14404,"ant":14405,"##how":14406,"becky":14407,"capt":14408,"updates":14409,"armament":14410,"richie":14411,"pal":14412,"##graph":14413,"halloween":14414,"mayo":14415,"##ssen":14416,"##bone":14417,"cara":14418,"serena":14419,"fcc":14420,"dolls":14421,"obligations":14422,"##dling":14423,"violated":14424,"lafayette":14425,"jakarta":14426,"exploitation":14427,"##ime":14428,"infamous":14429,"iconic":14430,"##lah":14431,"##park":14432,"kitty":14433,"moody":14434,"reginald":14435,"dread":14436,"spill":14437,"crystals":14438,"olivier":14439,"modeled":14440,"bluff":14441,"equilibrium":14442,"separating":14443,"notices":14444,"ordnance":14445,"extinction":14446,"onset":14447,"cosmic":14448,"attachment":14449,"sammy":14450,"expose":14451,"privy":14452,"anchored":14453,"##bil":14454,"abbott":14455,"admits":14456,"bending":14457,"baritone":14458,"emmanuel":14459,"policeman":14460,"vaughan":14461,"winged":14462,"climax":14463,"dresses":14464,"denny":14465,"polytechnic":14466,"mohamed":14467,"burmese":14468,"authentic":14469,"nikki":14470,"genetics":14471,"grandparents":14472,"homestead":14473,"gaza":14474,"postponed":14475,"metacritic":14476,"una":14477,"##sby":14478,"##bat":14479,"unstable":14480,"dissertation":14481,"##rial":14482,"##cian":14483,"curls":14484,"obscure":14485,"uncovered":14486,"bronx":14487,"praying":14488,"disappearing":14489,"##hoe":14490,"prehistoric":14491,"coke":14492,"turret":14493,"mutations":14494,"nonprofit":14495,"pits":14496,"monaco":14497,"##ي":14498,"##usion":14499,"prominently":14500,"dispatched":14501,"podium":14502,"##mir":14503,"uci":14504,"##uation":14505,"133":14506,"fortifications":14507,"birthplace":14508,"kendall":14509,"##lby":14510,"##oll":14511,"preacher":14512,"rack":14513,"goodman":14514,"##rman":14515,"persistent":14516,"##ott":14517,"countless":14518,"jaime":14519,"recorder":14520,"lexington":14521,"persecution":14522,"jumps":14523,"renewal":14524,"wagons":14525,"##11":14526,"crushing":14527,"##holder":14528,"decorations":14529,"##lake":14530,"abundance":14531,"wrath":14532,"laundry":14533,"£1":14534,"garde":14535,"##rp":14536,"jeanne":14537,"beetles":14538,"peasant":14539,"##sl":14540,"splitting":14541,"caste":14542,"sergei":14543,"##rer":14544,"##ema":14545,"scripts":14546,"##ively":14547,"rub":14548,"satellites":14549,"##vor":14550,"inscribed":14551,"verlag":14552,"scrapped":14553,"gale":14554,"packages":14555,"chick":14556,"potato":14557,"slogan":14558,"kathleen":14559,"arabs":14560,"##culture":14561,"counterparts":14562,"reminiscent":14563,"choral":14564,"##tead":14565,"rand":14566,"retains":14567,"bushes":14568,"dane":14569,"accomplish":14570,"courtesy":14571,"closes":14572,"##oth":14573,"slaughter":14574,"hague":14575,"krakow":14576,"lawson":14577,"tailed":14578,"elias":14579,"ginger":14580,"##ttes":14581,"canopy":14582,"betrayal":14583,"rebuilding":14584,"turf":14585,"##hof":14586,"frowning":14587,"allegiance":14588,"brigades":14589,"kicks":14590,"rebuild":14591,"polls":14592,"alias":14593,"nationalism":14594,"td":14595,"rowan":14596,"audition":14597,"bowie":14598,"fortunately":14599,"recognizes":14600,"harp":14601,"dillon":14602,"horrified":14603,"##oro":14604,"renault":14605,"##tics":14606,"ropes":14607,"##α":14608,"presumed":14609,"rewarded":14610,"infrared":14611,"wiping":14612,"accelerated":14613,"illustration":14614,"##rid":14615,"presses":14616,"practitioners":14617,"badminton":14618,"##iard":14619,"detained":14620,"##tera":14621,"recognizing":14622,"relates":14623,"misery":14624,"##sies":14625,"##tly":14626,"reproduction":14627,"piercing":14628,"potatoes":14629,"thornton":14630,"esther":14631,"manners":14632,"hbo":14633,"##aan":14634,"ours":14635,"bullshit":14636,"ernie":14637,"perennial":14638,"sensitivity":14639,"illuminated":14640,"rupert":14641,"##jin":14642,"##iss":14643,"##ear":14644,"rfc":14645,"nassau":14646,"##dock":14647,"staggered":14648,"socialism":14649,"##haven":14650,"appointments":14651,"nonsense":14652,"prestige":14653,"sharma":14654,"haul":14655,"##tical":14656,"solidarity":14657,"gps":14658,"##ook":14659,"##rata":14660,"igor":14661,"pedestrian":14662,"##uit":14663,"baxter":14664,"tenants":14665,"wires":14666,"medication":14667,"unlimited":14668,"guiding":14669,"impacts":14670,"diabetes":14671,"##rama":14672,"sasha":14673,"pas":14674,"clive":14675,"extraction":14676,"131":14677,"continually":14678,"constraints":14679,"##bilities":14680,"sonata":14681,"hunted":14682,"sixteenth":14683,"chu":14684,"planting":14685,"quote":14686,"mayer":14687,"pretended":14688,"abs":14689,"spat":14690,"##hua":14691,"ceramic":14692,"##cci":14693,"curtains":14694,"pigs":14695,"pitching":14696,"##dad":14697,"latvian":14698,"sore":14699,"dayton":14700,"##sted":14701,"##qi":14702,"patrols":14703,"slice":14704,"playground":14705,"##nted":14706,"shone":14707,"stool":14708,"apparatus":14709,"inadequate":14710,"mates":14711,"treason":14712,"##ija":14713,"desires":14714,"##liga":14715,"##croft":14716,"somalia":14717,"laurent":14718,"mir":14719,"leonardo":14720,"oracle":14721,"grape":14722,"obliged":14723,"chevrolet":14724,"thirteenth":14725,"stunning":14726,"enthusiastic":14727,"##ede":14728,"accounted":14729,"concludes":14730,"currents":14731,"basil":14732,"##kovic":14733,"drought":14734,"##rica":14735,"mai":14736,"##aire":14737,"shove":14738,"posting":14739,"##shed":14740,"pilgrimage":14741,"humorous":14742,"packing":14743,"fry":14744,"pencil":14745,"wines":14746,"smells":14747,"144":14748,"marilyn":14749,"aching":14750,"newest":14751,"clung":14752,"bon":14753,"neighbours":14754,"sanctioned":14755,"##pie":14756,"mug":14757,"##stock":14758,"drowning":14759,"##mma":14760,"hydraulic":14761,"##vil":14762,"hiring":14763,"reminder":14764,"lilly":14765,"investigators":14766,"##ncies":14767,"sour":14768,"##eous":14769,"compulsory":14770,"packet":14771,"##rion":14772,"##graphic":14773,"##elle":14774,"cannes":14775,"##inate":14776,"depressed":14777,"##rit":14778,"heroic":14779,"importantly":14780,"theresa":14781,"##tled":14782,"conway":14783,"saturn":14784,"marginal":14785,"rae":14786,"##xia":14787,"corresponds":14788,"royce":14789,"pact":14790,"jasper":14791,"explosives":14792,"packaging":14793,"aluminium":14794,"##ttered":14795,"denotes":14796,"rhythmic":14797,"spans":14798,"assignments":14799,"hereditary":14800,"outlined":14801,"originating":14802,"sundays":14803,"lad":14804,"reissued":14805,"greeting":14806,"beatrice":14807,"##dic":14808,"pillar":14809,"marcos":14810,"plots":14811,"handbook":14812,"alcoholic":14813,"judiciary":14814,"avant":14815,"slides":14816,"extract":14817,"masculine":14818,"blur":14819,"##eum":14820,"##force":14821,"homage":14822,"trembled":14823,"owens":14824,"hymn":14825,"trey":14826,"omega":14827,"signaling":14828,"socks":14829,"accumulated":14830,"reacted":14831,"attic":14832,"theo":14833,"lining":14834,"angie":14835,"distraction":14836,"primera":14837,"talbot":14838,"##key":14839,"1200":14840,"ti":14841,"creativity":14842,"billed":14843,"##hey":14844,"deacon":14845,"eduardo":14846,"identifies":14847,"proposition":14848,"dizzy":14849,"gunner":14850,"hogan":14851,"##yam":14852,"##pping":14853,"##hol":14854,"ja":14855,"##chan":14856,"jensen":14857,"reconstructed":14858,"##berger":14859,"clearance":14860,"darius":14861,"##nier":14862,"abe":14863,"harlem":14864,"plea":14865,"dei":14866,"circled":14867,"emotionally":14868,"notation":14869,"fascist":14870,"neville":14871,"exceeded":14872,"upwards":14873,"viable":14874,"ducks":14875,"##fo":14876,"workforce":14877,"racer":14878,"limiting":14879,"shri":14880,"##lson":14881,"possesses":14882,"1600":14883,"kerr":14884,"moths":14885,"devastating":14886,"laden":14887,"disturbing":14888,"locking":14889,"##cture":14890,"gal":14891,"fearing":14892,"accreditation":14893,"flavor":14894,"aide":14895,"1870s":14896,"mountainous":14897,"##baum":14898,"melt":14899,"##ures":14900,"motel":14901,"texture":14902,"servers":14903,"soda":14904,"##mb":14905,"herd":14906,"##nium":14907,"erect":14908,"puzzled":14909,"hum":14910,"peggy":14911,"examinations":14912,"gould":14913,"testified":14914,"geoff":14915,"ren":14916,"devised":14917,"sacks":14918,"##law":14919,"denial":14920,"posters":14921,"grunted":14922,"cesar":14923,"tutor":14924,"ec":14925,"gerry":14926,"offerings":14927,"byrne":14928,"falcons":14929,"combinations":14930,"ct":14931,"incoming":14932,"pardon":14933,"rocking":14934,"26th":14935,"avengers":14936,"flared":14937,"mankind":14938,"seller":14939,"uttar":14940,"loch":14941,"nadia":14942,"stroking":14943,"exposing":14944,"##hd":14945,"fertile":14946,"ancestral":14947,"instituted":14948,"##has":14949,"noises":14950,"prophecy":14951,"taxation":14952,"eminent":14953,"vivid":14954,"pol":14955,"##bol":14956,"dart":14957,"indirect":14958,"multimedia":14959,"notebook":14960,"upside":14961,"displaying":14962,"adrenaline":14963,"referenced":14964,"geometric":14965,"##iving":14966,"progression":14967,"##ddy":14968,"blunt":14969,"announce":14970,"##far":14971,"implementing":14972,"##lav":14973,"aggression":14974,"liaison":14975,"cooler":14976,"cares":14977,"headache":14978,"plantations":14979,"gorge":14980,"dots":14981,"impulse":14982,"thickness":14983,"ashamed":14984,"averaging":14985,"kathy":14986,"obligation":14987,"precursor":14988,"137":14989,"fowler":14990,"symmetry":14991,"thee":14992,"225":14993,"hears":14994,"##rai":14995,"undergoing":14996,"ads":14997,"butcher":14998,"bowler":14999,"##lip":15000,"cigarettes":15001,"subscription":15002,"goodness":15003,"##ically":15004,"browne":15005,"##hos":15006,"##tech":15007,"kyoto":15008,"donor":15009,"##erty":15010,"damaging":15011,"friction":15012,"drifting":15013,"expeditions":15014,"hardened":15015,"prostitution":15016,"152":15017,"fauna":15018,"blankets":15019,"claw":15020,"tossing":15021,"snarled":15022,"butterflies":15023,"recruits":15024,"investigative":15025,"coated":15026,"healed":15027,"138":15028,"communal":15029,"hai":15030,"xiii":15031,"academics":15032,"boone":15033,"psychologist":15034,"restless":15035,"lahore":15036,"stephens":15037,"mba":15038,"brendan":15039,"foreigners":15040,"printer":15041,"##pc":15042,"ached":15043,"explode":15044,"27th":15045,"deed":15046,"scratched":15047,"dared":15048,"##pole":15049,"cardiac":15050,"1780":15051,"okinawa":15052,"proto":15053,"commando":15054,"compelled":15055,"oddly":15056,"electrons":15057,"##base":15058,"replica":15059,"thanksgiving":15060,"##rist":15061,"sheila":15062,"deliberate":15063,"stafford":15064,"tidal":15065,"representations":15066,"hercules":15067,"ou":15068,"##path":15069,"##iated":15070,"kidnapping":15071,"lenses":15072,"##tling":15073,"deficit":15074,"samoa":15075,"mouths":15076,"consuming":15077,"computational":15078,"maze":15079,"granting":15080,"smirk":15081,"razor":15082,"fixture":15083,"ideals":15084,"inviting":15085,"aiden":15086,"nominal":15087,"##vs":15088,"issuing":15089,"julio":15090,"pitt":15091,"ramsey":15092,"docks":15093,"##oss":15094,"exhaust":15095,"##owed":15096,"bavarian":15097,"draped":15098,"anterior":15099,"mating":15100,"ethiopian":15101,"explores":15102,"noticing":15103,"##nton":15104,"discarded":15105,"convenience":15106,"hoffman":15107,"endowment":15108,"beasts":15109,"cartridge":15110,"mormon":15111,"paternal":15112,"probe":15113,"sleeves":15114,"interfere":15115,"lump":15116,"deadline":15117,"##rail":15118,"jenks":15119,"bulldogs":15120,"scrap":15121,"alternating":15122,"justified":15123,"reproductive":15124,"nam":15125,"seize":15126,"descending":15127,"secretariat":15128,"kirby":15129,"coupe":15130,"grouped":15131,"smash":15132,"panther":15133,"sedan":15134,"tapping":15135,"##18":15136,"lola":15137,"cheer":15138,"germanic":15139,"unfortunate":15140,"##eter":15141,"unrelated":15142,"##fan":15143,"subordinate":15144,"##sdale":15145,"suzanne":15146,"advertisement":15147,"##ility":15148,"horsepower":15149,"##lda":15150,"cautiously":15151,"discourse":15152,"luigi":15153,"##mans":15154,"##fields":15155,"noun":15156,"prevalent":15157,"mao":15158,"schneider":15159,"everett":15160,"surround":15161,"governorate":15162,"kira":15163,"##avia":15164,"westward":15165,"##take":15166,"misty":15167,"rails":15168,"sustainability":15169,"134":15170,"unused":15171,"##rating":15172,"packs":15173,"toast":15174,"unwilling":15175,"regulate":15176,"thy":15177,"suffrage":15178,"nile":15179,"awe":15180,"assam":15181,"definitions":15182,"travelers":15183,"affordable":15184,"##rb":15185,"conferred":15186,"sells":15187,"undefeated":15188,"beneficial":15189,"torso":15190,"basal":15191,"repeating":15192,"remixes":15193,"##pass":15194,"bahrain":15195,"cables":15196,"fang":15197,"##itated":15198,"excavated":15199,"numbering":15200,"statutory":15201,"##rey":15202,"deluxe":15203,"##lian":15204,"forested":15205,"ramirez":15206,"derbyshire":15207,"zeus":15208,"slamming":15209,"transfers":15210,"astronomer":15211,"banana":15212,"lottery":15213,"berg":15214,"histories":15215,"bamboo":15216,"##uchi":15217,"resurrection":15218,"posterior":15219,"bowls":15220,"vaguely":15221,"##thi":15222,"thou":15223,"preserving":15224,"tensed":15225,"offence":15226,"##inas":15227,"meyrick":15228,"callum":15229,"ridden":15230,"watt":15231,"langdon":15232,"tying":15233,"lowland":15234,"snorted":15235,"daring":15236,"truman":15237,"##hale":15238,"##girl":15239,"aura":15240,"overly":15241,"filing":15242,"weighing":15243,"goa":15244,"infections":15245,"philanthropist":15246,"saunders":15247,"eponymous":15248,"##owski":15249,"latitude":15250,"perspectives":15251,"reviewing":15252,"mets":15253,"commandant":15254,"radial":15255,"##kha":15256,"flashlight":15257,"reliability":15258,"koch":15259,"vowels":15260,"amazed":15261,"ada":15262,"elaine":15263,"supper":15264,"##rth":15265,"##encies":15266,"predator":15267,"debated":15268,"soviets":15269,"cola":15270,"##boards":15271,"##nah":15272,"compartment":15273,"crooked":15274,"arbitrary":15275,"fourteenth":15276,"##ctive":15277,"havana":15278,"majors":15279,"steelers":15280,"clips":15281,"profitable":15282,"ambush":15283,"exited":15284,"packers":15285,"##tile":15286,"nude":15287,"cracks":15288,"fungi":15289,"##е":15290,"limb":15291,"trousers":15292,"josie":15293,"shelby":15294,"tens":15295,"frederic":15296,"##ος":15297,"definite":15298,"smoothly":15299,"constellation":15300,"insult":15301,"baton":15302,"discs":15303,"lingering":15304,"##nco":15305,"conclusions":15306,"lent":15307,"staging":15308,"becker":15309,"grandpa":15310,"shaky":15311,"##tron":15312,"einstein":15313,"obstacles":15314,"sk":15315,"adverse":15316,"elle":15317,"economically":15318,"##moto":15319,"mccartney":15320,"thor":15321,"dismissal":15322,"motions":15323,"readings":15324,"nostrils":15325,"treatise":15326,"##pace":15327,"squeezing":15328,"evidently":15329,"prolonged":15330,"1783":15331,"venezuelan":15332,"je":15333,"marguerite":15334,"beirut":15335,"takeover":15336,"shareholders":15337,"##vent":15338,"denise":15339,"digit":15340,"airplay":15341,"norse":15342,"##bbling":15343,"imaginary":15344,"pills":15345,"hubert":15346,"blaze":15347,"vacated":15348,"eliminating":15349,"##ello":15350,"vine":15351,"mansfield":15352,"##tty":15353,"retrospective":15354,"barrow":15355,"borne":15356,"clutch":15357,"bail":15358,"forensic":15359,"weaving":15360,"##nett":15361,"##witz":15362,"desktop":15363,"citadel":15364,"promotions":15365,"worrying":15366,"dorset":15367,"ieee":15368,"subdivided":15369,"##iating":15370,"manned":15371,"expeditionary":15372,"pickup":15373,"synod":15374,"chuckle":15375,"185":15376,"barney":15377,"##rz":15378,"##ffin":15379,"functionality":15380,"karachi":15381,"litigation":15382,"meanings":15383,"uc":15384,"lick":15385,"turbo":15386,"anders":15387,"##ffed":15388,"execute":15389,"curl":15390,"oppose":15391,"ankles":15392,"typhoon":15393,"##د":15394,"##ache":15395,"##asia":15396,"linguistics":15397,"compassion":15398,"pressures":15399,"grazing":15400,"perfection":15401,"##iting":15402,"immunity":15403,"monopoly":15404,"muddy":15405,"backgrounds":15406,"136":15407,"namibia":15408,"francesca":15409,"monitors":15410,"attracting":15411,"stunt":15412,"tuition":15413,"##ии":15414,"vegetable":15415,"##mates":15416,"##quent":15417,"mgm":15418,"jen":15419,"complexes":15420,"forts":15421,"##ond":15422,"cellar":15423,"bites":15424,"seventeenth":15425,"royals":15426,"flemish":15427,"failures":15428,"mast":15429,"charities":15430,"##cular":15431,"peruvian":15432,"capitals":15433,"macmillan":15434,"ipswich":15435,"outward":15436,"frigate":15437,"postgraduate":15438,"folds":15439,"employing":15440,"##ouse":15441,"concurrently":15442,"fiery":15443,"##tai":15444,"contingent":15445,"nightmares":15446,"monumental":15447,"nicaragua":15448,"##kowski":15449,"lizard":15450,"mal":15451,"fielding":15452,"gig":15453,"reject":15454,"##pad":15455,"harding":15456,"##ipe":15457,"coastline":15458,"##cin":15459,"##nos":15460,"beethoven":15461,"humphrey":15462,"innovations":15463,"##tam":15464,"##nge":15465,"norris":15466,"doris":15467,"solicitor":15468,"huang":15469,"obey":15470,"141":15471,"##lc":15472,"niagara":15473,"##tton":15474,"shelves":15475,"aug":15476,"bourbon":15477,"curry":15478,"nightclub":15479,"specifications":15480,"hilton":15481,"##ndo":15482,"centennial":15483,"dispersed":15484,"worm":15485,"neglected":15486,"briggs":15487,"sm":15488,"font":15489,"kuala":15490,"uneasy":15491,"plc":15492,"##nstein":15493,"##bound":15494,"##aking":15495,"##burgh":15496,"awaiting":15497,"pronunciation":15498,"##bbed":15499,"##quest":15500,"eh":15501,"optimal":15502,"zhu":15503,"raped":15504,"greens":15505,"presided":15506,"brenda":15507,"worries":15508,"##life":15509,"venetian":15510,"marxist":15511,"turnout":15512,"##lius":15513,"refined":15514,"braced":15515,"sins":15516,"grasped":15517,"sunderland":15518,"nickel":15519,"speculated":15520,"lowell":15521,"cyrillic":15522,"communism":15523,"fundraising":15524,"resembling":15525,"colonists":15526,"mutant":15527,"freddie":15528,"usc":15529,"##mos":15530,"gratitude":15531,"##run":15532,"mural":15533,"##lous":15534,"chemist":15535,"wi":15536,"reminds":15537,"28th":15538,"steals":15539,"tess":15540,"pietro":15541,"##ingen":15542,"promoter":15543,"ri":15544,"microphone":15545,"honoured":15546,"rai":15547,"sant":15548,"##qui":15549,"feather":15550,"##nson":15551,"burlington":15552,"kurdish":15553,"terrorists":15554,"deborah":15555,"sickness":15556,"##wed":15557,"##eet":15558,"hazard":15559,"irritated":15560,"desperation":15561,"veil":15562,"clarity":15563,"##rik":15564,"jewels":15565,"xv":15566,"##gged":15567,"##ows":15568,"##cup":15569,"berkshire":15570,"unfair":15571,"mysteries":15572,"orchid":15573,"winced":15574,"exhaustion":15575,"renovations":15576,"stranded":15577,"obe":15578,"infinity":15579,"##nies":15580,"adapt":15581,"redevelopment":15582,"thanked":15583,"registry":15584,"olga":15585,"domingo":15586,"noir":15587,"tudor":15588,"ole":15589,"##atus":15590,"commenting":15591,"behaviors":15592,"##ais":15593,"crisp":15594,"pauline":15595,"probable":15596,"stirling":15597,"wigan":15598,"##bian":15599,"paralympics":15600,"panting":15601,"surpassed":15602,"##rew":15603,"luca":15604,"barred":15605,"pony":15606,"famed":15607,"##sters":15608,"cassandra":15609,"waiter":15610,"carolyn":15611,"exported":15612,"##orted":15613,"andres":15614,"destructive":15615,"deeds":15616,"jonah":15617,"castles":15618,"vacancy":15619,"suv":15620,"##glass":15621,"1788":15622,"orchard":15623,"yep":15624,"famine":15625,"belarusian":15626,"sprang":15627,"##forth":15628,"skinny":15629,"##mis":15630,"administrators":15631,"rotterdam":15632,"zambia":15633,"zhao":15634,"boiler":15635,"discoveries":15636,"##ride":15637,"##physics":15638,"lucius":15639,"disappointing":15640,"outreach":15641,"spoon":15642,"##frame":15643,"qualifications":15644,"unanimously":15645,"enjoys":15646,"regency":15647,"##iidae":15648,"stade":15649,"realism":15650,"veterinary":15651,"rodgers":15652,"dump":15653,"alain":15654,"chestnut":15655,"castile":15656,"censorship":15657,"rumble":15658,"gibbs":15659,"##itor":15660,"communion":15661,"reggae":15662,"inactivated":15663,"logs":15664,"loads":15665,"##houses":15666,"homosexual":15667,"##iano":15668,"ale":15669,"informs":15670,"##cas":15671,"phrases":15672,"plaster":15673,"linebacker":15674,"ambrose":15675,"kaiser":15676,"fascinated":15677,"850":15678,"limerick":15679,"recruitment":15680,"forge":15681,"mastered":15682,"##nding":15683,"leinster":15684,"rooted":15685,"threaten":15686,"##strom":15687,"borneo":15688,"##hes":15689,"suggestions":15690,"scholarships":15691,"propeller":15692,"documentaries":15693,"patronage":15694,"coats":15695,"constructing":15696,"invest":15697,"neurons":15698,"comet":15699,"entirety":15700,"shouts":15701,"identities":15702,"annoying":15703,"unchanged":15704,"wary":15705,"##antly":15706,"##ogy":15707,"neat":15708,"oversight":15709,"##kos":15710,"phillies":15711,"replay":15712,"constance":15713,"##kka":15714,"incarnation":15715,"humble":15716,"skies":15717,"minus":15718,"##acy":15719,"smithsonian":15720,"##chel":15721,"guerrilla":15722,"jar":15723,"cadets":15724,"##plate":15725,"surplus":15726,"audit":15727,"##aru":15728,"cracking":15729,"joanna":15730,"louisa":15731,"pacing":15732,"##lights":15733,"intentionally":15734,"##iri":15735,"diner":15736,"nwa":15737,"imprint":15738,"australians":15739,"tong":15740,"unprecedented":15741,"bunker":15742,"naive":15743,"specialists":15744,"ark":15745,"nichols":15746,"railing":15747,"leaked":15748,"pedal":15749,"##uka":15750,"shrub":15751,"longing":15752,"roofs":15753,"v8":15754,"captains":15755,"neural":15756,"tuned":15757,"##ntal":15758,"##jet":15759,"emission":15760,"medina":15761,"frantic":15762,"codex":15763,"definitive":15764,"sid":15765,"abolition":15766,"intensified":15767,"stocks":15768,"enrique":15769,"sustain":15770,"genoa":15771,"oxide":15772,"##written":15773,"clues":15774,"cha":15775,"##gers":15776,"tributaries":15777,"fragment":15778,"venom":15779,"##rity":15780,"##ente":15781,"##sca":15782,"muffled":15783,"vain":15784,"sire":15785,"laos":15786,"##ingly":15787,"##hana":15788,"hastily":15789,"snapping":15790,"surfaced":15791,"sentiment":15792,"motive":15793,"##oft":15794,"contests":15795,"approximate":15796,"mesa":15797,"luckily":15798,"dinosaur":15799,"exchanges":15800,"propelled":15801,"accord":15802,"bourne":15803,"relieve":15804,"tow":15805,"masks":15806,"offended":15807,"##ues":15808,"cynthia":15809,"##mmer":15810,"rains":15811,"bartender":15812,"zinc":15813,"reviewers":15814,"lois":15815,"##sai":15816,"legged":15817,"arrogant":15818,"rafe":15819,"rosie":15820,"comprise":15821,"handicap":15822,"blockade":15823,"inlet":15824,"lagoon":15825,"copied":15826,"drilling":15827,"shelley":15828,"petals":15829,"##inian":15830,"mandarin":15831,"obsolete":15832,"##inated":15833,"onward":15834,"arguably":15835,"productivity":15836,"cindy":15837,"praising":15838,"seldom":15839,"busch":15840,"discusses":15841,"raleigh":15842,"shortage":15843,"ranged":15844,"stanton":15845,"encouragement":15846,"firstly":15847,"conceded":15848,"overs":15849,"temporal":15850,"##uke":15851,"cbe":15852,"##bos":15853,"woo":15854,"certainty":15855,"pumps":15856,"##pton":15857,"stalked":15858,"##uli":15859,"lizzie":15860,"periodic":15861,"thieves":15862,"weaker":15863,"##night":15864,"gases":15865,"shoving":15866,"chooses":15867,"wc":15868,"##chemical":15869,"prompting":15870,"weights":15871,"##kill":15872,"robust":15873,"flanked":15874,"sticky":15875,"hu":15876,"tuberculosis":15877,"##eb":15878,"##eal":15879,"christchurch":15880,"resembled":15881,"wallet":15882,"reese":15883,"inappropriate":15884,"pictured":15885,"distract":15886,"fixing":15887,"fiddle":15888,"giggled":15889,"burger":15890,"heirs":15891,"hairy":15892,"mechanic":15893,"torque":15894,"apache":15895,"obsessed":15896,"chiefly":15897,"cheng":15898,"logging":15899,"##tag":15900,"extracted":15901,"meaningful":15902,"numb":15903,"##vsky":15904,"gloucestershire":15905,"reminding":15906,"##bay":15907,"unite":15908,"##lit":15909,"breeds":15910,"diminished":15911,"clown":15912,"glove":15913,"1860s":15914,"##ن":15915,"##ug":15916,"archibald":15917,"focal":15918,"freelance":15919,"sliced":15920,"depiction":15921,"##yk":15922,"organism":15923,"switches":15924,"sights":15925,"stray":15926,"crawling":15927,"##ril":15928,"lever":15929,"leningrad":15930,"interpretations":15931,"loops":15932,"anytime":15933,"reel":15934,"alicia":15935,"delighted":15936,"##ech":15937,"inhaled":15938,"xiv":15939,"suitcase":15940,"bernie":15941,"vega":15942,"licenses":15943,"northampton":15944,"exclusion":15945,"induction":15946,"monasteries":15947,"racecourse":15948,"homosexuality":15949,"##right":15950,"##sfield":15951,"##rky":15952,"dimitri":15953,"michele":15954,"alternatives":15955,"ions":15956,"commentators":15957,"genuinely":15958,"objected":15959,"pork":15960,"hospitality":15961,"fencing":15962,"stephan":15963,"warships":15964,"peripheral":15965,"wit":15966,"drunken":15967,"wrinkled":15968,"quentin":15969,"spends":15970,"departing":15971,"chung":15972,"numerical":15973,"spokesperson":15974,"##zone":15975,"johannesburg":15976,"caliber":15977,"killers":15978,"##udge":15979,"assumes":15980,"neatly":15981,"demographic":15982,"abigail":15983,"bloc":15984,"##vel":15985,"mounting":15986,"##lain":15987,"bentley":15988,"slightest":15989,"xu":15990,"recipients":15991,"##jk":15992,"merlin":15993,"##writer":15994,"seniors":15995,"prisons":15996,"blinking":15997,"hindwings":15998,"flickered":15999,"kappa":16000,"##hel":16001,"80s":16002,"strengthening":16003,"appealing":16004,"brewing":16005,"gypsy":16006,"mali":16007,"lashes":16008,"hulk":16009,"unpleasant":16010,"harassment":16011,"bio":16012,"treaties":16013,"predict":16014,"instrumentation":16015,"pulp":16016,"troupe":16017,"boiling":16018,"mantle":16019,"##ffe":16020,"ins":16021,"##vn":16022,"dividing":16023,"handles":16024,"verbs":16025,"##onal":16026,"coconut":16027,"senegal":16028,"340":16029,"thorough":16030,"gum":16031,"momentarily":16032,"##sto":16033,"cocaine":16034,"panicked":16035,"destined":16036,"##turing":16037,"teatro":16038,"denying":16039,"weary":16040,"captained":16041,"mans":16042,"##hawks":16043,"##code":16044,"wakefield":16045,"bollywood":16046,"thankfully":16047,"##16":16048,"cyril":16049,"##wu":16050,"amendments":16051,"##bahn":16052,"consultation":16053,"stud":16054,"reflections":16055,"kindness":16056,"1787":16057,"internally":16058,"##ovo":16059,"tex":16060,"mosaic":16061,"distribute":16062,"paddy":16063,"seeming":16064,"143":16065,"##hic":16066,"piers":16067,"##15":16068,"##mura":16069,"##verse":16070,"popularly":16071,"winger":16072,"kang":16073,"sentinel":16074,"mccoy":16075,"##anza":16076,"covenant":16077,"##bag":16078,"verge":16079,"fireworks":16080,"suppress":16081,"thrilled":16082,"dominate":16083,"##jar":16084,"swansea":16085,"##60":16086,"142":16087,"reconciliation":16088,"##ndi":16089,"stiffened":16090,"cue":16091,"dorian":16092,"##uf":16093,"damascus":16094,"amor":16095,"ida":16096,"foremost":16097,"##aga":16098,"porsche":16099,"unseen":16100,"dir":16101,"##had":16102,"##azi":16103,"stony":16104,"lexi":16105,"melodies":16106,"##nko":16107,"angular":16108,"integer":16109,"podcast":16110,"ants":16111,"inherent":16112,"jaws":16113,"justify":16114,"persona":16115,"##olved":16116,"josephine":16117,"##nr":16118,"##ressed":16119,"customary":16120,"flashes":16121,"gala":16122,"cyrus":16123,"glaring":16124,"backyard":16125,"ariel":16126,"physiology":16127,"greenland":16128,"html":16129,"stir":16130,"avon":16131,"atletico":16132,"finch":16133,"methodology":16134,"ked":16135,"##lent":16136,"mas":16137,"catholicism":16138,"townsend":16139,"branding":16140,"quincy":16141,"fits":16142,"containers":16143,"1777":16144,"ashore":16145,"aragon":16146,"##19":16147,"forearm":16148,"poisoning":16149,"##sd":16150,"adopting":16151,"conquer":16152,"grinding":16153,"amnesty":16154,"keller":16155,"finances":16156,"evaluate":16157,"forged":16158,"lankan":16159,"instincts":16160,"##uto":16161,"guam":16162,"bosnian":16163,"photographed":16164,"workplace":16165,"desirable":16166,"protector":16167,"##dog":16168,"allocation":16169,"intently":16170,"encourages":16171,"willy":16172,"##sten":16173,"bodyguard":16174,"electro":16175,"brighter":16176,"##ν":16177,"bihar":16178,"##chev":16179,"lasts":16180,"opener":16181,"amphibious":16182,"sal":16183,"verde":16184,"arte":16185,"##cope":16186,"captivity":16187,"vocabulary":16188,"yields":16189,"##tted":16190,"agreeing":16191,"desmond":16192,"pioneered":16193,"##chus":16194,"strap":16195,"campaigned":16196,"railroads":16197,"##ович":16198,"emblem":16199,"##dre":16200,"stormed":16201,"501":16202,"##ulous":16203,"marijuana":16204,"northumberland":16205,"##gn":16206,"##nath":16207,"bowen":16208,"landmarks":16209,"beaumont":16210,"##qua":16211,"danube":16212,"##bler":16213,"attorneys":16214,"th":16215,"ge":16216,"flyers":16217,"critique":16218,"villains":16219,"cass":16220,"mutation":16221,"acc":16222,"##0s":16223,"colombo":16224,"mckay":16225,"motif":16226,"sampling":16227,"concluding":16228,"syndicate":16229,"##rell":16230,"neon":16231,"stables":16232,"ds":16233,"warnings":16234,"clint":16235,"mourning":16236,"wilkinson":16237,"##tated":16238,"merrill":16239,"leopard":16240,"evenings":16241,"exhaled":16242,"emil":16243,"sonia":16244,"ezra":16245,"discrete":16246,"stove":16247,"farrell":16248,"fifteenth":16249,"prescribed":16250,"superhero":16251,"##rier":16252,"worms":16253,"helm":16254,"wren":16255,"##duction":16256,"##hc":16257,"expo":16258,"##rator":16259,"hq":16260,"unfamiliar":16261,"antony":16262,"prevents":16263,"acceleration":16264,"fiercely":16265,"mari":16266,"painfully":16267,"calculations":16268,"cheaper":16269,"ign":16270,"clifton":16271,"irvine":16272,"davenport":16273,"mozambique":16274,"##np":16275,"pierced":16276,"##evich":16277,"wonders":16278,"##wig":16279,"##cate":16280,"##iling":16281,"crusade":16282,"ware":16283,"##uel":16284,"enzymes":16285,"reasonably":16286,"mls":16287,"##coe":16288,"mater":16289,"ambition":16290,"bunny":16291,"eliot":16292,"kernel":16293,"##fin":16294,"asphalt":16295,"headmaster":16296,"torah":16297,"aden":16298,"lush":16299,"pins":16300,"waived":16301,"##care":16302,"##yas":16303,"joao":16304,"substrate":16305,"enforce":16306,"##grad":16307,"##ules":16308,"alvarez":16309,"selections":16310,"epidemic":16311,"tempted":16312,"##bit":16313,"bremen":16314,"translates":16315,"ensured":16316,"waterfront":16317,"29th":16318,"forrest":16319,"manny":16320,"malone":16321,"kramer":16322,"reigning":16323,"cookies":16324,"simpler":16325,"absorption":16326,"205":16327,"engraved":16328,"##ffy":16329,"evaluated":16330,"1778":16331,"haze":16332,"146":16333,"comforting":16334,"crossover":16335,"##abe":16336,"thorn":16337,"##rift":16338,"##imo":16339,"##pop":16340,"suppression":16341,"fatigue":16342,"cutter":16343,"##tr":16344,"201":16345,"wurttemberg":16346,"##orf":16347,"enforced":16348,"hovering":16349,"proprietary":16350,"gb":16351,"samurai":16352,"syllable":16353,"ascent":16354,"lacey":16355,"tick":16356,"lars":16357,"tractor":16358,"merchandise":16359,"rep":16360,"bouncing":16361,"defendants":16362,"##yre":16363,"huntington":16364,"##ground":16365,"##oko":16366,"standardized":16367,"##hor":16368,"##hima":16369,"assassinated":16370,"nu":16371,"predecessors":16372,"rainy":16373,"liar":16374,"assurance":16375,"lyrical":16376,"##uga":16377,"secondly":16378,"flattened":16379,"ios":16380,"parameter":16381,"undercover":16382,"##mity":16383,"bordeaux":16384,"punish":16385,"ridges":16386,"markers":16387,"exodus":16388,"inactive":16389,"hesitate":16390,"debbie":16391,"nyc":16392,"pledge":16393,"savoy":16394,"nagar":16395,"offset":16396,"organist":16397,"##tium":16398,"hesse":16399,"marin":16400,"converting":16401,"##iver":16402,"diagram":16403,"propulsion":16404,"pu":16405,"validity":16406,"reverted":16407,"supportive":16408,"##dc":16409,"ministries":16410,"clans":16411,"responds":16412,"proclamation":16413,"##inae":16414,"##ø":16415,"##rea":16416,"ein":16417,"pleading":16418,"patriot":16419,"sf":16420,"birch":16421,"islanders":16422,"strauss":16423,"hates":16424,"##dh":16425,"brandenburg":16426,"concession":16427,"rd":16428,"##ob":16429,"1900s":16430,"killings":16431,"textbook":16432,"antiquity":16433,"cinematography":16434,"wharf":16435,"embarrassing":16436,"setup":16437,"creed":16438,"farmland":16439,"inequality":16440,"centred":16441,"signatures":16442,"fallon":16443,"370":16444,"##ingham":16445,"##uts":16446,"ceylon":16447,"gazing":16448,"directive":16449,"laurie":16450,"##tern":16451,"globally":16452,"##uated":16453,"##dent":16454,"allah":16455,"excavation":16456,"threads":16457,"##cross":16458,"148":16459,"frantically":16460,"icc":16461,"utilize":16462,"determines":16463,"respiratory":16464,"thoughtful":16465,"receptions":16466,"##dicate":16467,"merging":16468,"chandra":16469,"seine":16470,"147":16471,"builders":16472,"builds":16473,"diagnostic":16474,"dev":16475,"visibility":16476,"goddamn":16477,"analyses":16478,"dhaka":16479,"cho":16480,"proves":16481,"chancel":16482,"concurrent":16483,"curiously":16484,"canadians":16485,"pumped":16486,"restoring":16487,"1850s":16488,"turtles":16489,"jaguar":16490,"sinister":16491,"spinal":16492,"traction":16493,"declan":16494,"vows":16495,"1784":16496,"glowed":16497,"capitalism":16498,"swirling":16499,"install":16500,"universidad":16501,"##lder":16502,"##oat":16503,"soloist":16504,"##genic":16505,"##oor":16506,"coincidence":16507,"beginnings":16508,"nissan":16509,"dip":16510,"resorts":16511,"caucasus":16512,"combustion":16513,"infectious":16514,"##eno":16515,"pigeon":16516,"serpent":16517,"##itating":16518,"conclude":16519,"masked":16520,"salad":16521,"jew":16522,"##gr":16523,"surreal":16524,"toni":16525,"##wc":16526,"harmonica":16527,"151":16528,"##gins":16529,"##etic":16530,"##coat":16531,"fishermen":16532,"intending":16533,"bravery":16534,"##wave":16535,"klaus":16536,"titan":16537,"wembley":16538,"taiwanese":16539,"ransom":16540,"40th":16541,"incorrect":16542,"hussein":16543,"eyelids":16544,"jp":16545,"cooke":16546,"dramas":16547,"utilities":16548,"##etta":16549,"##print":16550,"eisenhower":16551,"principally":16552,"granada":16553,"lana":16554,"##rak":16555,"openings":16556,"concord":16557,"##bl":16558,"bethany":16559,"connie":16560,"morality":16561,"sega":16562,"##mons":16563,"##nard":16564,"earnings":16565,"##kara":16566,"##cine":16567,"wii":16568,"communes":16569,"##rel":16570,"coma":16571,"composing":16572,"softened":16573,"severed":16574,"grapes":16575,"##17":16576,"nguyen":16577,"analyzed":16578,"warlord":16579,"hubbard":16580,"heavenly":16581,"behave":16582,"slovenian":16583,"##hit":16584,"##ony":16585,"hailed":16586,"filmmakers":16587,"trance":16588,"caldwell":16589,"skye":16590,"unrest":16591,"coward":16592,"likelihood":16593,"##aging":16594,"bern":16595,"sci":16596,"taliban":16597,"honolulu":16598,"propose":16599,"##wang":16600,"1700":16601,"browser":16602,"imagining":16603,"cobra":16604,"contributes":16605,"dukes":16606,"instinctively":16607,"conan":16608,"violinist":16609,"##ores":16610,"accessories":16611,"gradual":16612,"##amp":16613,"quotes":16614,"sioux":16615,"##dating":16616,"undertake":16617,"intercepted":16618,"sparkling":16619,"compressed":16620,"139":16621,"fungus":16622,"tombs":16623,"haley":16624,"imposing":16625,"rests":16626,"degradation":16627,"lincolnshire":16628,"retailers":16629,"wetlands":16630,"tulsa":16631,"distributor":16632,"dungeon":16633,"nun":16634,"greenhouse":16635,"convey":16636,"atlantis":16637,"aft":16638,"exits":16639,"oman":16640,"dresser":16641,"lyons":16642,"##sti":16643,"joking":16644,"eddy":16645,"judgement":16646,"omitted":16647,"digits":16648,"##cts":16649,"##game":16650,"juniors":16651,"##rae":16652,"cents":16653,"stricken":16654,"une":16655,"##ngo":16656,"wizards":16657,"weir":16658,"breton":16659,"nan":16660,"technician":16661,"fibers":16662,"liking":16663,"royalty":16664,"##cca":16665,"154":16666,"persia":16667,"terribly":16668,"magician":16669,"##rable":16670,"##unt":16671,"vance":16672,"cafeteria":16673,"booker":16674,"camille":16675,"warmer":16676,"##static":16677,"consume":16678,"cavern":16679,"gaps":16680,"compass":16681,"contemporaries":16682,"foyer":16683,"soothing":16684,"graveyard":16685,"maj":16686,"plunged":16687,"blush":16688,"##wear":16689,"cascade":16690,"demonstrates":16691,"ordinance":16692,"##nov":16693,"boyle":16694,"##lana":16695,"rockefeller":16696,"shaken":16697,"banjo":16698,"izzy":16699,"##ense":16700,"breathless":16701,"vines":16702,"##32":16703,"##eman":16704,"alterations":16705,"chromosome":16706,"dwellings":16707,"feudal":16708,"mole":16709,"153":16710,"catalonia":16711,"relics":16712,"tenant":16713,"mandated":16714,"##fm":16715,"fridge":16716,"hats":16717,"honesty":16718,"patented":16719,"raul":16720,"heap":16721,"cruisers":16722,"accusing":16723,"enlightenment":16724,"infants":16725,"wherein":16726,"chatham":16727,"contractors":16728,"zen":16729,"affinity":16730,"hc":16731,"osborne":16732,"piston":16733,"156":16734,"traps":16735,"maturity":16736,"##rana":16737,"lagos":16738,"##zal":16739,"peering":16740,"##nay":16741,"attendant":16742,"dealers":16743,"protocols":16744,"subset":16745,"prospects":16746,"biographical":16747,"##cre":16748,"artery":16749,"##zers":16750,"insignia":16751,"nuns":16752,"endured":16753,"##eration":16754,"recommend":16755,"schwartz":16756,"serbs":16757,"berger":16758,"cromwell":16759,"crossroads":16760,"##ctor":16761,"enduring":16762,"clasped":16763,"grounded":16764,"##bine":16765,"marseille":16766,"twitched":16767,"abel":16768,"choke":16769,"https":16770,"catalyst":16771,"moldova":16772,"italians":16773,"##tist":16774,"disastrous":16775,"wee":16776,"##oured":16777,"##nti":16778,"wwf":16779,"nope":16780,"##piration":16781,"##asa":16782,"expresses":16783,"thumbs":16784,"167":16785,"##nza":16786,"coca":16787,"1781":16788,"cheating":16789,"##ption":16790,"skipped":16791,"sensory":16792,"heidelberg":16793,"spies":16794,"satan":16795,"dangers":16796,"semifinal":16797,"202":16798,"bohemia":16799,"whitish":16800,"confusing":16801,"shipbuilding":16802,"relies":16803,"surgeons":16804,"landings":16805,"ravi":16806,"baku":16807,"moor":16808,"suffix":16809,"alejandro":16810,"##yana":16811,"litre":16812,"upheld":16813,"##unk":16814,"rajasthan":16815,"##rek":16816,"coaster":16817,"insists":16818,"posture":16819,"scenarios":16820,"etienne":16821,"favoured":16822,"appoint":16823,"transgender":16824,"elephants":16825,"poked":16826,"greenwood":16827,"defences":16828,"fulfilled":16829,"militant":16830,"somali":16831,"1758":16832,"chalk":16833,"potent":16834,"##ucci":16835,"migrants":16836,"wink":16837,"assistants":16838,"nos":16839,"restriction":16840,"activism":16841,"niger":16842,"##ario":16843,"colon":16844,"shaun":16845,"##sat":16846,"daphne":16847,"##erated":16848,"swam":16849,"congregations":16850,"reprise":16851,"considerations":16852,"magnet":16853,"playable":16854,"xvi":16855,"##р":16856,"overthrow":16857,"tobias":16858,"knob":16859,"chavez":16860,"coding":16861,"##mers":16862,"propped":16863,"katrina":16864,"orient":16865,"newcomer":16866,"##suke":16867,"temperate":16868,"##pool":16869,"farmhouse":16870,"interrogation":16871,"##vd":16872,"committing":16873,"##vert":16874,"forthcoming":16875,"strawberry":16876,"joaquin":16877,"macau":16878,"ponds":16879,"shocking":16880,"siberia":16881,"##cellular":16882,"chant":16883,"contributors":16884,"##nant":16885,"##ologists":16886,"sped":16887,"absorb":16888,"hail":16889,"1782":16890,"spared":16891,"##hore":16892,"barbados":16893,"karate":16894,"opus":16895,"originates":16896,"saul":16897,"##xie":16898,"evergreen":16899,"leaped":16900,"##rock":16901,"correlation":16902,"exaggerated":16903,"weekday":16904,"unification":16905,"bump":16906,"tracing":16907,"brig":16908,"afb":16909,"pathways":16910,"utilizing":16911,"##ners":16912,"mod":16913,"mb":16914,"disturbance":16915,"kneeling":16916,"##stad":16917,"##guchi":16918,"100th":16919,"pune":16920,"##thy":16921,"decreasing":16922,"168":16923,"manipulation":16924,"miriam":16925,"academia":16926,"ecosystem":16927,"occupational":16928,"rbi":16929,"##lem":16930,"rift":16931,"##14":16932,"rotary":16933,"stacked":16934,"incorporation":16935,"awakening":16936,"generators":16937,"guerrero":16938,"racist":16939,"##omy":16940,"cyber":16941,"derivatives":16942,"culminated":16943,"allie":16944,"annals":16945,"panzer":16946,"sainte":16947,"wikipedia":16948,"pops":16949,"zu":16950,"austro":16951,"##vate":16952,"algerian":16953,"politely":16954,"nicholson":16955,"mornings":16956,"educate":16957,"tastes":16958,"thrill":16959,"dartmouth":16960,"##gating":16961,"db":16962,"##jee":16963,"regan":16964,"differing":16965,"concentrating":16966,"choreography":16967,"divinity":16968,"##media":16969,"pledged":16970,"alexandre":16971,"routing":16972,"gregor":16973,"madeline":16974,"##idal":16975,"apocalypse":16976,"##hora":16977,"gunfire":16978,"culminating":16979,"elves":16980,"fined":16981,"liang":16982,"lam":16983,"programmed":16984,"tar":16985,"guessing":16986,"transparency":16987,"gabrielle":16988,"##gna":16989,"cancellation":16990,"flexibility":16991,"##lining":16992,"accession":16993,"shea":16994,"stronghold":16995,"nets":16996,"specializes":16997,"##rgan":16998,"abused":16999,"hasan":17000,"sgt":17001,"ling":17002,"exceeding":17003,"##₄":17004,"admiration":17005,"supermarket":17006,"##ark":17007,"photographers":17008,"specialised":17009,"tilt":17010,"resonance":17011,"hmm":17012,"perfume":17013,"380":17014,"sami":17015,"threatens":17016,"garland":17017,"botany":17018,"guarding":17019,"boiled":17020,"greet":17021,"puppy":17022,"russo":17023,"supplier":17024,"wilmington":17025,"vibrant":17026,"vijay":17027,"##bius":17028,"paralympic":17029,"grumbled":17030,"paige":17031,"faa":17032,"licking":17033,"margins":17034,"hurricanes":17035,"##gong":17036,"fest":17037,"grenade":17038,"ripping":17039,"##uz":17040,"counseling":17041,"weigh":17042,"##sian":17043,"needles":17044,"wiltshire":17045,"edison":17046,"costly":17047,"##not":17048,"fulton":17049,"tramway":17050,"redesigned":17051,"staffordshire":17052,"cache":17053,"gasping":17054,"watkins":17055,"sleepy":17056,"candidacy":17057,"##group":17058,"monkeys":17059,"timeline":17060,"throbbing":17061,"##bid":17062,"##sos":17063,"berth":17064,"uzbekistan":17065,"vanderbilt":17066,"bothering":17067,"overturned":17068,"ballots":17069,"gem":17070,"##iger":17071,"sunglasses":17072,"subscribers":17073,"hooker":17074,"compelling":17075,"ang":17076,"exceptionally":17077,"saloon":17078,"stab":17079,"##rdi":17080,"carla":17081,"terrifying":17082,"rom":17083,"##vision":17084,"coil":17085,"##oids":17086,"satisfying":17087,"vendors":17088,"31st":17089,"mackay":17090,"deities":17091,"overlooked":17092,"ambient":17093,"bahamas":17094,"felipe":17095,"olympia":17096,"whirled":17097,"botanist":17098,"advertised":17099,"tugging":17100,"##dden":17101,"disciples":17102,"morales":17103,"unionist":17104,"rites":17105,"foley":17106,"morse":17107,"motives":17108,"creepy":17109,"##₀":17110,"soo":17111,"##sz":17112,"bargain":17113,"highness":17114,"frightening":17115,"turnpike":17116,"tory":17117,"reorganization":17118,"##cer":17119,"depict":17120,"biographer":17121,"##walk":17122,"unopposed":17123,"manifesto":17124,"##gles":17125,"institut":17126,"emile":17127,"accidental":17128,"kapoor":17129,"##dam":17130,"kilkenny":17131,"cortex":17132,"lively":17133,"##13":17134,"romanesque":17135,"jain":17136,"shan":17137,"cannons":17138,"##ood":17139,"##ske":17140,"petrol":17141,"echoing":17142,"amalgamated":17143,"disappears":17144,"cautious":17145,"proposes":17146,"sanctions":17147,"trenton":17148,"##ر":17149,"flotilla":17150,"aus":17151,"contempt":17152,"tor":17153,"canary":17154,"cote":17155,"theirs":17156,"##hun":17157,"conceptual":17158,"deleted":17159,"fascinating":17160,"paso":17161,"blazing":17162,"elf":17163,"honourable":17164,"hutchinson":17165,"##eiro":17166,"##outh":17167,"##zin":17168,"surveyor":17169,"tee":17170,"amidst":17171,"wooded":17172,"reissue":17173,"intro":17174,"##ono":17175,"cobb":17176,"shelters":17177,"newsletter":17178,"hanson":17179,"brace":17180,"encoding":17181,"confiscated":17182,"dem":17183,"caravan":17184,"marino":17185,"scroll":17186,"melodic":17187,"cows":17188,"imam":17189,"##adi":17190,"##aneous":17191,"northward":17192,"searches":17193,"biodiversity":17194,"cora":17195,"310":17196,"roaring":17197,"##bers":17198,"connell":17199,"theologian":17200,"halo":17201,"compose":17202,"pathetic":17203,"unmarried":17204,"dynamo":17205,"##oot":17206,"az":17207,"calculation":17208,"toulouse":17209,"deserves":17210,"humour":17211,"nr":17212,"forgiveness":17213,"tam":17214,"undergone":17215,"martyr":17216,"pamela":17217,"myths":17218,"whore":17219,"counselor":17220,"hicks":17221,"290":17222,"heavens":17223,"battleship":17224,"electromagnetic":17225,"##bbs":17226,"stellar":17227,"establishments":17228,"presley":17229,"hopped":17230,"##chin":17231,"temptation":17232,"90s":17233,"wills":17234,"nas":17235,"##yuan":17236,"nhs":17237,"##nya":17238,"seminars":17239,"##yev":17240,"adaptations":17241,"gong":17242,"asher":17243,"lex":17244,"indicator":17245,"sikh":17246,"tobago":17247,"cites":17248,"goin":17249,"##yte":17250,"satirical":17251,"##gies":17252,"characterised":17253,"correspond":17254,"bubbles":17255,"lure":17256,"participates":17257,"##vid":17258,"eruption":17259,"skate":17260,"therapeutic":17261,"1785":17262,"canals":17263,"wholesale":17264,"defaulted":17265,"sac":17266,"460":17267,"petit":17268,"##zzled":17269,"virgil":17270,"leak":17271,"ravens":17272,"256":17273,"portraying":17274,"##yx":17275,"ghetto":17276,"creators":17277,"dams":17278,"portray":17279,"vicente":17280,"##rington":17281,"fae":17282,"namesake":17283,"bounty":17284,"##arium":17285,"joachim":17286,"##ota":17287,"##iser":17288,"aforementioned":17289,"axle":17290,"snout":17291,"depended":17292,"dismantled":17293,"reuben":17294,"480":17295,"##ibly":17296,"gallagher":17297,"##lau":17298,"##pd":17299,"earnest":17300,"##ieu":17301,"##iary":17302,"inflicted":17303,"objections":17304,"##llar":17305,"asa":17306,"gritted":17307,"##athy":17308,"jericho":17309,"##sea":17310,"##was":17311,"flick":17312,"underside":17313,"ceramics":17314,"undead":17315,"substituted":17316,"195":17317,"eastward":17318,"undoubtedly":17319,"wheeled":17320,"chimney":17321,"##iche":17322,"guinness":17323,"cb":17324,"##ager":17325,"siding":17326,"##bell":17327,"traitor":17328,"baptiste":17329,"disguised":17330,"inauguration":17331,"149":17332,"tipperary":17333,"choreographer":17334,"perched":17335,"warmed":17336,"stationary":17337,"eco":17338,"##ike":17339,"##ntes":17340,"bacterial":17341,"##aurus":17342,"flores":17343,"phosphate":17344,"##core":17345,"attacker":17346,"invaders":17347,"alvin":17348,"intersects":17349,"a1":17350,"indirectly":17351,"immigrated":17352,"businessmen":17353,"cornelius":17354,"valves":17355,"narrated":17356,"pill":17357,"sober":17358,"ul":17359,"nationale":17360,"monastic":17361,"applicants":17362,"scenery":17363,"##jack":17364,"161":17365,"motifs":17366,"constitutes":17367,"cpu":17368,"##osh":17369,"jurisdictions":17370,"sd":17371,"tuning":17372,"irritation":17373,"woven":17374,"##uddin":17375,"fertility":17376,"gao":17377,"##erie":17378,"antagonist":17379,"impatient":17380,"glacial":17381,"hides":17382,"boarded":17383,"denominations":17384,"interception":17385,"##jas":17386,"cookie":17387,"nicola":17388,"##tee":17389,"algebraic":17390,"marquess":17391,"bahn":17392,"parole":17393,"buyers":17394,"bait":17395,"turbines":17396,"paperwork":17397,"bestowed":17398,"natasha":17399,"renee":17400,"oceans":17401,"purchases":17402,"157":17403,"vaccine":17404,"215":17405,"##tock":17406,"fixtures":17407,"playhouse":17408,"integrate":17409,"jai":17410,"oswald":17411,"intellectuals":17412,"##cky":17413,"booked":17414,"nests":17415,"mortimer":17416,"##isi":17417,"obsession":17418,"sept":17419,"##gler":17420,"##sum":17421,"440":17422,"scrutiny":17423,"simultaneous":17424,"squinted":17425,"##shin":17426,"collects":17427,"oven":17428,"shankar":17429,"penned":17430,"remarkably":17431,"##я":17432,"slips":17433,"luggage":17434,"spectral":17435,"1786":17436,"collaborations":17437,"louie":17438,"consolidation":17439,"##ailed":17440,"##ivating":17441,"420":17442,"hoover":17443,"blackpool":17444,"harness":17445,"ignition":17446,"vest":17447,"tails":17448,"belmont":17449,"mongol":17450,"skinner":17451,"##nae":17452,"visually":17453,"mage":17454,"derry":17455,"##tism":17456,"##unce":17457,"stevie":17458,"transitional":17459,"##rdy":17460,"redskins":17461,"drying":17462,"prep":17463,"prospective":17464,"##21":17465,"annoyance":17466,"oversee":17467,"##loaded":17468,"fills":17469,"##books":17470,"##iki":17471,"announces":17472,"fda":17473,"scowled":17474,"respects":17475,"prasad":17476,"mystic":17477,"tucson":17478,"##vale":17479,"revue":17480,"springer":17481,"bankrupt":17482,"1772":17483,"aristotle":17484,"salvatore":17485,"habsburg":17486,"##geny":17487,"dal":17488,"natal":17489,"nut":17490,"pod":17491,"chewing":17492,"darts":17493,"moroccan":17494,"walkover":17495,"rosario":17496,"lenin":17497,"punjabi":17498,"##ße":17499,"grossed":17500,"scattering":17501,"wired":17502,"invasive":17503,"hui":17504,"polynomial":17505,"corridors":17506,"wakes":17507,"gina":17508,"portrays":17509,"##cratic":17510,"arid":17511,"retreating":17512,"erich":17513,"irwin":17514,"sniper":17515,"##dha":17516,"linen":17517,"lindsey":17518,"maneuver":17519,"butch":17520,"shutting":17521,"socio":17522,"bounce":17523,"commemorative":17524,"postseason":17525,"jeremiah":17526,"pines":17527,"275":17528,"mystical":17529,"beads":17530,"bp":17531,"abbas":17532,"furnace":17533,"bidding":17534,"consulted":17535,"assaulted":17536,"empirical":17537,"rubble":17538,"enclosure":17539,"sob":17540,"weakly":17541,"cancel":17542,"polly":17543,"yielded":17544,"##emann":17545,"curly":17546,"prediction":17547,"battered":17548,"70s":17549,"vhs":17550,"jacqueline":17551,"render":17552,"sails":17553,"barked":17554,"detailing":17555,"grayson":17556,"riga":17557,"sloane":17558,"raging":17559,"##yah":17560,"herbs":17561,"bravo":17562,"##athlon":17563,"alloy":17564,"giggle":17565,"imminent":17566,"suffers":17567,"assumptions":17568,"waltz":17569,"##itate":17570,"accomplishments":17571,"##ited":17572,"bathing":17573,"remixed":17574,"deception":17575,"prefix":17576,"##emia":17577,"deepest":17578,"##tier":17579,"##eis":17580,"balkan":17581,"frogs":17582,"##rong":17583,"slab":17584,"##pate":17585,"philosophers":17586,"peterborough":17587,"grains":17588,"imports":17589,"dickinson":17590,"rwanda":17591,"##atics":17592,"1774":17593,"dirk":17594,"lan":17595,"tablets":17596,"##rove":17597,"clone":17598,"##rice":17599,"caretaker":17600,"hostilities":17601,"mclean":17602,"##gre":17603,"regimental":17604,"treasures":17605,"norms":17606,"impose":17607,"tsar":17608,"tango":17609,"diplomacy":17610,"variously":17611,"complain":17612,"192":17613,"recognise":17614,"arrests":17615,"1779":17616,"celestial":17617,"pulitzer":17618,"##dus":17619,"bing":17620,"libretto":17621,"##moor":17622,"adele":17623,"splash":17624,"##rite":17625,"expectation":17626,"lds":17627,"confronts":17628,"##izer":17629,"spontaneous":17630,"harmful":17631,"wedge":17632,"entrepreneurs":17633,"buyer":17634,"##ope":17635,"bilingual":17636,"translate":17637,"rugged":17638,"conner":17639,"circulated":17640,"uae":17641,"eaton":17642,"##gra":17643,"##zzle":17644,"lingered":17645,"lockheed":17646,"vishnu":17647,"reelection":17648,"alonso":17649,"##oom":17650,"joints":17651,"yankee":17652,"headline":17653,"cooperate":17654,"heinz":17655,"laureate":17656,"invading":17657,"##sford":17658,"echoes":17659,"scandinavian":17660,"##dham":17661,"hugging":17662,"vitamin":17663,"salute":17664,"micah":17665,"hind":17666,"trader":17667,"##sper":17668,"radioactive":17669,"##ndra":17670,"militants":17671,"poisoned":17672,"ratified":17673,"remark":17674,"campeonato":17675,"deprived":17676,"wander":17677,"prop":17678,"##dong":17679,"outlook":17680,"##tani":17681,"##rix":17682,"##eye":17683,"chiang":17684,"darcy":17685,"##oping":17686,"mandolin":17687,"spice":17688,"statesman":17689,"babylon":17690,"182":17691,"walled":17692,"forgetting":17693,"afro":17694,"##cap":17695,"158":17696,"giorgio":17697,"buffer":17698,"##polis":17699,"planetary":17700,"##gis":17701,"overlap":17702,"terminals":17703,"kinda":17704,"centenary":17705,"##bir":17706,"arising":17707,"manipulate":17708,"elm":17709,"ke":17710,"1770":17711,"ak":17712,"##tad":17713,"chrysler":17714,"mapped":17715,"moose":17716,"pomeranian":17717,"quad":17718,"macarthur":17719,"assemblies":17720,"shoreline":17721,"recalls":17722,"stratford":17723,"##rted":17724,"noticeable":17725,"##evic":17726,"imp":17727,"##rita":17728,"##sque":17729,"accustomed":17730,"supplying":17731,"tents":17732,"disgusted":17733,"vogue":17734,"sipped":17735,"filters":17736,"khz":17737,"reno":17738,"selecting":17739,"luftwaffe":17740,"mcmahon":17741,"tyne":17742,"masterpiece":17743,"carriages":17744,"collided":17745,"dunes":17746,"exercised":17747,"flare":17748,"remembers":17749,"muzzle":17750,"##mobile":17751,"heck":17752,"##rson":17753,"burgess":17754,"lunged":17755,"middleton":17756,"boycott":17757,"bilateral":17758,"##sity":17759,"hazardous":17760,"lumpur":17761,"multiplayer":17762,"spotlight":17763,"jackets":17764,"goldman":17765,"liege":17766,"porcelain":17767,"rag":17768,"waterford":17769,"benz":17770,"attracts":17771,"hopeful":17772,"battling":17773,"ottomans":17774,"kensington":17775,"baked":17776,"hymns":17777,"cheyenne":17778,"lattice":17779,"levine":17780,"borrow":17781,"polymer":17782,"clashes":17783,"michaels":17784,"monitored":17785,"commitments":17786,"denounced":17787,"##25":17788,"##von":17789,"cavity":17790,"##oney":17791,"hobby":17792,"akin":17793,"##holders":17794,"futures":17795,"intricate":17796,"cornish":17797,"patty":17798,"##oned":17799,"illegally":17800,"dolphin":17801,"##lag":17802,"barlow":17803,"yellowish":17804,"maddie":17805,"apologized":17806,"luton":17807,"plagued":17808,"##puram":17809,"nana":17810,"##rds":17811,"sway":17812,"fanny":17813,"łodz":17814,"##rino":17815,"psi":17816,"suspicions":17817,"hanged":17818,"##eding":17819,"initiate":17820,"charlton":17821,"##por":17822,"nak":17823,"competent":17824,"235":17825,"analytical":17826,"annex":17827,"wardrobe":17828,"reservations":17829,"##rma":17830,"sect":17831,"162":17832,"fairfax":17833,"hedge":17834,"piled":17835,"buckingham":17836,"uneven":17837,"bauer":17838,"simplicity":17839,"snyder":17840,"interpret":17841,"accountability":17842,"donors":17843,"moderately":17844,"byrd":17845,"continents":17846,"##cite":17847,"##max":17848,"disciple":17849,"hr":17850,"jamaican":17851,"ping":17852,"nominees":17853,"##uss":17854,"mongolian":17855,"diver":17856,"attackers":17857,"eagerly":17858,"ideological":17859,"pillows":17860,"miracles":17861,"apartheid":17862,"revolver":17863,"sulfur":17864,"clinics":17865,"moran":17866,"163":17867,"##enko":17868,"ile":17869,"katy":17870,"rhetoric":17871,"##icated":17872,"chronology":17873,"recycling":17874,"##hrer":17875,"elongated":17876,"mughal":17877,"pascal":17878,"profiles":17879,"vibration":17880,"databases":17881,"domination":17882,"##fare":17883,"##rant":17884,"matthias":17885,"digest":17886,"rehearsal":17887,"polling":17888,"weiss":17889,"initiation":17890,"reeves":17891,"clinging":17892,"flourished":17893,"impress":17894,"ngo":17895,"##hoff":17896,"##ume":17897,"buckley":17898,"symposium":17899,"rhythms":17900,"weed":17901,"emphasize":17902,"transforming":17903,"##taking":17904,"##gence":17905,"##yman":17906,"accountant":17907,"analyze":17908,"flicker":17909,"foil":17910,"priesthood":17911,"voluntarily":17912,"decreases":17913,"##80":17914,"##hya":17915,"slater":17916,"sv":17917,"charting":17918,"mcgill":17919,"##lde":17920,"moreno":17921,"##iu":17922,"besieged":17923,"zur":17924,"robes":17925,"##phic":17926,"admitting":17927,"api":17928,"deported":17929,"turmoil":17930,"peyton":17931,"earthquakes":17932,"##ares":17933,"nationalists":17934,"beau":17935,"clair":17936,"brethren":17937,"interrupt":17938,"welch":17939,"curated":17940,"galerie":17941,"requesting":17942,"164":17943,"##ested":17944,"impending":17945,"steward":17946,"viper":17947,"##vina":17948,"complaining":17949,"beautifully":17950,"brandy":17951,"foam":17952,"nl":17953,"1660":17954,"##cake":17955,"alessandro":17956,"punches":17957,"laced":17958,"explanations":17959,"##lim":17960,"attribute":17961,"clit":17962,"reggie":17963,"discomfort":17964,"##cards":17965,"smoothed":17966,"whales":17967,"##cene":17968,"adler":17969,"countered":17970,"duffy":17971,"disciplinary":17972,"widening":17973,"recipe":17974,"reliance":17975,"conducts":17976,"goats":17977,"gradient":17978,"preaching":17979,"##shaw":17980,"matilda":17981,"quasi":17982,"striped":17983,"meridian":17984,"cannabis":17985,"cordoba":17986,"certificates":17987,"##agh":17988,"##tering":17989,"graffiti":17990,"hangs":17991,"pilgrims":17992,"repeats":17993,"##ych":17994,"revive":17995,"urine":17996,"etat":17997,"##hawk":17998,"fueled":17999,"belts":18000,"fuzzy":18001,"susceptible":18002,"##hang":18003,"mauritius":18004,"salle":18005,"sincere":18006,"beers":18007,"hooks":18008,"##cki":18009,"arbitration":18010,"entrusted":18011,"advise":18012,"sniffed":18013,"seminar":18014,"junk":18015,"donnell":18016,"processors":18017,"principality":18018,"strapped":18019,"celia":18020,"mendoza":18021,"everton":18022,"fortunes":18023,"prejudice":18024,"starving":18025,"reassigned":18026,"steamer":18027,"##lund":18028,"tuck":18029,"evenly":18030,"foreman":18031,"##ffen":18032,"dans":18033,"375":18034,"envisioned":18035,"slit":18036,"##xy":18037,"baseman":18038,"liberia":18039,"rosemary":18040,"##weed":18041,"electrified":18042,"periodically":18043,"potassium":18044,"stride":18045,"contexts":18046,"sperm":18047,"slade":18048,"mariners":18049,"influx":18050,"bianca":18051,"subcommittee":18052,"##rane":18053,"spilling":18054,"icao":18055,"estuary":18056,"##nock":18057,"delivers":18058,"iphone":18059,"##ulata":18060,"isa":18061,"mira":18062,"bohemian":18063,"dessert":18064,"##sbury":18065,"welcoming":18066,"proudly":18067,"slowing":18068,"##chs":18069,"musee":18070,"ascension":18071,"russ":18072,"##vian":18073,"waits":18074,"##psy":18075,"africans":18076,"exploit":18077,"##morphic":18078,"gov":18079,"eccentric":18080,"crab":18081,"peck":18082,"##ull":18083,"entrances":18084,"formidable":18085,"marketplace":18086,"groom":18087,"bolted":18088,"metabolism":18089,"patton":18090,"robbins":18091,"courier":18092,"payload":18093,"endure":18094,"##ifier":18095,"andes":18096,"refrigerator":18097,"##pr":18098,"ornate":18099,"##uca":18100,"ruthless":18101,"illegitimate":18102,"masonry":18103,"strasbourg":18104,"bikes":18105,"adobe":18106,"##³":18107,"apples":18108,"quintet":18109,"willingly":18110,"niche":18111,"bakery":18112,"corpses":18113,"energetic":18114,"##cliffe":18115,"##sser":18116,"##ards":18117,"177":18118,"centimeters":18119,"centro":18120,"fuscous":18121,"cretaceous":18122,"rancho":18123,"##yde":18124,"andrei":18125,"telecom":18126,"tottenham":18127,"oasis":18128,"ordination":18129,"vulnerability":18130,"presiding":18131,"corey":18132,"cp":18133,"penguins":18134,"sims":18135,"##pis":18136,"malawi":18137,"piss":18138,"##48":18139,"correction":18140,"##cked":18141,"##ffle":18142,"##ryn":18143,"countdown":18144,"detectives":18145,"psychiatrist":18146,"psychedelic":18147,"dinosaurs":18148,"blouse":18149,"##get":18150,"choi":18151,"vowed":18152,"##oz":18153,"randomly":18154,"##pol":18155,"49ers":18156,"scrub":18157,"blanche":18158,"bruins":18159,"dusseldorf":18160,"##using":18161,"unwanted":18162,"##ums":18163,"212":18164,"dominique":18165,"elevations":18166,"headlights":18167,"om":18168,"laguna":18169,"##oga":18170,"1750":18171,"famously":18172,"ignorance":18173,"shrewsbury":18174,"##aine":18175,"ajax":18176,"breuning":18177,"che":18178,"confederacy":18179,"greco":18180,"overhaul":18181,"##screen":18182,"paz":18183,"skirts":18184,"disagreement":18185,"cruelty":18186,"jagged":18187,"phoebe":18188,"shifter":18189,"hovered":18190,"viruses":18191,"##wes":18192,"mandy":18193,"##lined":18194,"##gc":18195,"landlord":18196,"squirrel":18197,"dashed":18198,"##ι":18199,"ornamental":18200,"gag":18201,"wally":18202,"grange":18203,"literal":18204,"spurs":18205,"undisclosed":18206,"proceeding":18207,"yin":18208,"##text":18209,"billie":18210,"orphan":18211,"spanned":18212,"humidity":18213,"indy":18214,"weighted":18215,"presentations":18216,"explosions":18217,"lucian":18218,"##tary":18219,"vaughn":18220,"hindus":18221,"##anga":18222,"##hell":18223,"psycho":18224,"171":18225,"daytona":18226,"protects":18227,"efficiently":18228,"rematch":18229,"sly":18230,"tandem":18231,"##oya":18232,"rebranded":18233,"impaired":18234,"hee":18235,"metropolis":18236,"peach":18237,"godfrey":18238,"diaspora":18239,"ethnicity":18240,"prosperous":18241,"gleaming":18242,"dar":18243,"grossing":18244,"playback":18245,"##rden":18246,"stripe":18247,"pistols":18248,"##tain":18249,"births":18250,"labelled":18251,"##cating":18252,"172":18253,"rudy":18254,"alba":18255,"##onne":18256,"aquarium":18257,"hostility":18258,"##gb":18259,"##tase":18260,"shudder":18261,"sumatra":18262,"hardest":18263,"lakers":18264,"consonant":18265,"creeping":18266,"demos":18267,"homicide":18268,"capsule":18269,"zeke":18270,"liberties":18271,"expulsion":18272,"pueblo":18273,"##comb":18274,"trait":18275,"transporting":18276,"##ddin":18277,"##neck":18278,"##yna":18279,"depart":18280,"gregg":18281,"mold":18282,"ledge":18283,"hangar":18284,"oldham":18285,"playboy":18286,"termination":18287,"analysts":18288,"gmbh":18289,"romero":18290,"##itic":18291,"insist":18292,"cradle":18293,"filthy":18294,"brightness":18295,"slash":18296,"shootout":18297,"deposed":18298,"bordering":18299,"##truct":18300,"isis":18301,"microwave":18302,"tumbled":18303,"sheltered":18304,"cathy":18305,"werewolves":18306,"messy":18307,"andersen":18308,"convex":18309,"clapped":18310,"clinched":18311,"satire":18312,"wasting":18313,"edo":18314,"vc":18315,"rufus":18316,"##jak":18317,"mont":18318,"##etti":18319,"poznan":18320,"##keeping":18321,"restructuring":18322,"transverse":18323,"##rland":18324,"azerbaijani":18325,"slovene":18326,"gestures":18327,"roommate":18328,"choking":18329,"shear":18330,"##quist":18331,"vanguard":18332,"oblivious":18333,"##hiro":18334,"disagreed":18335,"baptism":18336,"##lich":18337,"coliseum":18338,"##aceae":18339,"salvage":18340,"societe":18341,"cory":18342,"locke":18343,"relocation":18344,"relying":18345,"versailles":18346,"ahl":18347,"swelling":18348,"##elo":18349,"cheerful":18350,"##word":18351,"##edes":18352,"gin":18353,"sarajevo":18354,"obstacle":18355,"diverted":18356,"##nac":18357,"messed":18358,"thoroughbred":18359,"fluttered":18360,"utrecht":18361,"chewed":18362,"acquaintance":18363,"assassins":18364,"dispatch":18365,"mirza":18366,"##wart":18367,"nike":18368,"salzburg":18369,"swell":18370,"yen":18371,"##gee":18372,"idle":18373,"ligue":18374,"samson":18375,"##nds":18376,"##igh":18377,"playful":18378,"spawned":18379,"##cise":18380,"tease":18381,"##case":18382,"burgundy":18383,"##bot":18384,"stirring":18385,"skeptical":18386,"interceptions":18387,"marathi":18388,"##dies":18389,"bedrooms":18390,"aroused":18391,"pinch":18392,"##lik":18393,"preferences":18394,"tattoos":18395,"buster":18396,"digitally":18397,"projecting":18398,"rust":18399,"##ital":18400,"kitten":18401,"priorities":18402,"addison":18403,"pseudo":18404,"##guard":18405,"dusk":18406,"icons":18407,"sermon":18408,"##psis":18409,"##iba":18410,"bt":18411,"##lift":18412,"##xt":18413,"ju":18414,"truce":18415,"rink":18416,"##dah":18417,"##wy":18418,"defects":18419,"psychiatry":18420,"offences":18421,"calculate":18422,"glucose":18423,"##iful":18424,"##rized":18425,"##unda":18426,"francaise":18427,"##hari":18428,"richest":18429,"warwickshire":18430,"carly":18431,"1763":18432,"purity":18433,"redemption":18434,"lending":18435,"##cious":18436,"muse":18437,"bruises":18438,"cerebral":18439,"aero":18440,"carving":18441,"##name":18442,"preface":18443,"terminology":18444,"invade":18445,"monty":18446,"##int":18447,"anarchist":18448,"blurred":18449,"##iled":18450,"rossi":18451,"treats":18452,"guts":18453,"shu":18454,"foothills":18455,"ballads":18456,"undertaking":18457,"premise":18458,"cecilia":18459,"affiliates":18460,"blasted":18461,"conditional":18462,"wilder":18463,"minors":18464,"drone":18465,"rudolph":18466,"buffy":18467,"swallowing":18468,"horton":18469,"attested":18470,"##hop":18471,"rutherford":18472,"howell":18473,"primetime":18474,"livery":18475,"penal":18476,"##bis":18477,"minimize":18478,"hydro":18479,"wrecked":18480,"wrought":18481,"palazzo":18482,"##gling":18483,"cans":18484,"vernacular":18485,"friedman":18486,"nobleman":18487,"shale":18488,"walnut":18489,"danielle":18490,"##ection":18491,"##tley":18492,"sears":18493,"##kumar":18494,"chords":18495,"lend":18496,"flipping":18497,"streamed":18498,"por":18499,"dracula":18500,"gallons":18501,"sacrifices":18502,"gamble":18503,"orphanage":18504,"##iman":18505,"mckenzie":18506,"##gible":18507,"boxers":18508,"daly":18509,"##balls":18510,"##ان":18511,"208":18512,"##ific":18513,"##rative":18514,"##iq":18515,"exploited":18516,"slated":18517,"##uity":18518,"circling":18519,"hillary":18520,"pinched":18521,"goldberg":18522,"provost":18523,"campaigning":18524,"lim":18525,"piles":18526,"ironically":18527,"jong":18528,"mohan":18529,"successors":18530,"usaf":18531,"##tem":18532,"##ught":18533,"autobiographical":18534,"haute":18535,"preserves":18536,"##ending":18537,"acquitted":18538,"comparisons":18539,"203":18540,"hydroelectric":18541,"gangs":18542,"cypriot":18543,"torpedoes":18544,"rushes":18545,"chrome":18546,"derive":18547,"bumps":18548,"instability":18549,"fiat":18550,"pets":18551,"##mbe":18552,"silas":18553,"dye":18554,"reckless":18555,"settler":18556,"##itation":18557,"info":18558,"heats":18559,"##writing":18560,"176":18561,"canonical":18562,"maltese":18563,"fins":18564,"mushroom":18565,"stacy":18566,"aspen":18567,"avid":18568,"##kur":18569,"##loading":18570,"vickers":18571,"gaston":18572,"hillside":18573,"statutes":18574,"wilde":18575,"gail":18576,"kung":18577,"sabine":18578,"comfortably":18579,"motorcycles":18580,"##rgo":18581,"169":18582,"pneumonia":18583,"fetch":18584,"##sonic":18585,"axel":18586,"faintly":18587,"parallels":18588,"##oop":18589,"mclaren":18590,"spouse":18591,"compton":18592,"interdisciplinary":18593,"miner":18594,"##eni":18595,"181":18596,"clamped":18597,"##chal":18598,"##llah":18599,"separates":18600,"versa":18601,"##mler":18602,"scarborough":18603,"labrador":18604,"##lity":18605,"##osing":18606,"rutgers":18607,"hurdles":18608,"como":18609,"166":18610,"burt":18611,"divers":18612,"##100":18613,"wichita":18614,"cade":18615,"coincided":18616,"##erson":18617,"bruised":18618,"mla":18619,"##pper":18620,"vineyard":18621,"##ili":18622,"##brush":18623,"notch":18624,"mentioning":18625,"jase":18626,"hearted":18627,"kits":18628,"doe":18629,"##acle":18630,"pomerania":18631,"##ady":18632,"ronan":18633,"seizure":18634,"pavel":18635,"problematic":18636,"##zaki":18637,"domenico":18638,"##ulin":18639,"catering":18640,"penelope":18641,"dependence":18642,"parental":18643,"emilio":18644,"ministerial":18645,"atkinson":18646,"##bolic":18647,"clarkson":18648,"chargers":18649,"colby":18650,"grill":18651,"peeked":18652,"arises":18653,"summon":18654,"##aged":18655,"fools":18656,"##grapher":18657,"faculties":18658,"qaeda":18659,"##vial":18660,"garner":18661,"refurbished":18662,"##hwa":18663,"geelong":18664,"disasters":18665,"nudged":18666,"bs":18667,"shareholder":18668,"lori":18669,"algae":18670,"reinstated":18671,"rot":18672,"##ades":18673,"##nous":18674,"invites":18675,"stainless":18676,"183":18677,"inclusive":18678,"##itude":18679,"diocesan":18680,"til":18681,"##icz":18682,"denomination":18683,"##xa":18684,"benton":18685,"floral":18686,"registers":18687,"##ider":18688,"##erman":18689,"##kell":18690,"absurd":18691,"brunei":18692,"guangzhou":18693,"hitter":18694,"retaliation":18695,"##uled":18696,"##eve":18697,"blanc":18698,"nh":18699,"consistency":18700,"contamination":18701,"##eres":18702,"##rner":18703,"dire":18704,"palermo":18705,"broadcasters":18706,"diaries":18707,"inspire":18708,"vols":18709,"brewer":18710,"tightening":18711,"ky":18712,"mixtape":18713,"hormone":18714,"##tok":18715,"stokes":18716,"##color":18717,"##dly":18718,"##ssi":18719,"pg":18720,"##ometer":18721,"##lington":18722,"sanitation":18723,"##tility":18724,"intercontinental":18725,"apps":18726,"##adt":18727,"¹⁄₂":18728,"cylinders":18729,"economies":18730,"favourable":18731,"unison":18732,"croix":18733,"gertrude":18734,"odyssey":18735,"vanity":18736,"dangling":18737,"##logists":18738,"upgrades":18739,"dice":18740,"middleweight":18741,"practitioner":18742,"##ight":18743,"206":18744,"henrik":18745,"parlor":18746,"orion":18747,"angered":18748,"lac":18749,"python":18750,"blurted":18751,"##rri":18752,"sensual":18753,"intends":18754,"swings":18755,"angled":18756,"##phs":18757,"husky":18758,"attain":18759,"peerage":18760,"precinct":18761,"textiles":18762,"cheltenham":18763,"shuffled":18764,"dai":18765,"confess":18766,"tasting":18767,"bhutan":18768,"##riation":18769,"tyrone":18770,"segregation":18771,"abrupt":18772,"ruiz":18773,"##rish":18774,"smirked":18775,"blackwell":18776,"confidential":18777,"browning":18778,"amounted":18779,"##put":18780,"vase":18781,"scarce":18782,"fabulous":18783,"raided":18784,"staple":18785,"guyana":18786,"unemployed":18787,"glider":18788,"shay":18789,"##tow":18790,"carmine":18791,"troll":18792,"intervene":18793,"squash":18794,"superstar":18795,"##uce":18796,"cylindrical":18797,"len":18798,"roadway":18799,"researched":18800,"handy":18801,"##rium":18802,"##jana":18803,"meta":18804,"lao":18805,"declares":18806,"##rring":18807,"##tadt":18808,"##elin":18809,"##kova":18810,"willem":18811,"shrubs":18812,"napoleonic":18813,"realms":18814,"skater":18815,"qi":18816,"volkswagen":18817,"##ł":18818,"tad":18819,"hara":18820,"archaeologist":18821,"awkwardly":18822,"eerie":18823,"##kind":18824,"wiley":18825,"##heimer":18826,"##24":18827,"titus":18828,"organizers":18829,"cfl":18830,"crusaders":18831,"lama":18832,"usb":18833,"vent":18834,"enraged":18835,"thankful":18836,"occupants":18837,"maximilian":18838,"##gaard":18839,"possessing":18840,"textbooks":18841,"##oran":18842,"collaborator":18843,"quaker":18844,"##ulo":18845,"avalanche":18846,"mono":18847,"silky":18848,"straits":18849,"isaiah":18850,"mustang":18851,"surged":18852,"resolutions":18853,"potomac":18854,"descend":18855,"cl":18856,"kilograms":18857,"plato":18858,"strains":18859,"saturdays":18860,"##olin":18861,"bernstein":18862,"##ype":18863,"holstein":18864,"ponytail":18865,"##watch":18866,"belize":18867,"conversely":18868,"heroine":18869,"perpetual":18870,"##ylus":18871,"charcoal":18872,"piedmont":18873,"glee":18874,"negotiating":18875,"backdrop":18876,"prologue":18877,"##jah":18878,"##mmy":18879,"pasadena":18880,"climbs":18881,"ramos":18882,"sunni":18883,"##holm":18884,"##tner":18885,"##tri":18886,"anand":18887,"deficiency":18888,"hertfordshire":18889,"stout":18890,"##avi":18891,"aperture":18892,"orioles":18893,"##irs":18894,"doncaster":18895,"intrigued":18896,"bombed":18897,"coating":18898,"otis":18899,"##mat":18900,"cocktail":18901,"##jit":18902,"##eto":18903,"amir":18904,"arousal":18905,"sar":18906,"##proof":18907,"##act":18908,"##ories":18909,"dixie":18910,"pots":18911,"##bow":18912,"whereabouts":18913,"159":18914,"##fted":18915,"drains":18916,"bullying":18917,"cottages":18918,"scripture":18919,"coherent":18920,"fore":18921,"poe":18922,"appetite":18923,"##uration":18924,"sampled":18925,"##ators":18926,"##dp":18927,"derrick":18928,"rotor":18929,"jays":18930,"peacock":18931,"installment":18932,"##rro":18933,"advisors":18934,"##coming":18935,"rodeo":18936,"scotch":18937,"##mot":18938,"##db":18939,"##fen":18940,"##vant":18941,"ensued":18942,"rodrigo":18943,"dictatorship":18944,"martyrs":18945,"twenties":18946,"##н":18947,"towed":18948,"incidence":18949,"marta":18950,"rainforest":18951,"sai":18952,"scaled":18953,"##cles":18954,"oceanic":18955,"qualifiers":18956,"symphonic":18957,"mcbride":18958,"dislike":18959,"generalized":18960,"aubrey":18961,"colonization":18962,"##iation":18963,"##lion":18964,"##ssing":18965,"disliked":18966,"lublin":18967,"salesman":18968,"##ulates":18969,"spherical":18970,"whatsoever":18971,"sweating":18972,"avalon":18973,"contention":18974,"punt":18975,"severity":18976,"alderman":18977,"atari":18978,"##dina":18979,"##grant":18980,"##rop":18981,"scarf":18982,"seville":18983,"vertices":18984,"annexation":18985,"fairfield":18986,"fascination":18987,"inspiring":18988,"launches":18989,"palatinate":18990,"regretted":18991,"##rca":18992,"feral":18993,"##iom":18994,"elk":18995,"nap":18996,"olsen":18997,"reddy":18998,"yong":18999,"##leader":19000,"##iae":19001,"garment":19002,"transports":19003,"feng":19004,"gracie":19005,"outrage":19006,"viceroy":19007,"insides":19008,"##esis":19009,"breakup":19010,"grady":19011,"organizer":19012,"softer":19013,"grimaced":19014,"222":19015,"murals":19016,"galicia":19017,"arranging":19018,"vectors":19019,"##rsten":19020,"bas":19021,"##sb":19022,"##cens":19023,"sloan":19024,"##eka":19025,"bitten":19026,"ara":19027,"fender":19028,"nausea":19029,"bumped":19030,"kris":19031,"banquet":19032,"comrades":19033,"detector":19034,"persisted":19035,"##llan":19036,"adjustment":19037,"endowed":19038,"cinemas":19039,"##shot":19040,"sellers":19041,"##uman":19042,"peek":19043,"epa":19044,"kindly":19045,"neglect":19046,"simpsons":19047,"talon":19048,"mausoleum":19049,"runaway":19050,"hangul":19051,"lookout":19052,"##cic":19053,"rewards":19054,"coughed":19055,"acquainted":19056,"chloride":19057,"##ald":19058,"quicker":19059,"accordion":19060,"neolithic":19061,"##qa":19062,"artemis":19063,"coefficient":19064,"lenny":19065,"pandora":19066,"tx":19067,"##xed":19068,"ecstasy":19069,"litter":19070,"segunda":19071,"chairperson":19072,"gemma":19073,"hiss":19074,"rumor":19075,"vow":19076,"nasal":19077,"antioch":19078,"compensate":19079,"patiently":19080,"transformers":19081,"##eded":19082,"judo":19083,"morrow":19084,"penis":19085,"posthumous":19086,"philips":19087,"bandits":19088,"husbands":19089,"denote":19090,"flaming":19091,"##any":19092,"##phones":19093,"langley":19094,"yorker":19095,"1760":19096,"walters":19097,"##uo":19098,"##kle":19099,"gubernatorial":19100,"fatty":19101,"samsung":19102,"leroy":19103,"outlaw":19104,"##nine":19105,"unpublished":19106,"poole":19107,"jakob":19108,"##ᵢ":19109,"##ₙ":19110,"crete":19111,"distorted":19112,"superiority":19113,"##dhi":19114,"intercept":19115,"crust":19116,"mig":19117,"claus":19118,"crashes":19119,"positioning":19120,"188":19121,"stallion":19122,"301":19123,"frontal":19124,"armistice":19125,"##estinal":19126,"elton":19127,"aj":19128,"encompassing":19129,"camel":19130,"commemorated":19131,"malaria":19132,"woodward":19133,"calf":19134,"cigar":19135,"penetrate":19136,"##oso":19137,"willard":19138,"##rno":19139,"##uche":19140,"illustrate":19141,"amusing":19142,"convergence":19143,"noteworthy":19144,"##lma":19145,"##rva":19146,"journeys":19147,"realise":19148,"manfred":19149,"##sable":19150,"410":19151,"##vocation":19152,"hearings":19153,"fiance":19154,"##posed":19155,"educators":19156,"provoked":19157,"adjusting":19158,"##cturing":19159,"modular":19160,"stockton":19161,"paterson":19162,"vlad":19163,"rejects":19164,"electors":19165,"selena":19166,"maureen":19167,"##tres":19168,"uber":19169,"##rce":19170,"swirled":19171,"##num":19172,"proportions":19173,"nanny":19174,"pawn":19175,"naturalist":19176,"parma":19177,"apostles":19178,"awoke":19179,"ethel":19180,"wen":19181,"##bey":19182,"monsoon":19183,"overview":19184,"##inating":19185,"mccain":19186,"rendition":19187,"risky":19188,"adorned":19189,"##ih":19190,"equestrian":19191,"germain":19192,"nj":19193,"conspicuous":19194,"confirming":19195,"##yoshi":19196,"shivering":19197,"##imeter":19198,"milestone":19199,"rumours":19200,"flinched":19201,"bounds":19202,"smacked":19203,"token":19204,"##bei":19205,"lectured":19206,"automobiles":19207,"##shore":19208,"impacted":19209,"##iable":19210,"nouns":19211,"nero":19212,"##leaf":19213,"ismail":19214,"prostitute":19215,"trams":19216,"##lace":19217,"bridget":19218,"sud":19219,"stimulus":19220,"impressions":19221,"reins":19222,"revolves":19223,"##oud":19224,"##gned":19225,"giro":19226,"honeymoon":19227,"##swell":19228,"criterion":19229,"##sms":19230,"##uil":19231,"libyan":19232,"prefers":19233,"##osition":19234,"211":19235,"preview":19236,"sucks":19237,"accusation":19238,"bursts":19239,"metaphor":19240,"diffusion":19241,"tolerate":19242,"faye":19243,"betting":19244,"cinematographer":19245,"liturgical":19246,"specials":19247,"bitterly":19248,"humboldt":19249,"##ckle":19250,"flux":19251,"rattled":19252,"##itzer":19253,"archaeologists":19254,"odor":19255,"authorised":19256,"marshes":19257,"discretion":19258,"##ов":19259,"alarmed":19260,"archaic":19261,"inverse":19262,"##leton":19263,"explorers":19264,"##pine":19265,"drummond":19266,"tsunami":19267,"woodlands":19268,"##minate":19269,"##tland":19270,"booklet":19271,"insanity":19272,"owning":19273,"insert":19274,"crafted":19275,"calculus":19276,"##tore":19277,"receivers":19278,"##bt":19279,"stung":19280,"##eca":19281,"##nched":19282,"prevailing":19283,"travellers":19284,"eyeing":19285,"lila":19286,"graphs":19287,"##borne":19288,"178":19289,"julien":19290,"##won":19291,"morale":19292,"adaptive":19293,"therapist":19294,"erica":19295,"cw":19296,"libertarian":19297,"bowman":19298,"pitches":19299,"vita":19300,"##ional":19301,"crook":19302,"##ads":19303,"##entation":19304,"caledonia":19305,"mutiny":19306,"##sible":19307,"1840s":19308,"automation":19309,"##ß":19310,"flock":19311,"##pia":19312,"ironic":19313,"pathology":19314,"##imus":19315,"remarried":19316,"##22":19317,"joker":19318,"withstand":19319,"energies":19320,"##att":19321,"shropshire":19322,"hostages":19323,"madeleine":19324,"tentatively":19325,"conflicting":19326,"mateo":19327,"recipes":19328,"euros":19329,"ol":19330,"mercenaries":19331,"nico":19332,"##ndon":19333,"albuquerque":19334,"augmented":19335,"mythical":19336,"bel":19337,"freud":19338,"##child":19339,"cough":19340,"##lica":19341,"365":19342,"freddy":19343,"lillian":19344,"genetically":19345,"nuremberg":19346,"calder":19347,"209":19348,"bonn":19349,"outdoors":19350,"paste":19351,"suns":19352,"urgency":19353,"vin":19354,"restraint":19355,"tyson":19356,"##cera":19357,"##selle":19358,"barrage":19359,"bethlehem":19360,"kahn":19361,"##par":19362,"mounts":19363,"nippon":19364,"barony":19365,"happier":19366,"ryu":19367,"makeshift":19368,"sheldon":19369,"blushed":19370,"castillo":19371,"barking":19372,"listener":19373,"taped":19374,"bethel":19375,"fluent":19376,"headlines":19377,"pornography":19378,"rum":19379,"disclosure":19380,"sighing":19381,"mace":19382,"doubling":19383,"gunther":19384,"manly":19385,"##plex":19386,"rt":19387,"interventions":19388,"physiological":19389,"forwards":19390,"emerges":19391,"##tooth":19392,"##gny":19393,"compliment":19394,"rib":19395,"recession":19396,"visibly":19397,"barge":19398,"faults":19399,"connector":19400,"exquisite":19401,"prefect":19402,"##rlin":19403,"patio":19404,"##cured":19405,"elevators":19406,"brandt":19407,"italics":19408,"pena":19409,"173":19410,"wasp":19411,"satin":19412,"ea":19413,"botswana":19414,"graceful":19415,"respectable":19416,"##jima":19417,"##rter":19418,"##oic":19419,"franciscan":19420,"generates":19421,"##dl":19422,"alfredo":19423,"disgusting":19424,"##olate":19425,"##iously":19426,"sherwood":19427,"warns":19428,"cod":19429,"promo":19430,"cheryl":19431,"sino":19432,"##ة":19433,"##escu":19434,"twitch":19435,"##zhi":19436,"brownish":19437,"thom":19438,"ortiz":19439,"##dron":19440,"densely":19441,"##beat":19442,"carmel":19443,"reinforce":19444,"##bana":19445,"187":19446,"anastasia":19447,"downhill":19448,"vertex":19449,"contaminated":19450,"remembrance":19451,"harmonic":19452,"homework":19453,"##sol":19454,"fiancee":19455,"gears":19456,"olds":19457,"angelica":19458,"loft":19459,"ramsay":19460,"quiz":19461,"colliery":19462,"sevens":19463,"##cape":19464,"autism":19465,"##hil":19466,"walkway":19467,"##boats":19468,"ruben":19469,"abnormal":19470,"ounce":19471,"khmer":19472,"##bbe":19473,"zachary":19474,"bedside":19475,"morphology":19476,"punching":19477,"##olar":19478,"sparrow":19479,"convinces":19480,"##35":19481,"hewitt":19482,"queer":19483,"remastered":19484,"rods":19485,"mabel":19486,"solemn":19487,"notified":19488,"lyricist":19489,"symmetric":19490,"##xide":19491,"174":19492,"encore":19493,"passports":19494,"wildcats":19495,"##uni":19496,"baja":19497,"##pac":19498,"mildly":19499,"##ease":19500,"bleed":19501,"commodity":19502,"mounds":19503,"glossy":19504,"orchestras":19505,"##omo":19506,"damian":19507,"prelude":19508,"ambitions":19509,"##vet":19510,"awhile":19511,"remotely":19512,"##aud":19513,"asserts":19514,"imply":19515,"##iques":19516,"distinctly":19517,"modelling":19518,"remedy":19519,"##dded":19520,"windshield":19521,"dani":19522,"xiao":19523,"##endra":19524,"audible":19525,"powerplant":19526,"1300":19527,"invalid":19528,"elemental":19529,"acquisitions":19530,"##hala":19531,"immaculate":19532,"libby":19533,"plata":19534,"smuggling":19535,"ventilation":19536,"denoted":19537,"minh":19538,"##morphism":19539,"430":19540,"differed":19541,"dion":19542,"kelley":19543,"lore":19544,"mocking":19545,"sabbath":19546,"spikes":19547,"hygiene":19548,"drown":19549,"runoff":19550,"stylized":19551,"tally":19552,"liberated":19553,"aux":19554,"interpreter":19555,"righteous":19556,"aba":19557,"siren":19558,"reaper":19559,"pearce":19560,"millie":19561,"##cier":19562,"##yra":19563,"gaius":19564,"##iso":19565,"captures":19566,"##ttering":19567,"dorm":19568,"claudio":19569,"##sic":19570,"benches":19571,"knighted":19572,"blackness":19573,"##ored":19574,"discount":19575,"fumble":19576,"oxidation":19577,"routed":19578,"##ς":19579,"novak":19580,"perpendicular":19581,"spoiled":19582,"fracture":19583,"splits":19584,"##urt":19585,"pads":19586,"topology":19587,"##cats":19588,"axes":19589,"fortunate":19590,"offenders":19591,"protestants":19592,"esteem":19593,"221":19594,"broadband":19595,"convened":19596,"frankly":19597,"hound":19598,"prototypes":19599,"isil":19600,"facilitated":19601,"keel":19602,"##sher":19603,"sahara":19604,"awaited":19605,"bubba":19606,"orb":19607,"prosecutors":19608,"186":19609,"hem":19610,"520":19611,"##xing":19612,"relaxing":19613,"remnant":19614,"romney":19615,"sorted":19616,"slalom":19617,"stefano":19618,"ulrich":19619,"##active":19620,"exemption":19621,"folder":19622,"pauses":19623,"foliage":19624,"hitchcock":19625,"epithet":19626,"204":19627,"criticisms":19628,"##aca":19629,"ballistic":19630,"brody":19631,"hinduism":19632,"chaotic":19633,"youths":19634,"equals":19635,"##pala":19636,"pts":19637,"thicker":19638,"analogous":19639,"capitalist":19640,"improvised":19641,"overseeing":19642,"sinatra":19643,"ascended":19644,"beverage":19645,"##tl":19646,"straightforward":19647,"##kon":19648,"curran":19649,"##west":19650,"bois":19651,"325":19652,"induce":19653,"surveying":19654,"emperors":19655,"sax":19656,"unpopular":19657,"##kk":19658,"cartoonist":19659,"fused":19660,"##mble":19661,"unto":19662,"##yuki":19663,"localities":19664,"##cko":19665,"##ln":19666,"darlington":19667,"slain":19668,"academie":19669,"lobbying":19670,"sediment":19671,"puzzles":19672,"##grass":19673,"defiance":19674,"dickens":19675,"manifest":19676,"tongues":19677,"alumnus":19678,"arbor":19679,"coincide":19680,"184":19681,"appalachian":19682,"mustafa":19683,"examiner":19684,"cabaret":19685,"traumatic":19686,"yves":19687,"bracelet":19688,"draining":19689,"heroin":19690,"magnum":19691,"baths":19692,"odessa":19693,"consonants":19694,"mitsubishi":19695,"##gua":19696,"kellan":19697,"vaudeville":19698,"##fr":19699,"joked":19700,"null":19701,"straps":19702,"probation":19703,"##ław":19704,"ceded":19705,"interfaces":19706,"##pas":19707,"##zawa":19708,"blinding":19709,"viet":19710,"224":19711,"rothschild":19712,"museo":19713,"640":19714,"huddersfield":19715,"##vr":19716,"tactic":19717,"##storm":19718,"brackets":19719,"dazed":19720,"incorrectly":19721,"##vu":19722,"reg":19723,"glazed":19724,"fearful":19725,"manifold":19726,"benefited":19727,"irony":19728,"##sun":19729,"stumbling":19730,"##rte":19731,"willingness":19732,"balkans":19733,"mei":19734,"wraps":19735,"##aba":19736,"injected":19737,"##lea":19738,"gu":19739,"syed":19740,"harmless":19741,"##hammer":19742,"bray":19743,"takeoff":19744,"poppy":19745,"timor":19746,"cardboard":19747,"astronaut":19748,"purdue":19749,"weeping":19750,"southbound":19751,"cursing":19752,"stalls":19753,"diagonal":19754,"##neer":19755,"lamar":19756,"bryce":19757,"comte":19758,"weekdays":19759,"harrington":19760,"##uba":19761,"negatively":19762,"##see":19763,"lays":19764,"grouping":19765,"##cken":19766,"##henko":19767,"affirmed":19768,"halle":19769,"modernist":19770,"##lai":19771,"hodges":19772,"smelling":19773,"aristocratic":19774,"baptized":19775,"dismiss":19776,"justification":19777,"oilers":19778,"##now":19779,"coupling":19780,"qin":19781,"snack":19782,"healer":19783,"##qing":19784,"gardener":19785,"layla":19786,"battled":19787,"formulated":19788,"stephenson":19789,"gravitational":19790,"##gill":19791,"##jun":19792,"1768":19793,"granny":19794,"coordinating":19795,"suites":19796,"##cd":19797,"##ioned":19798,"monarchs":19799,"##cote":19800,"##hips":19801,"sep":19802,"blended":19803,"apr":19804,"barrister":19805,"deposition":19806,"fia":19807,"mina":19808,"policemen":19809,"paranoid":19810,"##pressed":19811,"churchyard":19812,"covert":19813,"crumpled":19814,"creep":19815,"abandoning":19816,"tr":19817,"transmit":19818,"conceal":19819,"barr":19820,"understands":19821,"readiness":19822,"spire":19823,"##cology":19824,"##enia":19825,"##erry":19826,"610":19827,"startling":19828,"unlock":19829,"vida":19830,"bowled":19831,"slots":19832,"##nat":19833,"##islav":19834,"spaced":19835,"trusting":19836,"admire":19837,"rig":19838,"##ink":19839,"slack":19840,"##70":19841,"mv":19842,"207":19843,"casualty":19844,"##wei":19845,"classmates":19846,"##odes":19847,"##rar":19848,"##rked":19849,"amherst":19850,"furnished":19851,"evolve":19852,"foundry":19853,"menace":19854,"mead":19855,"##lein":19856,"flu":19857,"wesleyan":19858,"##kled":19859,"monterey":19860,"webber":19861,"##vos":19862,"wil":19863,"##mith":19864,"##на":19865,"bartholomew":19866,"justices":19867,"restrained":19868,"##cke":19869,"amenities":19870,"191":19871,"mediated":19872,"sewage":19873,"trenches":19874,"ml":19875,"mainz":19876,"##thus":19877,"1800s":19878,"##cula":19879,"##inski":19880,"caine":19881,"bonding":19882,"213":19883,"converts":19884,"spheres":19885,"superseded":19886,"marianne":19887,"crypt":19888,"sweaty":19889,"ensign":19890,"historia":19891,"##br":19892,"spruce":19893,"##post":19894,"##ask":19895,"forks":19896,"thoughtfully":19897,"yukon":19898,"pamphlet":19899,"ames":19900,"##uter":19901,"karma":19902,"##yya":19903,"bryn":19904,"negotiation":19905,"sighs":19906,"incapable":19907,"##mbre":19908,"##ntial":19909,"actresses":19910,"taft":19911,"##mill":19912,"luce":19913,"prevailed":19914,"##amine":19915,"1773":19916,"motionless":19917,"envoy":19918,"testify":19919,"investing":19920,"sculpted":19921,"instructors":19922,"provence":19923,"kali":19924,"cullen":19925,"horseback":19926,"##while":19927,"goodwin":19928,"##jos":19929,"gaa":19930,"norte":19931,"##ldon":19932,"modify":19933,"wavelength":19934,"abd":19935,"214":19936,"skinned":19937,"sprinter":19938,"forecast":19939,"scheduling":19940,"marries":19941,"squared":19942,"tentative":19943,"##chman":19944,"boer":19945,"##isch":19946,"bolts":19947,"swap":19948,"fisherman":19949,"assyrian":19950,"impatiently":19951,"guthrie":19952,"martins":19953,"murdoch":19954,"194":19955,"tanya":19956,"nicely":19957,"dolly":19958,"lacy":19959,"med":19960,"##45":19961,"syn":19962,"decks":19963,"fashionable":19964,"millionaire":19965,"##ust":19966,"surfing":19967,"##ml":19968,"##ision":19969,"heaved":19970,"tammy":19971,"consulate":19972,"attendees":19973,"routinely":19974,"197":19975,"fuse":19976,"saxophonist":19977,"backseat":19978,"malaya":19979,"##lord":19980,"scowl":19981,"tau":19982,"##ishly":19983,"193":19984,"sighted":19985,"steaming":19986,"##rks":19987,"303":19988,"911":19989,"##holes":19990,"##hong":19991,"ching":19992,"##wife":19993,"bless":19994,"conserved":19995,"jurassic":19996,"stacey":19997,"unix":19998,"zion":19999,"chunk":20000,"rigorous":20001,"blaine":20002,"198":20003,"peabody":20004,"slayer":20005,"dismay":20006,"brewers":20007,"nz":20008,"##jer":20009,"det":20010,"##glia":20011,"glover":20012,"postwar":20013,"int":20014,"penetration":20015,"sylvester":20016,"imitation":20017,"vertically":20018,"airlift":20019,"heiress":20020,"knoxville":20021,"viva":20022,"##uin":20023,"390":20024,"macon":20025,"##rim":20026,"##fighter":20027,"##gonal":20028,"janice":20029,"##orescence":20030,"##wari":20031,"marius":20032,"belongings":20033,"leicestershire":20034,"196":20035,"blanco":20036,"inverted":20037,"preseason":20038,"sanity":20039,"sobbing":20040,"##due":20041,"##elt":20042,"##dled":20043,"collingwood":20044,"regeneration":20045,"flickering":20046,"shortest":20047,"##mount":20048,"##osi":20049,"feminism":20050,"##lat":20051,"sherlock":20052,"cabinets":20053,"fumbled":20054,"northbound":20055,"precedent":20056,"snaps":20057,"##mme":20058,"researching":20059,"##akes":20060,"guillaume":20061,"insights":20062,"manipulated":20063,"vapor":20064,"neighbour":20065,"sap":20066,"gangster":20067,"frey":20068,"f1":20069,"stalking":20070,"scarcely":20071,"callie":20072,"barnett":20073,"tendencies":20074,"audi":20075,"doomed":20076,"assessing":20077,"slung":20078,"panchayat":20079,"ambiguous":20080,"bartlett":20081,"##etto":20082,"distributing":20083,"violating":20084,"wolverhampton":20085,"##hetic":20086,"swami":20087,"histoire":20088,"##urus":20089,"liable":20090,"pounder":20091,"groin":20092,"hussain":20093,"larsen":20094,"popping":20095,"surprises":20096,"##atter":20097,"vie":20098,"curt":20099,"##station":20100,"mute":20101,"relocate":20102,"musicals":20103,"authorization":20104,"richter":20105,"##sef":20106,"immortality":20107,"tna":20108,"bombings":20109,"##press":20110,"deteriorated":20111,"yiddish":20112,"##acious":20113,"robbed":20114,"colchester":20115,"cs":20116,"pmid":20117,"ao":20118,"verified":20119,"balancing":20120,"apostle":20121,"swayed":20122,"recognizable":20123,"oxfordshire":20124,"retention":20125,"nottinghamshire":20126,"contender":20127,"judd":20128,"invitational":20129,"shrimp":20130,"uhf":20131,"##icient":20132,"cleaner":20133,"longitudinal":20134,"tanker":20135,"##mur":20136,"acronym":20137,"broker":20138,"koppen":20139,"sundance":20140,"suppliers":20141,"##gil":20142,"4000":20143,"clipped":20144,"fuels":20145,"petite":20146,"##anne":20147,"landslide":20148,"helene":20149,"diversion":20150,"populous":20151,"landowners":20152,"auspices":20153,"melville":20154,"quantitative":20155,"##xes":20156,"ferries":20157,"nicky":20158,"##llus":20159,"doo":20160,"haunting":20161,"roche":20162,"carver":20163,"downed":20164,"unavailable":20165,"##pathy":20166,"approximation":20167,"hiroshima":20168,"##hue":20169,"garfield":20170,"valle":20171,"comparatively":20172,"keyboardist":20173,"traveler":20174,"##eit":20175,"congestion":20176,"calculating":20177,"subsidiaries":20178,"##bate":20179,"serb":20180,"modernization":20181,"fairies":20182,"deepened":20183,"ville":20184,"averages":20185,"##lore":20186,"inflammatory":20187,"tonga":20188,"##itch":20189,"co₂":20190,"squads":20191,"##hea":20192,"gigantic":20193,"serum":20194,"enjoyment":20195,"retailer":20196,"verona":20197,"35th":20198,"cis":20199,"##phobic":20200,"magna":20201,"technicians":20202,"##vati":20203,"arithmetic":20204,"##sport":20205,"levin":20206,"##dation":20207,"amtrak":20208,"chow":20209,"sienna":20210,"##eyer":20211,"backstage":20212,"entrepreneurship":20213,"##otic":20214,"learnt":20215,"tao":20216,"##udy":20217,"worcestershire":20218,"formulation":20219,"baggage":20220,"hesitant":20221,"bali":20222,"sabotage":20223,"##kari":20224,"barren":20225,"enhancing":20226,"murmur":20227,"pl":20228,"freshly":20229,"putnam":20230,"syntax":20231,"aces":20232,"medicines":20233,"resentment":20234,"bandwidth":20235,"##sier":20236,"grins":20237,"chili":20238,"guido":20239,"##sei":20240,"framing":20241,"implying":20242,"gareth":20243,"lissa":20244,"genevieve":20245,"pertaining":20246,"admissions":20247,"geo":20248,"thorpe":20249,"proliferation":20250,"sato":20251,"bela":20252,"analyzing":20253,"parting":20254,"##gor":20255,"awakened":20256,"##isman":20257,"huddled":20258,"secrecy":20259,"##kling":20260,"hush":20261,"gentry":20262,"540":20263,"dungeons":20264,"##ego":20265,"coasts":20266,"##utz":20267,"sacrificed":20268,"##chule":20269,"landowner":20270,"mutually":20271,"prevalence":20272,"programmer":20273,"adolescent":20274,"disrupted":20275,"seaside":20276,"gee":20277,"trusts":20278,"vamp":20279,"georgie":20280,"##nesian":20281,"##iol":20282,"schedules":20283,"sindh":20284,"##market":20285,"etched":20286,"hm":20287,"sparse":20288,"bey":20289,"beaux":20290,"scratching":20291,"gliding":20292,"unidentified":20293,"216":20294,"collaborating":20295,"gems":20296,"jesuits":20297,"oro":20298,"accumulation":20299,"shaping":20300,"mbe":20301,"anal":20302,"##xin":20303,"231":20304,"enthusiasts":20305,"newscast":20306,"##egan":20307,"janata":20308,"dewey":20309,"parkinson":20310,"179":20311,"ankara":20312,"biennial":20313,"towering":20314,"dd":20315,"inconsistent":20316,"950":20317,"##chet":20318,"thriving":20319,"terminate":20320,"cabins":20321,"furiously":20322,"eats":20323,"advocating":20324,"donkey":20325,"marley":20326,"muster":20327,"phyllis":20328,"leiden":20329,"##user":20330,"grassland":20331,"glittering":20332,"iucn":20333,"loneliness":20334,"217":20335,"memorandum":20336,"armenians":20337,"##ddle":20338,"popularized":20339,"rhodesia":20340,"60s":20341,"lame":20342,"##illon":20343,"sans":20344,"bikini":20345,"header":20346,"orbits":20347,"##xx":20348,"##finger":20349,"##ulator":20350,"sharif":20351,"spines":20352,"biotechnology":20353,"strolled":20354,"naughty":20355,"yates":20356,"##wire":20357,"fremantle":20358,"milo":20359,"##mour":20360,"abducted":20361,"removes":20362,"##atin":20363,"humming":20364,"wonderland":20365,"##chrome":20366,"##ester":20367,"hume":20368,"pivotal":20369,"##rates":20370,"armand":20371,"grams":20372,"believers":20373,"elector":20374,"rte":20375,"apron":20376,"bis":20377,"scraped":20378,"##yria":20379,"endorsement":20380,"initials":20381,"##llation":20382,"eps":20383,"dotted":20384,"hints":20385,"buzzing":20386,"emigration":20387,"nearer":20388,"##tom":20389,"indicators":20390,"##ulu":20391,"coarse":20392,"neutron":20393,"protectorate":20394,"##uze":20395,"directional":20396,"exploits":20397,"pains":20398,"loire":20399,"1830s":20400,"proponents":20401,"guggenheim":20402,"rabbits":20403,"ritchie":20404,"305":20405,"hectare":20406,"inputs":20407,"hutton":20408,"##raz":20409,"verify":20410,"##ako":20411,"boilers":20412,"longitude":20413,"##lev":20414,"skeletal":20415,"yer":20416,"emilia":20417,"citrus":20418,"compromised":20419,"##gau":20420,"pokemon":20421,"prescription":20422,"paragraph":20423,"eduard":20424,"cadillac":20425,"attire":20426,"categorized":20427,"kenyan":20428,"weddings":20429,"charley":20430,"##bourg":20431,"entertain":20432,"monmouth":20433,"##lles":20434,"nutrients":20435,"davey":20436,"mesh":20437,"incentive":20438,"practised":20439,"ecosystems":20440,"kemp":20441,"subdued":20442,"overheard":20443,"##rya":20444,"bodily":20445,"maxim":20446,"##nius":20447,"apprenticeship":20448,"ursula":20449,"##fight":20450,"lodged":20451,"rug":20452,"silesian":20453,"unconstitutional":20454,"patel":20455,"inspected":20456,"coyote":20457,"unbeaten":20458,"##hak":20459,"34th":20460,"disruption":20461,"convict":20462,"parcel":20463,"##cl":20464,"##nham":20465,"collier":20466,"implicated":20467,"mallory":20468,"##iac":20469,"##lab":20470,"susannah":20471,"winkler":20472,"##rber":20473,"shia":20474,"phelps":20475,"sediments":20476,"graphical":20477,"robotic":20478,"##sner":20479,"adulthood":20480,"mart":20481,"smoked":20482,"##isto":20483,"kathryn":20484,"clarified":20485,"##aran":20486,"divides":20487,"convictions":20488,"oppression":20489,"pausing":20490,"burying":20491,"##mt":20492,"federico":20493,"mathias":20494,"eileen":20495,"##tana":20496,"kite":20497,"hunched":20498,"##acies":20499,"189":20500,"##atz":20501,"disadvantage":20502,"liza":20503,"kinetic":20504,"greedy":20505,"paradox":20506,"yokohama":20507,"dowager":20508,"trunks":20509,"ventured":20510,"##gement":20511,"gupta":20512,"vilnius":20513,"olaf":20514,"##thest":20515,"crimean":20516,"hopper":20517,"##ej":20518,"progressively":20519,"arturo":20520,"mouthed":20521,"arrondissement":20522,"##fusion":20523,"rubin":20524,"simulcast":20525,"oceania":20526,"##orum":20527,"##stra":20528,"##rred":20529,"busiest":20530,"intensely":20531,"navigator":20532,"cary":20533,"##vine":20534,"##hini":20535,"##bies":20536,"fife":20537,"rowe":20538,"rowland":20539,"posing":20540,"insurgents":20541,"shafts":20542,"lawsuits":20543,"activate":20544,"conor":20545,"inward":20546,"culturally":20547,"garlic":20548,"265":20549,"##eering":20550,"eclectic":20551,"##hui":20552,"##kee":20553,"##nl":20554,"furrowed":20555,"vargas":20556,"meteorological":20557,"rendezvous":20558,"##aus":20559,"culinary":20560,"commencement":20561,"##dition":20562,"quota":20563,"##notes":20564,"mommy":20565,"salaries":20566,"overlapping":20567,"mule":20568,"##iology":20569,"##mology":20570,"sums":20571,"wentworth":20572,"##isk":20573,"##zione":20574,"mainline":20575,"subgroup":20576,"##illy":20577,"hack":20578,"plaintiff":20579,"verdi":20580,"bulb":20581,"differentiation":20582,"engagements":20583,"multinational":20584,"supplemented":20585,"bertrand":20586,"caller":20587,"regis":20588,"##naire":20589,"##sler":20590,"##arts":20591,"##imated":20592,"blossom":20593,"propagation":20594,"kilometer":20595,"viaduct":20596,"vineyards":20597,"##uate":20598,"beckett":20599,"optimization":20600,"golfer":20601,"songwriters":20602,"seminal":20603,"semitic":20604,"thud":20605,"volatile":20606,"evolving":20607,"ridley":20608,"##wley":20609,"trivial":20610,"distributions":20611,"scandinavia":20612,"jiang":20613,"##ject":20614,"wrestled":20615,"insistence":20616,"##dio":20617,"emphasizes":20618,"napkin":20619,"##ods":20620,"adjunct":20621,"rhyme":20622,"##ricted":20623,"##eti":20624,"hopeless":20625,"surrounds":20626,"tremble":20627,"32nd":20628,"smoky":20629,"##ntly":20630,"oils":20631,"medicinal":20632,"padded":20633,"steer":20634,"wilkes":20635,"219":20636,"255":20637,"concessions":20638,"hue":20639,"uniquely":20640,"blinded":20641,"landon":20642,"yahoo":20643,"##lane":20644,"hendrix":20645,"commemorating":20646,"dex":20647,"specify":20648,"chicks":20649,"##ggio":20650,"intercity":20651,"1400":20652,"morley":20653,"##torm":20654,"highlighting":20655,"##oting":20656,"pang":20657,"oblique":20658,"stalled":20659,"##liner":20660,"flirting":20661,"newborn":20662,"1769":20663,"bishopric":20664,"shaved":20665,"232":20666,"currie":20667,"##ush":20668,"dharma":20669,"spartan":20670,"##ooped":20671,"favorites":20672,"smug":20673,"novella":20674,"sirens":20675,"abusive":20676,"creations":20677,"espana":20678,"##lage":20679,"paradigm":20680,"semiconductor":20681,"sheen":20682,"##rdo":20683,"##yen":20684,"##zak":20685,"nrl":20686,"renew":20687,"##pose":20688,"##tur":20689,"adjutant":20690,"marches":20691,"norma":20692,"##enity":20693,"ineffective":20694,"weimar":20695,"grunt":20696,"##gat":20697,"lordship":20698,"plotting":20699,"expenditure":20700,"infringement":20701,"lbs":20702,"refrain":20703,"av":20704,"mimi":20705,"mistakenly":20706,"postmaster":20707,"1771":20708,"##bara":20709,"ras":20710,"motorsports":20711,"tito":20712,"199":20713,"subjective":20714,"##zza":20715,"bully":20716,"stew":20717,"##kaya":20718,"prescott":20719,"1a":20720,"##raphic":20721,"##zam":20722,"bids":20723,"styling":20724,"paranormal":20725,"reeve":20726,"sneaking":20727,"exploding":20728,"katz":20729,"akbar":20730,"migrant":20731,"syllables":20732,"indefinitely":20733,"##ogical":20734,"destroys":20735,"replaces":20736,"applause":20737,"##phine":20738,"pest":20739,"##fide":20740,"218":20741,"articulated":20742,"bertie":20743,"##thing":20744,"##cars":20745,"##ptic":20746,"courtroom":20747,"crowley":20748,"aesthetics":20749,"cummings":20750,"tehsil":20751,"hormones":20752,"titanic":20753,"dangerously":20754,"##ibe":20755,"stadion":20756,"jaenelle":20757,"auguste":20758,"ciudad":20759,"##chu":20760,"mysore":20761,"partisans":20762,"##sio":20763,"lucan":20764,"philipp":20765,"##aly":20766,"debating":20767,"henley":20768,"interiors":20769,"##rano":20770,"##tious":20771,"homecoming":20772,"beyonce":20773,"usher":20774,"henrietta":20775,"prepares":20776,"weeds":20777,"##oman":20778,"ely":20779,"plucked":20780,"##pire":20781,"##dable":20782,"luxurious":20783,"##aq":20784,"artifact":20785,"password":20786,"pasture":20787,"juno":20788,"maddy":20789,"minsk":20790,"##dder":20791,"##ologies":20792,"##rone":20793,"assessments":20794,"martian":20795,"royalist":20796,"1765":20797,"examines":20798,"##mani":20799,"##rge":20800,"nino":20801,"223":20802,"parry":20803,"scooped":20804,"relativity":20805,"##eli":20806,"##uting":20807,"##cao":20808,"congregational":20809,"noisy":20810,"traverse":20811,"##agawa":20812,"strikeouts":20813,"nickelodeon":20814,"obituary":20815,"transylvania":20816,"binds":20817,"depictions":20818,"polk":20819,"trolley":20820,"##yed":20821,"##lard":20822,"breeders":20823,"##under":20824,"dryly":20825,"hokkaido":20826,"1762":20827,"strengths":20828,"stacks":20829,"bonaparte":20830,"connectivity":20831,"neared":20832,"prostitutes":20833,"stamped":20834,"anaheim":20835,"gutierrez":20836,"sinai":20837,"##zzling":20838,"bram":20839,"fresno":20840,"madhya":20841,"##86":20842,"proton":20843,"##lena":20844,"##llum":20845,"##phon":20846,"reelected":20847,"wanda":20848,"##anus":20849,"##lb":20850,"ample":20851,"distinguishing":20852,"##yler":20853,"grasping":20854,"sermons":20855,"tomato":20856,"bland":20857,"stimulation":20858,"avenues":20859,"##eux":20860,"spreads":20861,"scarlett":20862,"fern":20863,"pentagon":20864,"assert":20865,"baird":20866,"chesapeake":20867,"ir":20868,"calmed":20869,"distortion":20870,"fatalities":20871,"##olis":20872,"correctional":20873,"pricing":20874,"##astic":20875,"##gina":20876,"prom":20877,"dammit":20878,"ying":20879,"collaborate":20880,"##chia":20881,"welterweight":20882,"33rd":20883,"pointer":20884,"substitution":20885,"bonded":20886,"umpire":20887,"communicating":20888,"multitude":20889,"paddle":20890,"##obe":20891,"federally":20892,"intimacy":20893,"##insky":20894,"betray":20895,"ssr":20896,"##lett":20897,"##lean":20898,"##lves":20899,"##therapy":20900,"airbus":20901,"##tery":20902,"functioned":20903,"ud":20904,"bearer":20905,"biomedical":20906,"netflix":20907,"##hire":20908,"##nca":20909,"condom":20910,"brink":20911,"ik":20912,"##nical":20913,"macy":20914,"##bet":20915,"flap":20916,"gma":20917,"experimented":20918,"jelly":20919,"lavender":20920,"##icles":20921,"##ulia":20922,"munro":20923,"##mian":20924,"##tial":20925,"rye":20926,"##rle":20927,"60th":20928,"gigs":20929,"hottest":20930,"rotated":20931,"predictions":20932,"fuji":20933,"bu":20934,"##erence":20935,"##omi":20936,"barangay":20937,"##fulness":20938,"##sas":20939,"clocks":20940,"##rwood":20941,"##liness":20942,"cereal":20943,"roe":20944,"wight":20945,"decker":20946,"uttered":20947,"babu":20948,"onion":20949,"xml":20950,"forcibly":20951,"##df":20952,"petra":20953,"sarcasm":20954,"hartley":20955,"peeled":20956,"storytelling":20957,"##42":20958,"##xley":20959,"##ysis":20960,"##ffa":20961,"fibre":20962,"kiel":20963,"auditor":20964,"fig":20965,"harald":20966,"greenville":20967,"##berries":20968,"geographically":20969,"nell":20970,"quartz":20971,"##athic":20972,"cemeteries":20973,"##lr":20974,"crossings":20975,"nah":20976,"holloway":20977,"reptiles":20978,"chun":20979,"sichuan":20980,"snowy":20981,"660":20982,"corrections":20983,"##ivo":20984,"zheng":20985,"ambassadors":20986,"blacksmith":20987,"fielded":20988,"fluids":20989,"hardcover":20990,"turnover":20991,"medications":20992,"melvin":20993,"academies":20994,"##erton":20995,"ro":20996,"roach":20997,"absorbing":20998,"spaniards":20999,"colton":21000,"##founded":21001,"outsider":21002,"espionage":21003,"kelsey":21004,"245":21005,"edible":21006,"##ulf":21007,"dora":21008,"establishes":21009,"##sham":21010,"##tries":21011,"contracting":21012,"##tania":21013,"cinematic":21014,"costello":21015,"nesting":21016,"##uron":21017,"connolly":21018,"duff":21019,"##nology":21020,"mma":21021,"##mata":21022,"fergus":21023,"sexes":21024,"gi":21025,"optics":21026,"spectator":21027,"woodstock":21028,"banning":21029,"##hee":21030,"##fle":21031,"differentiate":21032,"outfielder":21033,"refinery":21034,"226":21035,"312":21036,"gerhard":21037,"horde":21038,"lair":21039,"drastically":21040,"##udi":21041,"landfall":21042,"##cheng":21043,"motorsport":21044,"odi":21045,"##achi":21046,"predominant":21047,"quay":21048,"skins":21049,"##ental":21050,"edna":21051,"harshly":21052,"complementary":21053,"murdering":21054,"##aves":21055,"wreckage":21056,"##90":21057,"ono":21058,"outstretched":21059,"lennox":21060,"munitions":21061,"galen":21062,"reconcile":21063,"470":21064,"scalp":21065,"bicycles":21066,"gillespie":21067,"questionable":21068,"rosenberg":21069,"guillermo":21070,"hostel":21071,"jarvis":21072,"kabul":21073,"volvo":21074,"opium":21075,"yd":21076,"##twined":21077,"abuses":21078,"decca":21079,"outpost":21080,"##cino":21081,"sensible":21082,"neutrality":21083,"##64":21084,"ponce":21085,"anchorage":21086,"atkins":21087,"turrets":21088,"inadvertently":21089,"disagree":21090,"libre":21091,"vodka":21092,"reassuring":21093,"weighs":21094,"##yal":21095,"glide":21096,"jumper":21097,"ceilings":21098,"repertory":21099,"outs":21100,"stain":21101,"##bial":21102,"envy":21103,"##ucible":21104,"smashing":21105,"heightened":21106,"policing":21107,"hyun":21108,"mixes":21109,"lai":21110,"prima":21111,"##ples":21112,"celeste":21113,"##bina":21114,"lucrative":21115,"intervened":21116,"kc":21117,"manually":21118,"##rned":21119,"stature":21120,"staffed":21121,"bun":21122,"bastards":21123,"nairobi":21124,"priced":21125,"##auer":21126,"thatcher":21127,"##kia":21128,"tripped":21129,"comune":21130,"##ogan":21131,"##pled":21132,"brasil":21133,"incentives":21134,"emanuel":21135,"hereford":21136,"musica":21137,"##kim":21138,"benedictine":21139,"biennale":21140,"##lani":21141,"eureka":21142,"gardiner":21143,"rb":21144,"knocks":21145,"sha":21146,"##ael":21147,"##elled":21148,"##onate":21149,"efficacy":21150,"ventura":21151,"masonic":21152,"sanford":21153,"maize":21154,"leverage":21155,"##feit":21156,"capacities":21157,"santana":21158,"##aur":21159,"novelty":21160,"vanilla":21161,"##cter":21162,"##tour":21163,"benin":21164,"##oir":21165,"##rain":21166,"neptune":21167,"drafting":21168,"tallinn":21169,"##cable":21170,"humiliation":21171,"##boarding":21172,"schleswig":21173,"fabian":21174,"bernardo":21175,"liturgy":21176,"spectacle":21177,"sweeney":21178,"pont":21179,"routledge":21180,"##tment":21181,"cosmos":21182,"ut":21183,"hilt":21184,"sleek":21185,"universally":21186,"##eville":21187,"##gawa":21188,"typed":21189,"##dry":21190,"favors":21191,"allegheny":21192,"glaciers":21193,"##rly":21194,"recalling":21195,"aziz":21196,"##log":21197,"parasite":21198,"requiem":21199,"auf":21200,"##berto":21201,"##llin":21202,"illumination":21203,"##breaker":21204,"##issa":21205,"festivities":21206,"bows":21207,"govern":21208,"vibe":21209,"vp":21210,"333":21211,"sprawled":21212,"larson":21213,"pilgrim":21214,"bwf":21215,"leaping":21216,"##rts":21217,"##ssel":21218,"alexei":21219,"greyhound":21220,"hoarse":21221,"##dler":21222,"##oration":21223,"seneca":21224,"##cule":21225,"gaping":21226,"##ulously":21227,"##pura":21228,"cinnamon":21229,"##gens":21230,"##rricular":21231,"craven":21232,"fantasies":21233,"houghton":21234,"engined":21235,"reigned":21236,"dictator":21237,"supervising":21238,"##oris":21239,"bogota":21240,"commentaries":21241,"unnatural":21242,"fingernails":21243,"spirituality":21244,"tighten":21245,"##tm":21246,"canadiens":21247,"protesting":21248,"intentional":21249,"cheers":21250,"sparta":21251,"##ytic":21252,"##iere":21253,"##zine":21254,"widen":21255,"belgarath":21256,"controllers":21257,"dodd":21258,"iaaf":21259,"navarre":21260,"##ication":21261,"defect":21262,"squire":21263,"steiner":21264,"whisky":21265,"##mins":21266,"560":21267,"inevitably":21268,"tome":21269,"##gold":21270,"chew":21271,"##uid":21272,"##lid":21273,"elastic":21274,"##aby":21275,"streaked":21276,"alliances":21277,"jailed":21278,"regal":21279,"##ined":21280,"##phy":21281,"czechoslovak":21282,"narration":21283,"absently":21284,"##uld":21285,"bluegrass":21286,"guangdong":21287,"quran":21288,"criticizing":21289,"hose":21290,"hari":21291,"##liest":21292,"##owa":21293,"skier":21294,"streaks":21295,"deploy":21296,"##lom":21297,"raft":21298,"bose":21299,"dialed":21300,"huff":21301,"##eira":21302,"haifa":21303,"simplest":21304,"bursting":21305,"endings":21306,"ib":21307,"sultanate":21308,"##titled":21309,"franks":21310,"whitman":21311,"ensures":21312,"sven":21313,"##ggs":21314,"collaborators":21315,"forster":21316,"organising":21317,"ui":21318,"banished":21319,"napier":21320,"injustice":21321,"teller":21322,"layered":21323,"thump":21324,"##otti":21325,"roc":21326,"battleships":21327,"evidenced":21328,"fugitive":21329,"sadie":21330,"robotics":21331,"##roud":21332,"equatorial":21333,"geologist":21334,"##iza":21335,"yielding":21336,"##bron":21337,"##sr":21338,"internationale":21339,"mecca":21340,"##diment":21341,"sbs":21342,"skyline":21343,"toad":21344,"uploaded":21345,"reflective":21346,"undrafted":21347,"lal":21348,"leafs":21349,"bayern":21350,"##dai":21351,"lakshmi":21352,"shortlisted":21353,"##stick":21354,"##wicz":21355,"camouflage":21356,"donate":21357,"af":21358,"christi":21359,"lau":21360,"##acio":21361,"disclosed":21362,"nemesis":21363,"1761":21364,"assemble":21365,"straining":21366,"northamptonshire":21367,"tal":21368,"##asi":21369,"bernardino":21370,"premature":21371,"heidi":21372,"42nd":21373,"coefficients":21374,"galactic":21375,"reproduce":21376,"buzzed":21377,"sensations":21378,"zionist":21379,"monsieur":21380,"myrtle":21381,"##eme":21382,"archery":21383,"strangled":21384,"musically":21385,"viewpoint":21386,"antiquities":21387,"bei":21388,"trailers":21389,"seahawks":21390,"cured":21391,"pee":21392,"preferring":21393,"tasmanian":21394,"lange":21395,"sul":21396,"##mail":21397,"##working":21398,"colder":21399,"overland":21400,"lucivar":21401,"massey":21402,"gatherings":21403,"haitian":21404,"##smith":21405,"disapproval":21406,"flaws":21407,"##cco":21408,"##enbach":21409,"1766":21410,"npr":21411,"##icular":21412,"boroughs":21413,"creole":21414,"forums":21415,"techno":21416,"1755":21417,"dent":21418,"abdominal":21419,"streetcar":21420,"##eson":21421,"##stream":21422,"procurement":21423,"gemini":21424,"predictable":21425,"##tya":21426,"acheron":21427,"christoph":21428,"feeder":21429,"fronts":21430,"vendor":21431,"bernhard":21432,"jammu":21433,"tumors":21434,"slang":21435,"##uber":21436,"goaltender":21437,"twists":21438,"curving":21439,"manson":21440,"vuelta":21441,"mer":21442,"peanut":21443,"confessions":21444,"pouch":21445,"unpredictable":21446,"allowance":21447,"theodor":21448,"vascular":21449,"##factory":21450,"bala":21451,"authenticity":21452,"metabolic":21453,"coughing":21454,"nanjing":21455,"##cea":21456,"pembroke":21457,"##bard":21458,"splendid":21459,"36th":21460,"ff":21461,"hourly":21462,"##ahu":21463,"elmer":21464,"handel":21465,"##ivate":21466,"awarding":21467,"thrusting":21468,"dl":21469,"experimentation":21470,"##hesion":21471,"##46":21472,"caressed":21473,"entertained":21474,"steak":21475,"##rangle":21476,"biologist":21477,"orphans":21478,"baroness":21479,"oyster":21480,"stepfather":21481,"##dridge":21482,"mirage":21483,"reefs":21484,"speeding":21485,"##31":21486,"barons":21487,"1764":21488,"227":21489,"inhabit":21490,"preached":21491,"repealed":21492,"##tral":21493,"honoring":21494,"boogie":21495,"captives":21496,"administer":21497,"johanna":21498,"##imate":21499,"gel":21500,"suspiciously":21501,"1767":21502,"sobs":21503,"##dington":21504,"backbone":21505,"hayward":21506,"garry":21507,"##folding":21508,"##nesia":21509,"maxi":21510,"##oof":21511,"##ppe":21512,"ellison":21513,"galileo":21514,"##stand":21515,"crimea":21516,"frenzy":21517,"amour":21518,"bumper":21519,"matrices":21520,"natalia":21521,"baking":21522,"garth":21523,"palestinians":21524,"##grove":21525,"smack":21526,"conveyed":21527,"ensembles":21528,"gardening":21529,"##manship":21530,"##rup":21531,"##stituting":21532,"1640":21533,"harvesting":21534,"topography":21535,"jing":21536,"shifters":21537,"dormitory":21538,"##carriage":21539,"##lston":21540,"ist":21541,"skulls":21542,"##stadt":21543,"dolores":21544,"jewellery":21545,"sarawak":21546,"##wai":21547,"##zier":21548,"fences":21549,"christy":21550,"confinement":21551,"tumbling":21552,"credibility":21553,"fir":21554,"stench":21555,"##bria":21556,"##plication":21557,"##nged":21558,"##sam":21559,"virtues":21560,"##belt":21561,"marjorie":21562,"pba":21563,"##eem":21564,"##made":21565,"celebrates":21566,"schooner":21567,"agitated":21568,"barley":21569,"fulfilling":21570,"anthropologist":21571,"##pro":21572,"restrict":21573,"novi":21574,"regulating":21575,"##nent":21576,"padres":21577,"##rani":21578,"##hesive":21579,"loyola":21580,"tabitha":21581,"milky":21582,"olson":21583,"proprietor":21584,"crambidae":21585,"guarantees":21586,"intercollegiate":21587,"ljubljana":21588,"hilda":21589,"##sko":21590,"ignorant":21591,"hooded":21592,"##lts":21593,"sardinia":21594,"##lidae":21595,"##vation":21596,"frontman":21597,"privileged":21598,"witchcraft":21599,"##gp":21600,"jammed":21601,"laude":21602,"poking":21603,"##than":21604,"bracket":21605,"amazement":21606,"yunnan":21607,"##erus":21608,"maharaja":21609,"linnaeus":21610,"264":21611,"commissioning":21612,"milano":21613,"peacefully":21614,"##logies":21615,"akira":21616,"rani":21617,"regulator":21618,"##36":21619,"grasses":21620,"##rance":21621,"luzon":21622,"crows":21623,"compiler":21624,"gretchen":21625,"seaman":21626,"edouard":21627,"tab":21628,"buccaneers":21629,"ellington":21630,"hamlets":21631,"whig":21632,"socialists":21633,"##anto":21634,"directorial":21635,"easton":21636,"mythological":21637,"##kr":21638,"##vary":21639,"rhineland":21640,"semantic":21641,"taut":21642,"dune":21643,"inventions":21644,"succeeds":21645,"##iter":21646,"replication":21647,"branched":21648,"##pired":21649,"jul":21650,"prosecuted":21651,"kangaroo":21652,"penetrated":21653,"##avian":21654,"middlesbrough":21655,"doses":21656,"bleak":21657,"madam":21658,"predatory":21659,"relentless":21660,"##vili":21661,"reluctance":21662,"##vir":21663,"hailey":21664,"crore":21665,"silvery":21666,"1759":21667,"monstrous":21668,"swimmers":21669,"transmissions":21670,"hawthorn":21671,"informing":21672,"##eral":21673,"toilets":21674,"caracas":21675,"crouch":21676,"kb":21677,"##sett":21678,"295":21679,"cartel":21680,"hadley":21681,"##aling":21682,"alexia":21683,"yvonne":21684,"##biology":21685,"cinderella":21686,"eton":21687,"superb":21688,"blizzard":21689,"stabbing":21690,"industrialist":21691,"maximus":21692,"##gm":21693,"##orus":21694,"groves":21695,"maud":21696,"clade":21697,"oversized":21698,"comedic":21699,"##bella":21700,"rosen":21701,"nomadic":21702,"fulham":21703,"montane":21704,"beverages":21705,"galaxies":21706,"redundant":21707,"swarm":21708,"##rot":21709,"##folia":21710,"##llis":21711,"buckinghamshire":21712,"fen":21713,"bearings":21714,"bahadur":21715,"##rom":21716,"gilles":21717,"phased":21718,"dynamite":21719,"faber":21720,"benoit":21721,"vip":21722,"##ount":21723,"##wd":21724,"booking":21725,"fractured":21726,"tailored":21727,"anya":21728,"spices":21729,"westwood":21730,"cairns":21731,"auditions":21732,"inflammation":21733,"steamed":21734,"##rocity":21735,"##acion":21736,"##urne":21737,"skyla":21738,"thereof":21739,"watford":21740,"torment":21741,"archdeacon":21742,"transforms":21743,"lulu":21744,"demeanor":21745,"fucked":21746,"serge":21747,"##sor":21748,"mckenna":21749,"minas":21750,"entertainer":21751,"##icide":21752,"caress":21753,"originate":21754,"residue":21755,"##sty":21756,"1740":21757,"##ilised":21758,"##org":21759,"beech":21760,"##wana":21761,"subsidies":21762,"##ghton":21763,"emptied":21764,"gladstone":21765,"ru":21766,"firefighters":21767,"voodoo":21768,"##rcle":21769,"het":21770,"nightingale":21771,"tamara":21772,"edmond":21773,"ingredient":21774,"weaknesses":21775,"silhouette":21776,"285":21777,"compatibility":21778,"withdrawing":21779,"hampson":21780,"##mona":21781,"anguish":21782,"giggling":21783,"##mber":21784,"bookstore":21785,"##jiang":21786,"southernmost":21787,"tilting":21788,"##vance":21789,"bai":21790,"economical":21791,"rf":21792,"briefcase":21793,"dreadful":21794,"hinted":21795,"projections":21796,"shattering":21797,"totaling":21798,"##rogate":21799,"analogue":21800,"indicted":21801,"periodical":21802,"fullback":21803,"##dman":21804,"haynes":21805,"##tenberg":21806,"##ffs":21807,"##ishment":21808,"1745":21809,"thirst":21810,"stumble":21811,"penang":21812,"vigorous":21813,"##ddling":21814,"##kor":21815,"##lium":21816,"octave":21817,"##ove":21818,"##enstein":21819,"##inen":21820,"##ones":21821,"siberian":21822,"##uti":21823,"cbn":21824,"repeal":21825,"swaying":21826,"##vington":21827,"khalid":21828,"tanaka":21829,"unicorn":21830,"otago":21831,"plastered":21832,"lobe":21833,"riddle":21834,"##rella":21835,"perch":21836,"##ishing":21837,"croydon":21838,"filtered":21839,"graeme":21840,"tripoli":21841,"##ossa":21842,"crocodile":21843,"##chers":21844,"sufi":21845,"mined":21846,"##tung":21847,"inferno":21848,"lsu":21849,"##phi":21850,"swelled":21851,"utilizes":21852,"£2":21853,"cale":21854,"periodicals":21855,"styx":21856,"hike":21857,"informally":21858,"coop":21859,"lund":21860,"##tidae":21861,"ala":21862,"hen":21863,"qui":21864,"transformations":21865,"disposed":21866,"sheath":21867,"chickens":21868,"##cade":21869,"fitzroy":21870,"sas":21871,"silesia":21872,"unacceptable":21873,"odisha":21874,"1650":21875,"sabrina":21876,"pe":21877,"spokane":21878,"ratios":21879,"athena":21880,"massage":21881,"shen":21882,"dilemma":21883,"##drum":21884,"##riz":21885,"##hul":21886,"corona":21887,"doubtful":21888,"niall":21889,"##pha":21890,"##bino":21891,"fines":21892,"cite":21893,"acknowledging":21894,"bangor":21895,"ballard":21896,"bathurst":21897,"##resh":21898,"huron":21899,"mustered":21900,"alzheimer":21901,"garments":21902,"kinase":21903,"tyre":21904,"warship":21905,"##cp":21906,"flashback":21907,"pulmonary":21908,"braun":21909,"cheat":21910,"kamal":21911,"cyclists":21912,"constructions":21913,"grenades":21914,"ndp":21915,"traveller":21916,"excuses":21917,"stomped":21918,"signalling":21919,"trimmed":21920,"futsal":21921,"mosques":21922,"relevance":21923,"##wine":21924,"wta":21925,"##23":21926,"##vah":21927,"##lter":21928,"hoc":21929,"##riding":21930,"optimistic":21931,"##´s":21932,"deco":21933,"sim":21934,"interacting":21935,"rejecting":21936,"moniker":21937,"waterways":21938,"##ieri":21939,"##oku":21940,"mayors":21941,"gdansk":21942,"outnumbered":21943,"pearls":21944,"##ended":21945,"##hampton":21946,"fairs":21947,"totals":21948,"dominating":21949,"262":21950,"notions":21951,"stairway":21952,"compiling":21953,"pursed":21954,"commodities":21955,"grease":21956,"yeast":21957,"##jong":21958,"carthage":21959,"griffiths":21960,"residual":21961,"amc":21962,"contraction":21963,"laird":21964,"sapphire":21965,"##marine":21966,"##ivated":21967,"amalgamation":21968,"dissolve":21969,"inclination":21970,"lyle":21971,"packaged":21972,"altitudes":21973,"suez":21974,"canons":21975,"graded":21976,"lurched":21977,"narrowing":21978,"boasts":21979,"guise":21980,"wed":21981,"enrico":21982,"##ovsky":21983,"rower":21984,"scarred":21985,"bree":21986,"cub":21987,"iberian":21988,"protagonists":21989,"bargaining":21990,"proposing":21991,"trainers":21992,"voyages":21993,"vans":21994,"fishes":21995,"##aea":21996,"##ivist":21997,"##verance":21998,"encryption":21999,"artworks":22000,"kazan":22001,"sabre":22002,"cleopatra":22003,"hepburn":22004,"rotting":22005,"supremacy":22006,"mecklenburg":22007,"##brate":22008,"burrows":22009,"hazards":22010,"outgoing":22011,"flair":22012,"organizes":22013,"##ctions":22014,"scorpion":22015,"##usions":22016,"boo":22017,"234":22018,"chevalier":22019,"dunedin":22020,"slapping":22021,"##34":22022,"ineligible":22023,"pensions":22024,"##38":22025,"##omic":22026,"manufactures":22027,"emails":22028,"bismarck":22029,"238":22030,"weakening":22031,"blackish":22032,"ding":22033,"mcgee":22034,"quo":22035,"##rling":22036,"northernmost":22037,"xx":22038,"manpower":22039,"greed":22040,"sampson":22041,"clicking":22042,"##ange":22043,"##horpe":22044,"##inations":22045,"##roving":22046,"torre":22047,"##eptive":22048,"##moral":22049,"symbolism":22050,"38th":22051,"asshole":22052,"meritorious":22053,"outfits":22054,"splashed":22055,"biographies":22056,"sprung":22057,"astros":22058,"##tale":22059,"302":22060,"737":22061,"filly":22062,"raoul":22063,"nw":22064,"tokugawa":22065,"linden":22066,"clubhouse":22067,"##apa":22068,"tracts":22069,"romano":22070,"##pio":22071,"putin":22072,"tags":22073,"##note":22074,"chained":22075,"dickson":22076,"gunshot":22077,"moe":22078,"gunn":22079,"rashid":22080,"##tails":22081,"zipper":22082,"##bas":22083,"##nea":22084,"contrasted":22085,"##ply":22086,"##udes":22087,"plum":22088,"pharaoh":22089,"##pile":22090,"aw":22091,"comedies":22092,"ingrid":22093,"sandwiches":22094,"subdivisions":22095,"1100":22096,"mariana":22097,"nokia":22098,"kamen":22099,"hz":22100,"delaney":22101,"veto":22102,"herring":22103,"##words":22104,"possessive":22105,"outlines":22106,"##roup":22107,"siemens":22108,"stairwell":22109,"rc":22110,"gallantry":22111,"messiah":22112,"palais":22113,"yells":22114,"233":22115,"zeppelin":22116,"##dm":22117,"bolivar":22118,"##cede":22119,"smackdown":22120,"mckinley":22121,"##mora":22122,"##yt":22123,"muted":22124,"geologic":22125,"finely":22126,"unitary":22127,"avatar":22128,"hamas":22129,"maynard":22130,"rees":22131,"bog":22132,"contrasting":22133,"##rut":22134,"liv":22135,"chico":22136,"disposition":22137,"pixel":22138,"##erate":22139,"becca":22140,"dmitry":22141,"yeshiva":22142,"narratives":22143,"##lva":22144,"##ulton":22145,"mercenary":22146,"sharpe":22147,"tempered":22148,"navigate":22149,"stealth":22150,"amassed":22151,"keynes":22152,"##lini":22153,"untouched":22154,"##rrie":22155,"havoc":22156,"lithium":22157,"##fighting":22158,"abyss":22159,"graf":22160,"southward":22161,"wolverine":22162,"balloons":22163,"implements":22164,"ngos":22165,"transitions":22166,"##icum":22167,"ambushed":22168,"concacaf":22169,"dormant":22170,"economists":22171,"##dim":22172,"costing":22173,"csi":22174,"rana":22175,"universite":22176,"boulders":22177,"verity":22178,"##llon":22179,"collin":22180,"mellon":22181,"misses":22182,"cypress":22183,"fluorescent":22184,"lifeless":22185,"spence":22186,"##ulla":22187,"crewe":22188,"shepard":22189,"pak":22190,"revelations":22191,"##م":22192,"jolly":22193,"gibbons":22194,"paw":22195,"##dro":22196,"##quel":22197,"freeing":22198,"##test":22199,"shack":22200,"fries":22201,"palatine":22202,"##51":22203,"##hiko":22204,"accompaniment":22205,"cruising":22206,"recycled":22207,"##aver":22208,"erwin":22209,"sorting":22210,"synthesizers":22211,"dyke":22212,"realities":22213,"sg":22214,"strides":22215,"enslaved":22216,"wetland":22217,"##ghan":22218,"competence":22219,"gunpowder":22220,"grassy":22221,"maroon":22222,"reactors":22223,"objection":22224,"##oms":22225,"carlson":22226,"gearbox":22227,"macintosh":22228,"radios":22229,"shelton":22230,"##sho":22231,"clergyman":22232,"prakash":22233,"254":22234,"mongols":22235,"trophies":22236,"oricon":22237,"228":22238,"stimuli":22239,"twenty20":22240,"cantonese":22241,"cortes":22242,"mirrored":22243,"##saurus":22244,"bhp":22245,"cristina":22246,"melancholy":22247,"##lating":22248,"enjoyable":22249,"nuevo":22250,"##wny":22251,"downfall":22252,"schumacher":22253,"##ind":22254,"banging":22255,"lausanne":22256,"rumbled":22257,"paramilitary":22258,"reflex":22259,"ax":22260,"amplitude":22261,"migratory":22262,"##gall":22263,"##ups":22264,"midi":22265,"barnard":22266,"lastly":22267,"sherry":22268,"##hp":22269,"##nall":22270,"keystone":22271,"##kra":22272,"carleton":22273,"slippery":22274,"##53":22275,"coloring":22276,"foe":22277,"socket":22278,"otter":22279,"##rgos":22280,"mats":22281,"##tose":22282,"consultants":22283,"bafta":22284,"bison":22285,"topping":22286,"##km":22287,"490":22288,"primal":22289,"abandonment":22290,"transplant":22291,"atoll":22292,"hideous":22293,"mort":22294,"pained":22295,"reproduced":22296,"tae":22297,"howling":22298,"##turn":22299,"unlawful":22300,"billionaire":22301,"hotter":22302,"poised":22303,"lansing":22304,"##chang":22305,"dinamo":22306,"retro":22307,"messing":22308,"nfc":22309,"domesday":22310,"##mina":22311,"blitz":22312,"timed":22313,"##athing":22314,"##kley":22315,"ascending":22316,"gesturing":22317,"##izations":22318,"signaled":22319,"tis":22320,"chinatown":22321,"mermaid":22322,"savanna":22323,"jameson":22324,"##aint":22325,"catalina":22326,"##pet":22327,"##hers":22328,"cochrane":22329,"cy":22330,"chatting":22331,"##kus":22332,"alerted":22333,"computation":22334,"mused":22335,"noelle":22336,"majestic":22337,"mohawk":22338,"campo":22339,"octagonal":22340,"##sant":22341,"##hend":22342,"241":22343,"aspiring":22344,"##mart":22345,"comprehend":22346,"iona":22347,"paralyzed":22348,"shimmering":22349,"swindon":22350,"rhone":22351,"##eley":22352,"reputed":22353,"configurations":22354,"pitchfork":22355,"agitation":22356,"francais":22357,"gillian":22358,"lipstick":22359,"##ilo":22360,"outsiders":22361,"pontifical":22362,"resisting":22363,"bitterness":22364,"sewer":22365,"rockies":22366,"##edd":22367,"##ucher":22368,"misleading":22369,"1756":22370,"exiting":22371,"galloway":22372,"##nging":22373,"risked":22374,"##heart":22375,"246":22376,"commemoration":22377,"schultz":22378,"##rka":22379,"integrating":22380,"##rsa":22381,"poses":22382,"shrieked":22383,"##weiler":22384,"guineas":22385,"gladys":22386,"jerking":22387,"owls":22388,"goldsmith":22389,"nightly":22390,"penetrating":22391,"##unced":22392,"lia":22393,"##33":22394,"ignited":22395,"betsy":22396,"##aring":22397,"##thorpe":22398,"follower":22399,"vigorously":22400,"##rave":22401,"coded":22402,"kiran":22403,"knit":22404,"zoology":22405,"tbilisi":22406,"##28":22407,"##bered":22408,"repository":22409,"govt":22410,"deciduous":22411,"dino":22412,"growling":22413,"##bba":22414,"enhancement":22415,"unleashed":22416,"chanting":22417,"pussy":22418,"biochemistry":22419,"##eric":22420,"kettle":22421,"repression":22422,"toxicity":22423,"nrhp":22424,"##arth":22425,"##kko":22426,"##bush":22427,"ernesto":22428,"commended":22429,"outspoken":22430,"242":22431,"mca":22432,"parchment":22433,"sms":22434,"kristen":22435,"##aton":22436,"bisexual":22437,"raked":22438,"glamour":22439,"navajo":22440,"a2":22441,"conditioned":22442,"showcased":22443,"##hma":22444,"spacious":22445,"youthful":22446,"##esa":22447,"usl":22448,"appliances":22449,"junta":22450,"brest":22451,"layne":22452,"conglomerate":22453,"enchanted":22454,"chao":22455,"loosened":22456,"picasso":22457,"circulating":22458,"inspect":22459,"montevideo":22460,"##centric":22461,"##kti":22462,"piazza":22463,"spurred":22464,"##aith":22465,"bari":22466,"freedoms":22467,"poultry":22468,"stamford":22469,"lieu":22470,"##ect":22471,"indigo":22472,"sarcastic":22473,"bahia":22474,"stump":22475,"attach":22476,"dvds":22477,"frankenstein":22478,"lille":22479,"approx":22480,"scriptures":22481,"pollen":22482,"##script":22483,"nmi":22484,"overseen":22485,"##ivism":22486,"tides":22487,"proponent":22488,"newmarket":22489,"inherit":22490,"milling":22491,"##erland":22492,"centralized":22493,"##rou":22494,"distributors":22495,"credentials":22496,"drawers":22497,"abbreviation":22498,"##lco":22499,"##xon":22500,"downing":22501,"uncomfortably":22502,"ripe":22503,"##oes":22504,"erase":22505,"franchises":22506,"##ever":22507,"populace":22508,"##bery":22509,"##khar":22510,"decomposition":22511,"pleas":22512,"##tet":22513,"daryl":22514,"sabah":22515,"##stle":22516,"##wide":22517,"fearless":22518,"genie":22519,"lesions":22520,"annette":22521,"##ogist":22522,"oboe":22523,"appendix":22524,"nair":22525,"dripped":22526,"petitioned":22527,"maclean":22528,"mosquito":22529,"parrot":22530,"rpg":22531,"hampered":22532,"1648":22533,"operatic":22534,"reservoirs":22535,"##tham":22536,"irrelevant":22537,"jolt":22538,"summarized":22539,"##fp":22540,"medallion":22541,"##taff":22542,"##−":22543,"clawed":22544,"harlow":22545,"narrower":22546,"goddard":22547,"marcia":22548,"bodied":22549,"fremont":22550,"suarez":22551,"altering":22552,"tempest":22553,"mussolini":22554,"porn":22555,"##isms":22556,"sweetly":22557,"oversees":22558,"walkers":22559,"solitude":22560,"grimly":22561,"shrines":22562,"hk":22563,"ich":22564,"supervisors":22565,"hostess":22566,"dietrich":22567,"legitimacy":22568,"brushes":22569,"expressive":22570,"##yp":22571,"dissipated":22572,"##rse":22573,"localized":22574,"systemic":22575,"##nikov":22576,"gettysburg":22577,"##js":22578,"##uaries":22579,"dialogues":22580,"muttering":22581,"251":22582,"housekeeper":22583,"sicilian":22584,"discouraged":22585,"##frey":22586,"beamed":22587,"kaladin":22588,"halftime":22589,"kidnap":22590,"##amo":22591,"##llet":22592,"1754":22593,"synonymous":22594,"depleted":22595,"instituto":22596,"insulin":22597,"reprised":22598,"##opsis":22599,"clashed":22600,"##ctric":22601,"interrupting":22602,"radcliffe":22603,"insisting":22604,"medici":22605,"1715":22606,"ejected":22607,"playfully":22608,"turbulent":22609,"##47":22610,"starvation":22611,"##rini":22612,"shipment":22613,"rebellious":22614,"petersen":22615,"verification":22616,"merits":22617,"##rified":22618,"cakes":22619,"##charged":22620,"1757":22621,"milford":22622,"shortages":22623,"spying":22624,"fidelity":22625,"##aker":22626,"emitted":22627,"storylines":22628,"harvested":22629,"seismic":22630,"##iform":22631,"cheung":22632,"kilda":22633,"theoretically":22634,"barbie":22635,"lynx":22636,"##rgy":22637,"##tius":22638,"goblin":22639,"mata":22640,"poisonous":22641,"##nburg":22642,"reactive":22643,"residues":22644,"obedience":22645,"##евич":22646,"conjecture":22647,"##rac":22648,"401":22649,"hating":22650,"sixties":22651,"kicker":22652,"moaning":22653,"motown":22654,"##bha":22655,"emancipation":22656,"neoclassical":22657,"##hering":22658,"consoles":22659,"ebert":22660,"professorship":22661,"##tures":22662,"sustaining":22663,"assaults":22664,"obeyed":22665,"affluent":22666,"incurred":22667,"tornadoes":22668,"##eber":22669,"##zow":22670,"emphasizing":22671,"highlanders":22672,"cheated":22673,"helmets":22674,"##ctus":22675,"internship":22676,"terence":22677,"bony":22678,"executions":22679,"legislators":22680,"berries":22681,"peninsular":22682,"tinged":22683,"##aco":22684,"1689":22685,"amplifier":22686,"corvette":22687,"ribbons":22688,"lavish":22689,"pennant":22690,"##lander":22691,"worthless":22692,"##chfield":22693,"##forms":22694,"mariano":22695,"pyrenees":22696,"expenditures":22697,"##icides":22698,"chesterfield":22699,"mandir":22700,"tailor":22701,"39th":22702,"sergey":22703,"nestled":22704,"willed":22705,"aristocracy":22706,"devotees":22707,"goodnight":22708,"raaf":22709,"rumored":22710,"weaponry":22711,"remy":22712,"appropriations":22713,"harcourt":22714,"burr":22715,"riaa":22716,"##lence":22717,"limitation":22718,"unnoticed":22719,"guo":22720,"soaking":22721,"swamps":22722,"##tica":22723,"collapsing":22724,"tatiana":22725,"descriptive":22726,"brigham":22727,"psalm":22728,"##chment":22729,"maddox":22730,"##lization":22731,"patti":22732,"caliph":22733,"##aja":22734,"akron":22735,"injuring":22736,"serra":22737,"##ganj":22738,"basins":22739,"##sari":22740,"astonished":22741,"launcher":22742,"##church":22743,"hilary":22744,"wilkins":22745,"sewing":22746,"##sf":22747,"stinging":22748,"##fia":22749,"##ncia":22750,"underwood":22751,"startup":22752,"##ition":22753,"compilations":22754,"vibrations":22755,"embankment":22756,"jurist":22757,"##nity":22758,"bard":22759,"juventus":22760,"groundwater":22761,"kern":22762,"palaces":22763,"helium":22764,"boca":22765,"cramped":22766,"marissa":22767,"soto":22768,"##worm":22769,"jae":22770,"princely":22771,"##ggy":22772,"faso":22773,"bazaar":22774,"warmly":22775,"##voking":22776,"229":22777,"pairing":22778,"##lite":22779,"##grate":22780,"##nets":22781,"wien":22782,"freaked":22783,"ulysses":22784,"rebirth":22785,"##alia":22786,"##rent":22787,"mummy":22788,"guzman":22789,"jimenez":22790,"stilled":22791,"##nitz":22792,"trajectory":22793,"tha":22794,"woken":22795,"archival":22796,"professions":22797,"##pts":22798,"##pta":22799,"hilly":22800,"shadowy":22801,"shrink":22802,"##bolt":22803,"norwood":22804,"glued":22805,"migrate":22806,"stereotypes":22807,"devoid":22808,"##pheus":22809,"625":22810,"evacuate":22811,"horrors":22812,"infancy":22813,"gotham":22814,"knowles":22815,"optic":22816,"downloaded":22817,"sachs":22818,"kingsley":22819,"parramatta":22820,"darryl":22821,"mor":22822,"##onale":22823,"shady":22824,"commence":22825,"confesses":22826,"kan":22827,"##meter":22828,"##placed":22829,"marlborough":22830,"roundabout":22831,"regents":22832,"frigates":22833,"io":22834,"##imating":22835,"gothenburg":22836,"revoked":22837,"carvings":22838,"clockwise":22839,"convertible":22840,"intruder":22841,"##sche":22842,"banged":22843,"##ogo":22844,"vicky":22845,"bourgeois":22846,"##mony":22847,"dupont":22848,"footing":22849,"##gum":22850,"pd":22851,"##real":22852,"buckle":22853,"yun":22854,"penthouse":22855,"sane":22856,"720":22857,"serviced":22858,"stakeholders":22859,"neumann":22860,"bb":22861,"##eers":22862,"comb":22863,"##gam":22864,"catchment":22865,"pinning":22866,"rallies":22867,"typing":22868,"##elles":22869,"forefront":22870,"freiburg":22871,"sweetie":22872,"giacomo":22873,"widowed":22874,"goodwill":22875,"worshipped":22876,"aspirations":22877,"midday":22878,"##vat":22879,"fishery":22880,"##trick":22881,"bournemouth":22882,"turk":22883,"243":22884,"hearth":22885,"ethanol":22886,"guadalajara":22887,"murmurs":22888,"sl":22889,"##uge":22890,"afforded":22891,"scripted":22892,"##hta":22893,"wah":22894,"##jn":22895,"coroner":22896,"translucent":22897,"252":22898,"memorials":22899,"puck":22900,"progresses":22901,"clumsy":22902,"##race":22903,"315":22904,"candace":22905,"recounted":22906,"##27":22907,"##slin":22908,"##uve":22909,"filtering":22910,"##mac":22911,"howl":22912,"strata":22913,"heron":22914,"leveled":22915,"##ays":22916,"dubious":22917,"##oja":22918,"##т":22919,"##wheel":22920,"citations":22921,"exhibiting":22922,"##laya":22923,"##mics":22924,"##pods":22925,"turkic":22926,"##lberg":22927,"injunction":22928,"##ennial":22929,"##mit":22930,"antibodies":22931,"##44":22932,"organise":22933,"##rigues":22934,"cardiovascular":22935,"cushion":22936,"inverness":22937,"##zquez":22938,"dia":22939,"cocoa":22940,"sibling":22941,"##tman":22942,"##roid":22943,"expanse":22944,"feasible":22945,"tunisian":22946,"algiers":22947,"##relli":22948,"rus":22949,"bloomberg":22950,"dso":22951,"westphalia":22952,"bro":22953,"tacoma":22954,"281":22955,"downloads":22956,"##ours":22957,"konrad":22958,"duran":22959,"##hdi":22960,"continuum":22961,"jett":22962,"compares":22963,"legislator":22964,"secession":22965,"##nable":22966,"##gues":22967,"##zuka":22968,"translating":22969,"reacher":22970,"##gley":22971,"##ła":22972,"aleppo":22973,"##agi":22974,"tc":22975,"orchards":22976,"trapping":22977,"linguist":22978,"versatile":22979,"drumming":22980,"postage":22981,"calhoun":22982,"superiors":22983,"##mx":22984,"barefoot":22985,"leary":22986,"##cis":22987,"ignacio":22988,"alfa":22989,"kaplan":22990,"##rogen":22991,"bratislava":22992,"mori":22993,"##vot":22994,"disturb":22995,"haas":22996,"313":22997,"cartridges":22998,"gilmore":22999,"radiated":23000,"salford":23001,"tunic":23002,"hades":23003,"##ulsive":23004,"archeological":23005,"delilah":23006,"magistrates":23007,"auditioned":23008,"brewster":23009,"charters":23010,"empowerment":23011,"blogs":23012,"cappella":23013,"dynasties":23014,"iroquois":23015,"whipping":23016,"##krishna":23017,"raceway":23018,"truths":23019,"myra":23020,"weaken":23021,"judah":23022,"mcgregor":23023,"##horse":23024,"mic":23025,"refueling":23026,"37th":23027,"burnley":23028,"bosses":23029,"markus":23030,"premio":23031,"query":23032,"##gga":23033,"dunbar":23034,"##economic":23035,"darkest":23036,"lyndon":23037,"sealing":23038,"commendation":23039,"reappeared":23040,"##mun":23041,"addicted":23042,"ezio":23043,"slaughtered":23044,"satisfactory":23045,"shuffle":23046,"##eves":23047,"##thic":23048,"##uj":23049,"fortification":23050,"warrington":23051,"##otto":23052,"resurrected":23053,"fargo":23054,"mane":23055,"##utable":23056,"##lei":23057,"##space":23058,"foreword":23059,"ox":23060,"##aris":23061,"##vern":23062,"abrams":23063,"hua":23064,"##mento":23065,"sakura":23066,"##alo":23067,"uv":23068,"sentimental":23069,"##skaya":23070,"midfield":23071,"##eses":23072,"sturdy":23073,"scrolls":23074,"macleod":23075,"##kyu":23076,"entropy":23077,"##lance":23078,"mitochondrial":23079,"cicero":23080,"excelled":23081,"thinner":23082,"convoys":23083,"perceive":23084,"##oslav":23085,"##urable":23086,"systematically":23087,"grind":23088,"burkina":23089,"287":23090,"##tagram":23091,"ops":23092,"##aman":23093,"guantanamo":23094,"##cloth":23095,"##tite":23096,"forcefully":23097,"wavy":23098,"##jou":23099,"pointless":23100,"##linger":23101,"##tze":23102,"layton":23103,"portico":23104,"superficial":23105,"clerical":23106,"outlaws":23107,"##hism":23108,"burials":23109,"muir":23110,"##inn":23111,"creditors":23112,"hauling":23113,"rattle":23114,"##leg":23115,"calais":23116,"monde":23117,"archers":23118,"reclaimed":23119,"dwell":23120,"wexford":23121,"hellenic":23122,"falsely":23123,"remorse":23124,"##tek":23125,"dough":23126,"furnishings":23127,"##uttered":23128,"gabon":23129,"neurological":23130,"novice":23131,"##igraphy":23132,"contemplated":23133,"pulpit":23134,"nightstand":23135,"saratoga":23136,"##istan":23137,"documenting":23138,"pulsing":23139,"taluk":23140,"##firmed":23141,"busted":23142,"marital":23143,"##rien":23144,"disagreements":23145,"wasps":23146,"##yes":23147,"hodge":23148,"mcdonnell":23149,"mimic":23150,"fran":23151,"pendant":23152,"dhabi":23153,"musa":23154,"##nington":23155,"congratulations":23156,"argent":23157,"darrell":23158,"concussion":23159,"losers":23160,"regrets":23161,"thessaloniki":23162,"reversal":23163,"donaldson":23164,"hardwood":23165,"thence":23166,"achilles":23167,"ritter":23168,"##eran":23169,"demonic":23170,"jurgen":23171,"prophets":23172,"goethe":23173,"eki":23174,"classmate":23175,"buff":23176,"##cking":23177,"yank":23178,"irrational":23179,"##inging":23180,"perished":23181,"seductive":23182,"qur":23183,"sourced":23184,"##crat":23185,"##typic":23186,"mustard":23187,"ravine":23188,"barre":23189,"horizontally":23190,"characterization":23191,"phylogenetic":23192,"boise":23193,"##dit":23194,"##runner":23195,"##tower":23196,"brutally":23197,"intercourse":23198,"seduce":23199,"##bbing":23200,"fay":23201,"ferris":23202,"ogden":23203,"amar":23204,"nik":23205,"unarmed":23206,"##inator":23207,"evaluating":23208,"kyrgyzstan":23209,"sweetness":23210,"##lford":23211,"##oki":23212,"mccormick":23213,"meiji":23214,"notoriety":23215,"stimulate":23216,"disrupt":23217,"figuring":23218,"instructional":23219,"mcgrath":23220,"##zoo":23221,"groundbreaking":23222,"##lto":23223,"flinch":23224,"khorasan":23225,"agrarian":23226,"bengals":23227,"mixer":23228,"radiating":23229,"##sov":23230,"ingram":23231,"pitchers":23232,"nad":23233,"tariff":23234,"##cript":23235,"tata":23236,"##codes":23237,"##emi":23238,"##ungen":23239,"appellate":23240,"lehigh":23241,"##bled":23242,"##giri":23243,"brawl":23244,"duct":23245,"texans":23246,"##ciation":23247,"##ropolis":23248,"skipper":23249,"speculative":23250,"vomit":23251,"doctrines":23252,"stresses":23253,"253":23254,"davy":23255,"graders":23256,"whitehead":23257,"jozef":23258,"timely":23259,"cumulative":23260,"haryana":23261,"paints":23262,"appropriately":23263,"boon":23264,"cactus":23265,"##ales":23266,"##pid":23267,"dow":23268,"legions":23269,"##pit":23270,"perceptions":23271,"1730":23272,"picturesque":23273,"##yse":23274,"periphery":23275,"rune":23276,"wr":23277,"##aha":23278,"celtics":23279,"sentencing":23280,"whoa":23281,"##erin":23282,"confirms":23283,"variance":23284,"425":23285,"moines":23286,"mathews":23287,"spade":23288,"rave":23289,"m1":23290,"fronted":23291,"fx":23292,"blending":23293,"alleging":23294,"reared":23295,"##gl":23296,"237":23297,"##paper":23298,"grassroots":23299,"eroded":23300,"##free":23301,"##physical":23302,"directs":23303,"ordeal":23304,"##sław":23305,"accelerate":23306,"hacker":23307,"rooftop":23308,"##inia":23309,"lev":23310,"buys":23311,"cebu":23312,"devote":23313,"##lce":23314,"specialising":23315,"##ulsion":23316,"choreographed":23317,"repetition":23318,"warehouses":23319,"##ryl":23320,"paisley":23321,"tuscany":23322,"analogy":23323,"sorcerer":23324,"hash":23325,"huts":23326,"shards":23327,"descends":23328,"exclude":23329,"nix":23330,"chaplin":23331,"gaga":23332,"ito":23333,"vane":23334,"##drich":23335,"causeway":23336,"misconduct":23337,"limo":23338,"orchestrated":23339,"glands":23340,"jana":23341,"##kot":23342,"u2":23343,"##mple":23344,"##sons":23345,"branching":23346,"contrasts":23347,"scoop":23348,"longed":23349,"##virus":23350,"chattanooga":23351,"##75":23352,"syrup":23353,"cornerstone":23354,"##tized":23355,"##mind":23356,"##iaceae":23357,"careless":23358,"precedence":23359,"frescoes":23360,"##uet":23361,"chilled":23362,"consult":23363,"modelled":23364,"snatch":23365,"peat":23366,"##thermal":23367,"caucasian":23368,"humane":23369,"relaxation":23370,"spins":23371,"temperance":23372,"##lbert":23373,"occupations":23374,"lambda":23375,"hybrids":23376,"moons":23377,"mp3":23378,"##oese":23379,"247":23380,"rolf":23381,"societal":23382,"yerevan":23383,"ness":23384,"##ssler":23385,"befriended":23386,"mechanized":23387,"nominate":23388,"trough":23389,"boasted":23390,"cues":23391,"seater":23392,"##hom":23393,"bends":23394,"##tangle":23395,"conductors":23396,"emptiness":23397,"##lmer":23398,"eurasian":23399,"adriatic":23400,"tian":23401,"##cie":23402,"anxiously":23403,"lark":23404,"propellers":23405,"chichester":23406,"jock":23407,"ev":23408,"2a":23409,"##holding":23410,"credible":23411,"recounts":23412,"tori":23413,"loyalist":23414,"abduction":23415,"##hoot":23416,"##redo":23417,"nepali":23418,"##mite":23419,"ventral":23420,"tempting":23421,"##ango":23422,"##crats":23423,"steered":23424,"##wice":23425,"javelin":23426,"dipping":23427,"laborers":23428,"prentice":23429,"looming":23430,"titanium":23431,"##ː":23432,"badges":23433,"emir":23434,"tensor":23435,"##ntation":23436,"egyptians":23437,"rash":23438,"denies":23439,"hawthorne":23440,"lombard":23441,"showers":23442,"wehrmacht":23443,"dietary":23444,"trojan":23445,"##reus":23446,"welles":23447,"executing":23448,"horseshoe":23449,"lifeboat":23450,"##lak":23451,"elsa":23452,"infirmary":23453,"nearing":23454,"roberta":23455,"boyer":23456,"mutter":23457,"trillion":23458,"joanne":23459,"##fine":23460,"##oked":23461,"sinks":23462,"vortex":23463,"uruguayan":23464,"clasp":23465,"sirius":23466,"##block":23467,"accelerator":23468,"prohibit":23469,"sunken":23470,"byu":23471,"chronological":23472,"diplomats":23473,"ochreous":23474,"510":23475,"symmetrical":23476,"1644":23477,"maia":23478,"##tology":23479,"salts":23480,"reigns":23481,"atrocities":23482,"##ия":23483,"hess":23484,"bared":23485,"issn":23486,"##vyn":23487,"cater":23488,"saturated":23489,"##cycle":23490,"##isse":23491,"sable":23492,"voyager":23493,"dyer":23494,"yusuf":23495,"##inge":23496,"fountains":23497,"wolff":23498,"##39":23499,"##nni":23500,"engraving":23501,"rollins":23502,"atheist":23503,"ominous":23504,"##ault":23505,"herr":23506,"chariot":23507,"martina":23508,"strung":23509,"##fell":23510,"##farlane":23511,"horrific":23512,"sahib":23513,"gazes":23514,"saetan":23515,"erased":23516,"ptolemy":23517,"##olic":23518,"flushing":23519,"lauderdale":23520,"analytic":23521,"##ices":23522,"530":23523,"navarro":23524,"beak":23525,"gorilla":23526,"herrera":23527,"broom":23528,"guadalupe":23529,"raiding":23530,"sykes":23531,"311":23532,"bsc":23533,"deliveries":23534,"1720":23535,"invasions":23536,"carmichael":23537,"tajikistan":23538,"thematic":23539,"ecumenical":23540,"sentiments":23541,"onstage":23542,"##rians":23543,"##brand":23544,"##sume":23545,"catastrophic":23546,"flanks":23547,"molten":23548,"##arns":23549,"waller":23550,"aimee":23551,"terminating":23552,"##icing":23553,"alternately":23554,"##oche":23555,"nehru":23556,"printers":23557,"outraged":23558,"##eving":23559,"empires":23560,"template":23561,"banners":23562,"repetitive":23563,"za":23564,"##oise":23565,"vegetarian":23566,"##tell":23567,"guiana":23568,"opt":23569,"cavendish":23570,"lucknow":23571,"synthesized":23572,"##hani":23573,"##mada":23574,"finalized":23575,"##ctable":23576,"fictitious":23577,"mayoral":23578,"unreliable":23579,"##enham":23580,"embracing":23581,"peppers":23582,"rbis":23583,"##chio":23584,"##neo":23585,"inhibition":23586,"slashed":23587,"togo":23588,"orderly":23589,"embroidered":23590,"safari":23591,"salty":23592,"236":23593,"barron":23594,"benito":23595,"totaled":23596,"##dak":23597,"pubs":23598,"simulated":23599,"caden":23600,"devin":23601,"tolkien":23602,"momma":23603,"welding":23604,"sesame":23605,"##ept":23606,"gottingen":23607,"hardness":23608,"630":23609,"shaman":23610,"temeraire":23611,"620":23612,"adequately":23613,"pediatric":23614,"##kit":23615,"ck":23616,"assertion":23617,"radicals":23618,"composure":23619,"cadence":23620,"seafood":23621,"beaufort":23622,"lazarus":23623,"mani":23624,"warily":23625,"cunning":23626,"kurdistan":23627,"249":23628,"cantata":23629,"##kir":23630,"ares":23631,"##41":23632,"##clusive":23633,"nape":23634,"townland":23635,"geared":23636,"insulted":23637,"flutter":23638,"boating":23639,"violate":23640,"draper":23641,"dumping":23642,"malmo":23643,"##hh":23644,"##romatic":23645,"firearm":23646,"alta":23647,"bono":23648,"obscured":23649,"##clave":23650,"exceeds":23651,"panorama":23652,"unbelievable":23653,"##train":23654,"preschool":23655,"##essed":23656,"disconnected":23657,"installing":23658,"rescuing":23659,"secretaries":23660,"accessibility":23661,"##castle":23662,"##drive":23663,"##ifice":23664,"##film":23665,"bouts":23666,"slug":23667,"waterway":23668,"mindanao":23669,"##buro":23670,"##ratic":23671,"halves":23672,"##ل":23673,"calming":23674,"liter":23675,"maternity":23676,"adorable":23677,"bragg":23678,"electrification":23679,"mcc":23680,"##dote":23681,"roxy":23682,"schizophrenia":23683,"##body":23684,"munoz":23685,"kaye":23686,"whaling":23687,"239":23688,"mil":23689,"tingling":23690,"tolerant":23691,"##ago":23692,"unconventional":23693,"volcanoes":23694,"##finder":23695,"deportivo":23696,"##llie":23697,"robson":23698,"kaufman":23699,"neuroscience":23700,"wai":23701,"deportation":23702,"masovian":23703,"scraping":23704,"converse":23705,"##bh":23706,"hacking":23707,"bulge":23708,"##oun":23709,"administratively":23710,"yao":23711,"580":23712,"amp":23713,"mammoth":23714,"booster":23715,"claremont":23716,"hooper":23717,"nomenclature":23718,"pursuits":23719,"mclaughlin":23720,"melinda":23721,"##sul":23722,"catfish":23723,"barclay":23724,"substrates":23725,"taxa":23726,"zee":23727,"originals":23728,"kimberly":23729,"packets":23730,"padma":23731,"##ality":23732,"borrowing":23733,"ostensibly":23734,"solvent":23735,"##bri":23736,"##genesis":23737,"##mist":23738,"lukas":23739,"shreveport":23740,"veracruz":23741,"##ь":23742,"##lou":23743,"##wives":23744,"cheney":23745,"tt":23746,"anatolia":23747,"hobbs":23748,"##zyn":23749,"cyclic":23750,"radiant":23751,"alistair":23752,"greenish":23753,"siena":23754,"dat":23755,"independents":23756,"##bation":23757,"conform":23758,"pieter":23759,"hyper":23760,"applicant":23761,"bradshaw":23762,"spores":23763,"telangana":23764,"vinci":23765,"inexpensive":23766,"nuclei":23767,"322":23768,"jang":23769,"nme":23770,"soho":23771,"spd":23772,"##ign":23773,"cradled":23774,"receptionist":23775,"pow":23776,"##43":23777,"##rika":23778,"fascism":23779,"##ifer":23780,"experimenting":23781,"##ading":23782,"##iec":23783,"##region":23784,"345":23785,"jocelyn":23786,"maris":23787,"stair":23788,"nocturnal":23789,"toro":23790,"constabulary":23791,"elgin":23792,"##kker":23793,"msc":23794,"##giving":23795,"##schen":23796,"##rase":23797,"doherty":23798,"doping":23799,"sarcastically":23800,"batter":23801,"maneuvers":23802,"##cano":23803,"##apple":23804,"##gai":23805,"##git":23806,"intrinsic":23807,"##nst":23808,"##stor":23809,"1753":23810,"showtime":23811,"cafes":23812,"gasps":23813,"lviv":23814,"ushered":23815,"##thed":23816,"fours":23817,"restart":23818,"astonishment":23819,"transmitting":23820,"flyer":23821,"shrugs":23822,"##sau":23823,"intriguing":23824,"cones":23825,"dictated":23826,"mushrooms":23827,"medial":23828,"##kovsky":23829,"##elman":23830,"escorting":23831,"gaped":23832,"##26":23833,"godfather":23834,"##door":23835,"##sell":23836,"djs":23837,"recaptured":23838,"timetable":23839,"vila":23840,"1710":23841,"3a":23842,"aerodrome":23843,"mortals":23844,"scientology":23845,"##orne":23846,"angelina":23847,"mag":23848,"convection":23849,"unpaid":23850,"insertion":23851,"intermittent":23852,"lego":23853,"##nated":23854,"endeavor":23855,"kota":23856,"pereira":23857,"##lz":23858,"304":23859,"bwv":23860,"glamorgan":23861,"insults":23862,"agatha":23863,"fey":23864,"##cend":23865,"fleetwood":23866,"mahogany":23867,"protruding":23868,"steamship":23869,"zeta":23870,"##arty":23871,"mcguire":23872,"suspense":23873,"##sphere":23874,"advising":23875,"urges":23876,"##wala":23877,"hurriedly":23878,"meteor":23879,"gilded":23880,"inline":23881,"arroyo":23882,"stalker":23883,"##oge":23884,"excitedly":23885,"revered":23886,"##cure":23887,"earle":23888,"introductory":23889,"##break":23890,"##ilde":23891,"mutants":23892,"puff":23893,"pulses":23894,"reinforcement":23895,"##haling":23896,"curses":23897,"lizards":23898,"stalk":23899,"correlated":23900,"##fixed":23901,"fallout":23902,"macquarie":23903,"##unas":23904,"bearded":23905,"denton":23906,"heaving":23907,"802":23908,"##ocation":23909,"winery":23910,"assign":23911,"dortmund":23912,"##lkirk":23913,"everest":23914,"invariant":23915,"charismatic":23916,"susie":23917,"##elling":23918,"bled":23919,"lesley":23920,"telegram":23921,"sumner":23922,"bk":23923,"##ogen":23924,"##к":23925,"wilcox":23926,"needy":23927,"colbert":23928,"duval":23929,"##iferous":23930,"##mbled":23931,"allotted":23932,"attends":23933,"imperative":23934,"##hita":23935,"replacements":23936,"hawker":23937,"##inda":23938,"insurgency":23939,"##zee":23940,"##eke":23941,"casts":23942,"##yla":23943,"680":23944,"ives":23945,"transitioned":23946,"##pack":23947,"##powering":23948,"authoritative":23949,"baylor":23950,"flex":23951,"cringed":23952,"plaintiffs":23953,"woodrow":23954,"##skie":23955,"drastic":23956,"ape":23957,"aroma":23958,"unfolded":23959,"commotion":23960,"nt":23961,"preoccupied":23962,"theta":23963,"routines":23964,"lasers":23965,"privatization":23966,"wand":23967,"domino":23968,"ek":23969,"clenching":23970,"nsa":23971,"strategically":23972,"showered":23973,"bile":23974,"handkerchief":23975,"pere":23976,"storing":23977,"christophe":23978,"insulting":23979,"316":23980,"nakamura":23981,"romani":23982,"asiatic":23983,"magdalena":23984,"palma":23985,"cruises":23986,"stripping":23987,"405":23988,"konstantin":23989,"soaring":23990,"##berman":23991,"colloquially":23992,"forerunner":23993,"havilland":23994,"incarcerated":23995,"parasites":23996,"sincerity":23997,"##utus":23998,"disks":23999,"plank":24000,"saigon":24001,"##ining":24002,"corbin":24003,"homo":24004,"ornaments":24005,"powerhouse":24006,"##tlement":24007,"chong":24008,"fastened":24009,"feasibility":24010,"idf":24011,"morphological":24012,"usable":24013,"##nish":24014,"##zuki":24015,"aqueduct":24016,"jaguars":24017,"keepers":24018,"##flies":24019,"aleksandr":24020,"faust":24021,"assigns":24022,"ewing":24023,"bacterium":24024,"hurled":24025,"tricky":24026,"hungarians":24027,"integers":24028,"wallis":24029,"321":24030,"yamaha":24031,"##isha":24032,"hushed":24033,"oblivion":24034,"aviator":24035,"evangelist":24036,"friars":24037,"##eller":24038,"monograph":24039,"ode":24040,"##nary":24041,"airplanes":24042,"labourers":24043,"charms":24044,"##nee":24045,"1661":24046,"hagen":24047,"tnt":24048,"rudder":24049,"fiesta":24050,"transcript":24051,"dorothea":24052,"ska":24053,"inhibitor":24054,"maccabi":24055,"retorted":24056,"raining":24057,"encompassed":24058,"clauses":24059,"menacing":24060,"1642":24061,"lineman":24062,"##gist":24063,"vamps":24064,"##ape":24065,"##dick":24066,"gloom":24067,"##rera":24068,"dealings":24069,"easing":24070,"seekers":24071,"##nut":24072,"##pment":24073,"helens":24074,"unmanned":24075,"##anu":24076,"##isson":24077,"basics":24078,"##amy":24079,"##ckman":24080,"adjustments":24081,"1688":24082,"brutality":24083,"horne":24084,"##zell":24085,"sui":24086,"##55":24087,"##mable":24088,"aggregator":24089,"##thal":24090,"rhino":24091,"##drick":24092,"##vira":24093,"counters":24094,"zoom":24095,"##01":24096,"##rting":24097,"mn":24098,"montenegrin":24099,"packard":24100,"##unciation":24101,"##♭":24102,"##kki":24103,"reclaim":24104,"scholastic":24105,"thugs":24106,"pulsed":24107,"##icia":24108,"syriac":24109,"quan":24110,"saddam":24111,"banda":24112,"kobe":24113,"blaming":24114,"buddies":24115,"dissent":24116,"##lusion":24117,"##usia":24118,"corbett":24119,"jaya":24120,"delle":24121,"erratic":24122,"lexie":24123,"##hesis":24124,"435":24125,"amiga":24126,"hermes":24127,"##pressing":24128,"##leen":24129,"chapels":24130,"gospels":24131,"jamal":24132,"##uating":24133,"compute":24134,"revolving":24135,"warp":24136,"##sso":24137,"##thes":24138,"armory":24139,"##eras":24140,"##gol":24141,"antrim":24142,"loki":24143,"##kow":24144,"##asian":24145,"##good":24146,"##zano":24147,"braid":24148,"handwriting":24149,"subdistrict":24150,"funky":24151,"pantheon":24152,"##iculate":24153,"concurrency":24154,"estimation":24155,"improper":24156,"juliana":24157,"##his":24158,"newcomers":24159,"johnstone":24160,"staten":24161,"communicated":24162,"##oco":24163,"##alle":24164,"sausage":24165,"stormy":24166,"##stered":24167,"##tters":24168,"superfamily":24169,"##grade":24170,"acidic":24171,"collateral":24172,"tabloid":24173,"##oped":24174,"##rza":24175,"bladder":24176,"austen":24177,"##ellant":24178,"mcgraw":24179,"##hay":24180,"hannibal":24181,"mein":24182,"aquino":24183,"lucifer":24184,"wo":24185,"badger":24186,"boar":24187,"cher":24188,"christensen":24189,"greenberg":24190,"interruption":24191,"##kken":24192,"jem":24193,"244":24194,"mocked":24195,"bottoms":24196,"cambridgeshire":24197,"##lide":24198,"sprawling":24199,"##bbly":24200,"eastwood":24201,"ghent":24202,"synth":24203,"##buck":24204,"advisers":24205,"##bah":24206,"nominally":24207,"hapoel":24208,"qu":24209,"daggers":24210,"estranged":24211,"fabricated":24212,"towels":24213,"vinnie":24214,"wcw":24215,"misunderstanding":24216,"anglia":24217,"nothin":24218,"unmistakable":24219,"##dust":24220,"##lova":24221,"chilly":24222,"marquette":24223,"truss":24224,"##edge":24225,"##erine":24226,"reece":24227,"##lty":24228,"##chemist":24229,"##connected":24230,"272":24231,"308":24232,"41st":24233,"bash":24234,"raion":24235,"waterfalls":24236,"##ump":24237,"##main":24238,"labyrinth":24239,"queue":24240,"theorist":24241,"##istle":24242,"bharatiya":24243,"flexed":24244,"soundtracks":24245,"rooney":24246,"leftist":24247,"patrolling":24248,"wharton":24249,"plainly":24250,"alleviate":24251,"eastman":24252,"schuster":24253,"topographic":24254,"engages":24255,"immensely":24256,"unbearable":24257,"fairchild":24258,"1620":24259,"dona":24260,"lurking":24261,"parisian":24262,"oliveira":24263,"ia":24264,"indictment":24265,"hahn":24266,"bangladeshi":24267,"##aster":24268,"vivo":24269,"##uming":24270,"##ential":24271,"antonia":24272,"expects":24273,"indoors":24274,"kildare":24275,"harlan":24276,"##logue":24277,"##ogenic":24278,"##sities":24279,"forgiven":24280,"##wat":24281,"childish":24282,"tavi":24283,"##mide":24284,"##orra":24285,"plausible":24286,"grimm":24287,"successively":24288,"scooted":24289,"##bola":24290,"##dget":24291,"##rith":24292,"spartans":24293,"emery":24294,"flatly":24295,"azure":24296,"epilogue":24297,"##wark":24298,"flourish":24299,"##iny":24300,"##tracted":24301,"##overs":24302,"##oshi":24303,"bestseller":24304,"distressed":24305,"receipt":24306,"spitting":24307,"hermit":24308,"topological":24309,"##cot":24310,"drilled":24311,"subunit":24312,"francs":24313,"##layer":24314,"eel":24315,"##fk":24316,"##itas":24317,"octopus":24318,"footprint":24319,"petitions":24320,"ufo":24321,"##say":24322,"##foil":24323,"interfering":24324,"leaking":24325,"palo":24326,"##metry":24327,"thistle":24328,"valiant":24329,"##pic":24330,"narayan":24331,"mcpherson":24332,"##fast":24333,"gonzales":24334,"##ym":24335,"##enne":24336,"dustin":24337,"novgorod":24338,"solos":24339,"##zman":24340,"doin":24341,"##raph":24342,"##patient":24343,"##meyer":24344,"soluble":24345,"ashland":24346,"cuffs":24347,"carole":24348,"pendleton":24349,"whistling":24350,"vassal":24351,"##river":24352,"deviation":24353,"revisited":24354,"constituents":24355,"rallied":24356,"rotate":24357,"loomed":24358,"##eil":24359,"##nting":24360,"amateurs":24361,"augsburg":24362,"auschwitz":24363,"crowns":24364,"skeletons":24365,"##cona":24366,"bonnet":24367,"257":24368,"dummy":24369,"globalization":24370,"simeon":24371,"sleeper":24372,"mandal":24373,"differentiated":24374,"##crow":24375,"##mare":24376,"milne":24377,"bundled":24378,"exasperated":24379,"talmud":24380,"owes":24381,"segregated":24382,"##feng":24383,"##uary":24384,"dentist":24385,"piracy":24386,"props":24387,"##rang":24388,"devlin":24389,"##torium":24390,"malicious":24391,"paws":24392,"##laid":24393,"dependency":24394,"##ergy":24395,"##fers":24396,"##enna":24397,"258":24398,"pistons":24399,"rourke":24400,"jed":24401,"grammatical":24402,"tres":24403,"maha":24404,"wig":24405,"512":24406,"ghostly":24407,"jayne":24408,"##achal":24409,"##creen":24410,"##ilis":24411,"##lins":24412,"##rence":24413,"designate":24414,"##with":24415,"arrogance":24416,"cambodian":24417,"clones":24418,"showdown":24419,"throttle":24420,"twain":24421,"##ception":24422,"lobes":24423,"metz":24424,"nagoya":24425,"335":24426,"braking":24427,"##furt":24428,"385":24429,"roaming":24430,"##minster":24431,"amin":24432,"crippled":24433,"##37":24434,"##llary":24435,"indifferent":24436,"hoffmann":24437,"idols":24438,"intimidating":24439,"1751":24440,"261":24441,"influenza":24442,"memo":24443,"onions":24444,"1748":24445,"bandage":24446,"consciously":24447,"##landa":24448,"##rage":24449,"clandestine":24450,"observes":24451,"swiped":24452,"tangle":24453,"##ener":24454,"##jected":24455,"##trum":24456,"##bill":24457,"##lta":24458,"hugs":24459,"congresses":24460,"josiah":24461,"spirited":24462,"##dek":24463,"humanist":24464,"managerial":24465,"filmmaking":24466,"inmate":24467,"rhymes":24468,"debuting":24469,"grimsby":24470,"ur":24471,"##laze":24472,"duplicate":24473,"vigor":24474,"##tf":24475,"republished":24476,"bolshevik":24477,"refurbishment":24478,"antibiotics":24479,"martini":24480,"methane":24481,"newscasts":24482,"royale":24483,"horizons":24484,"levant":24485,"iain":24486,"visas":24487,"##ischen":24488,"paler":24489,"##around":24490,"manifestation":24491,"snuck":24492,"alf":24493,"chop":24494,"futile":24495,"pedestal":24496,"rehab":24497,"##kat":24498,"bmg":24499,"kerman":24500,"res":24501,"fairbanks":24502,"jarrett":24503,"abstraction":24504,"saharan":24505,"##zek":24506,"1746":24507,"procedural":24508,"clearer":24509,"kincaid":24510,"sash":24511,"luciano":24512,"##ffey":24513,"crunch":24514,"helmut":24515,"##vara":24516,"revolutionaries":24517,"##tute":24518,"creamy":24519,"leach":24520,"##mmon":24521,"1747":24522,"permitting":24523,"nes":24524,"plight":24525,"wendell":24526,"##lese":24527,"contra":24528,"ts":24529,"clancy":24530,"ipa":24531,"mach":24532,"staples":24533,"autopsy":24534,"disturbances":24535,"nueva":24536,"karin":24537,"pontiac":24538,"##uding":24539,"proxy":24540,"venerable":24541,"haunt":24542,"leto":24543,"bergman":24544,"expands":24545,"##helm":24546,"wal":24547,"##pipe":24548,"canning":24549,"celine":24550,"cords":24551,"obesity":24552,"##enary":24553,"intrusion":24554,"planner":24555,"##phate":24556,"reasoned":24557,"sequencing":24558,"307":24559,"harrow":24560,"##chon":24561,"##dora":24562,"marred":24563,"mcintyre":24564,"repay":24565,"tarzan":24566,"darting":24567,"248":24568,"harrisburg":24569,"margarita":24570,"repulsed":24571,"##hur":24572,"##lding":24573,"belinda":24574,"hamburger":24575,"novo":24576,"compliant":24577,"runways":24578,"bingham":24579,"registrar":24580,"skyscraper":24581,"ic":24582,"cuthbert":24583,"improvisation":24584,"livelihood":24585,"##corp":24586,"##elial":24587,"admiring":24588,"##dened":24589,"sporadic":24590,"believer":24591,"casablanca":24592,"popcorn":24593,"##29":24594,"asha":24595,"shovel":24596,"##bek":24597,"##dice":24598,"coiled":24599,"tangible":24600,"##dez":24601,"casper":24602,"elsie":24603,"resin":24604,"tenderness":24605,"rectory":24606,"##ivision":24607,"avail":24608,"sonar":24609,"##mori":24610,"boutique":24611,"##dier":24612,"guerre":24613,"bathed":24614,"upbringing":24615,"vaulted":24616,"sandals":24617,"blessings":24618,"##naut":24619,"##utnant":24620,"1680":24621,"306":24622,"foxes":24623,"pia":24624,"corrosion":24625,"hesitantly":24626,"confederates":24627,"crystalline":24628,"footprints":24629,"shapiro":24630,"tirana":24631,"valentin":24632,"drones":24633,"45th":24634,"microscope":24635,"shipments":24636,"texted":24637,"inquisition":24638,"wry":24639,"guernsey":24640,"unauthorized":24641,"resigning":24642,"760":24643,"ripple":24644,"schubert":24645,"stu":24646,"reassure":24647,"felony":24648,"##ardo":24649,"brittle":24650,"koreans":24651,"##havan":24652,"##ives":24653,"dun":24654,"implicit":24655,"tyres":24656,"##aldi":24657,"##lth":24658,"magnolia":24659,"##ehan":24660,"##puri":24661,"##poulos":24662,"aggressively":24663,"fei":24664,"gr":24665,"familiarity":24666,"##poo":24667,"indicative":24668,"##trust":24669,"fundamentally":24670,"jimmie":24671,"overrun":24672,"395":24673,"anchors":24674,"moans":24675,"##opus":24676,"britannia":24677,"armagh":24678,"##ggle":24679,"purposely":24680,"seizing":24681,"##vao":24682,"bewildered":24683,"mundane":24684,"avoidance":24685,"cosmopolitan":24686,"geometridae":24687,"quartermaster":24688,"caf":24689,"415":24690,"chatter":24691,"engulfed":24692,"gleam":24693,"purge":24694,"##icate":24695,"juliette":24696,"jurisprudence":24697,"guerra":24698,"revisions":24699,"##bn":24700,"casimir":24701,"brew":24702,"##jm":24703,"1749":24704,"clapton":24705,"cloudy":24706,"conde":24707,"hermitage":24708,"278":24709,"simulations":24710,"torches":24711,"vincenzo":24712,"matteo":24713,"##rill":24714,"hidalgo":24715,"booming":24716,"westbound":24717,"accomplishment":24718,"tentacles":24719,"unaffected":24720,"##sius":24721,"annabelle":24722,"flopped":24723,"sloping":24724,"##litz":24725,"dreamer":24726,"interceptor":24727,"vu":24728,"##loh":24729,"consecration":24730,"copying":24731,"messaging":24732,"breaker":24733,"climates":24734,"hospitalized":24735,"1752":24736,"torino":24737,"afternoons":24738,"winfield":24739,"witnessing":24740,"##teacher":24741,"breakers":24742,"choirs":24743,"sawmill":24744,"coldly":24745,"##ege":24746,"sipping":24747,"haste":24748,"uninhabited":24749,"conical":24750,"bibliography":24751,"pamphlets":24752,"severn":24753,"edict":24754,"##oca":24755,"deux":24756,"illnesses":24757,"grips":24758,"##pl":24759,"rehearsals":24760,"sis":24761,"thinkers":24762,"tame":24763,"##keepers":24764,"1690":24765,"acacia":24766,"reformer":24767,"##osed":24768,"##rys":24769,"shuffling":24770,"##iring":24771,"##shima":24772,"eastbound":24773,"ionic":24774,"rhea":24775,"flees":24776,"littered":24777,"##oum":24778,"rocker":24779,"vomiting":24780,"groaning":24781,"champ":24782,"overwhelmingly":24783,"civilizations":24784,"paces":24785,"sloop":24786,"adoptive":24787,"##tish":24788,"skaters":24789,"##vres":24790,"aiding":24791,"mango":24792,"##joy":24793,"nikola":24794,"shriek":24795,"##ignon":24796,"pharmaceuticals":24797,"##mg":24798,"tuna":24799,"calvert":24800,"gustavo":24801,"stocked":24802,"yearbook":24803,"##urai":24804,"##mana":24805,"computed":24806,"subsp":24807,"riff":24808,"hanoi":24809,"kelvin":24810,"hamid":24811,"moors":24812,"pastures":24813,"summons":24814,"jihad":24815,"nectar":24816,"##ctors":24817,"bayou":24818,"untitled":24819,"pleasing":24820,"vastly":24821,"republics":24822,"intellect":24823,"##η":24824,"##ulio":24825,"##tou":24826,"crumbling":24827,"stylistic":24828,"sb":24829,"##ی":24830,"consolation":24831,"frequented":24832,"h₂o":24833,"walden":24834,"widows":24835,"##iens":24836,"404":24837,"##ignment":24838,"chunks":24839,"improves":24840,"288":24841,"grit":24842,"recited":24843,"##dev":24844,"snarl":24845,"sociological":24846,"##arte":24847,"##gul":24848,"inquired":24849,"##held":24850,"bruise":24851,"clube":24852,"consultancy":24853,"homogeneous":24854,"hornets":24855,"multiplication":24856,"pasta":24857,"prick":24858,"savior":24859,"##grin":24860,"##kou":24861,"##phile":24862,"yoon":24863,"##gara":24864,"grimes":24865,"vanishing":24866,"cheering":24867,"reacting":24868,"bn":24869,"distillery":24870,"##quisite":24871,"##vity":24872,"coe":24873,"dockyard":24874,"massif":24875,"##jord":24876,"escorts":24877,"voss":24878,"##valent":24879,"byte":24880,"chopped":24881,"hawke":24882,"illusions":24883,"workings":24884,"floats":24885,"##koto":24886,"##vac":24887,"kv":24888,"annapolis":24889,"madden":24890,"##onus":24891,"alvaro":24892,"noctuidae":24893,"##cum":24894,"##scopic":24895,"avenge":24896,"steamboat":24897,"forte":24898,"illustrates":24899,"erika":24900,"##trip":24901,"570":24902,"dew":24903,"nationalities":24904,"bran":24905,"manifested":24906,"thirsty":24907,"diversified":24908,"muscled":24909,"reborn":24910,"##standing":24911,"arson":24912,"##lessness":24913,"##dran":24914,"##logram":24915,"##boys":24916,"##kushima":24917,"##vious":24918,"willoughby":24919,"##phobia":24920,"286":24921,"alsace":24922,"dashboard":24923,"yuki":24924,"##chai":24925,"granville":24926,"myspace":24927,"publicized":24928,"tricked":24929,"##gang":24930,"adjective":24931,"##ater":24932,"relic":24933,"reorganisation":24934,"enthusiastically":24935,"indications":24936,"saxe":24937,"##lassified":24938,"consolidate":24939,"iec":24940,"padua":24941,"helplessly":24942,"ramps":24943,"renaming":24944,"regulars":24945,"pedestrians":24946,"accents":24947,"convicts":24948,"inaccurate":24949,"lowers":24950,"mana":24951,"##pati":24952,"barrie":24953,"bjp":24954,"outta":24955,"someplace":24956,"berwick":24957,"flanking":24958,"invoked":24959,"marrow":24960,"sparsely":24961,"excerpts":24962,"clothed":24963,"rei":24964,"##ginal":24965,"wept":24966,"##straße":24967,"##vish":24968,"alexa":24969,"excel":24970,"##ptive":24971,"membranes":24972,"aquitaine":24973,"creeks":24974,"cutler":24975,"sheppard":24976,"implementations":24977,"ns":24978,"##dur":24979,"fragrance":24980,"budge":24981,"concordia":24982,"magnesium":24983,"marcelo":24984,"##antes":24985,"gladly":24986,"vibrating":24987,"##rral":24988,"##ggles":24989,"montrose":24990,"##omba":24991,"lew":24992,"seamus":24993,"1630":24994,"cocky":24995,"##ament":24996,"##uen":24997,"bjorn":24998,"##rrick":24999,"fielder":25000,"fluttering":25001,"##lase":25002,"methyl":25003,"kimberley":25004,"mcdowell":25005,"reductions":25006,"barbed":25007,"##jic":25008,"##tonic":25009,"aeronautical":25010,"condensed":25011,"distracting":25012,"##promising":25013,"huffed":25014,"##cala":25015,"##sle":25016,"claudius":25017,"invincible":25018,"missy":25019,"pious":25020,"balthazar":25021,"ci":25022,"##lang":25023,"butte":25024,"combo":25025,"orson":25026,"##dication":25027,"myriad":25028,"1707":25029,"silenced":25030,"##fed":25031,"##rh":25032,"coco":25033,"netball":25034,"yourselves":25035,"##oza":25036,"clarify":25037,"heller":25038,"peg":25039,"durban":25040,"etudes":25041,"offender":25042,"roast":25043,"blackmail":25044,"curvature":25045,"##woods":25046,"vile":25047,"309":25048,"illicit":25049,"suriname":25050,"##linson":25051,"overture":25052,"1685":25053,"bubbling":25054,"gymnast":25055,"tucking":25056,"##mming":25057,"##ouin":25058,"maldives":25059,"##bala":25060,"gurney":25061,"##dda":25062,"##eased":25063,"##oides":25064,"backside":25065,"pinto":25066,"jars":25067,"racehorse":25068,"tending":25069,"##rdial":25070,"baronetcy":25071,"wiener":25072,"duly":25073,"##rke":25074,"barbarian":25075,"cupping":25076,"flawed":25077,"##thesis":25078,"bertha":25079,"pleistocene":25080,"puddle":25081,"swearing":25082,"##nob":25083,"##tically":25084,"fleeting":25085,"prostate":25086,"amulet":25087,"educating":25088,"##mined":25089,"##iti":25090,"##tler":25091,"75th":25092,"jens":25093,"respondents":25094,"analytics":25095,"cavaliers":25096,"papacy":25097,"raju":25098,"##iente":25099,"##ulum":25100,"##tip":25101,"funnel":25102,"271":25103,"disneyland":25104,"##lley":25105,"sociologist":25106,"##iam":25107,"2500":25108,"faulkner":25109,"louvre":25110,"menon":25111,"##dson":25112,"276":25113,"##ower":25114,"afterlife":25115,"mannheim":25116,"peptide":25117,"referees":25118,"comedians":25119,"meaningless":25120,"##anger":25121,"##laise":25122,"fabrics":25123,"hurley":25124,"renal":25125,"sleeps":25126,"##bour":25127,"##icle":25128,"breakout":25129,"kristin":25130,"roadside":25131,"animator":25132,"clover":25133,"disdain":25134,"unsafe":25135,"redesign":25136,"##urity":25137,"firth":25138,"barnsley":25139,"portage":25140,"reset":25141,"narrows":25142,"268":25143,"commandos":25144,"expansive":25145,"speechless":25146,"tubular":25147,"##lux":25148,"essendon":25149,"eyelashes":25150,"smashwords":25151,"##yad":25152,"##bang":25153,"##claim":25154,"craved":25155,"sprinted":25156,"chet":25157,"somme":25158,"astor":25159,"wrocław":25160,"orton":25161,"266":25162,"bane":25163,"##erving":25164,"##uing":25165,"mischief":25166,"##amps":25167,"##sund":25168,"scaling":25169,"terre":25170,"##xious":25171,"impairment":25172,"offenses":25173,"undermine":25174,"moi":25175,"soy":25176,"contiguous":25177,"arcadia":25178,"inuit":25179,"seam":25180,"##tops":25181,"macbeth":25182,"rebelled":25183,"##icative":25184,"##iot":25185,"590":25186,"elaborated":25187,"frs":25188,"uniformed":25189,"##dberg":25190,"259":25191,"powerless":25192,"priscilla":25193,"stimulated":25194,"980":25195,"qc":25196,"arboretum":25197,"frustrating":25198,"trieste":25199,"bullock":25200,"##nified":25201,"enriched":25202,"glistening":25203,"intern":25204,"##adia":25205,"locus":25206,"nouvelle":25207,"ollie":25208,"ike":25209,"lash":25210,"starboard":25211,"ee":25212,"tapestry":25213,"headlined":25214,"hove":25215,"rigged":25216,"##vite":25217,"pollock":25218,"##yme":25219,"thrive":25220,"clustered":25221,"cas":25222,"roi":25223,"gleamed":25224,"olympiad":25225,"##lino":25226,"pressured":25227,"regimes":25228,"##hosis":25229,"##lick":25230,"ripley":25231,"##ophone":25232,"kickoff":25233,"gallon":25234,"rockwell":25235,"##arable":25236,"crusader":25237,"glue":25238,"revolutions":25239,"scrambling":25240,"1714":25241,"grover":25242,"##jure":25243,"englishman":25244,"aztec":25245,"263":25246,"contemplating":25247,"coven":25248,"ipad":25249,"preach":25250,"triumphant":25251,"tufts":25252,"##esian":25253,"rotational":25254,"##phus":25255,"328":25256,"falkland":25257,"##brates":25258,"strewn":25259,"clarissa":25260,"rejoin":25261,"environmentally":25262,"glint":25263,"banded":25264,"drenched":25265,"moat":25266,"albanians":25267,"johor":25268,"rr":25269,"maestro":25270,"malley":25271,"nouveau":25272,"shaded":25273,"taxonomy":25274,"v6":25275,"adhere":25276,"bunk":25277,"airfields":25278,"##ritan":25279,"1741":25280,"encompass":25281,"remington":25282,"tran":25283,"##erative":25284,"amelie":25285,"mazda":25286,"friar":25287,"morals":25288,"passions":25289,"##zai":25290,"breadth":25291,"vis":25292,"##hae":25293,"argus":25294,"burnham":25295,"caressing":25296,"insider":25297,"rudd":25298,"##imov":25299,"##mini":25300,"##rso":25301,"italianate":25302,"murderous":25303,"textual":25304,"wainwright":25305,"armada":25306,"bam":25307,"weave":25308,"timer":25309,"##taken":25310,"##nh":25311,"fra":25312,"##crest":25313,"ardent":25314,"salazar":25315,"taps":25316,"tunis":25317,"##ntino":25318,"allegro":25319,"gland":25320,"philanthropic":25321,"##chester":25322,"implication":25323,"##optera":25324,"esq":25325,"judas":25326,"noticeably":25327,"wynn":25328,"##dara":25329,"inched":25330,"indexed":25331,"crises":25332,"villiers":25333,"bandit":25334,"royalties":25335,"patterned":25336,"cupboard":25337,"interspersed":25338,"accessory":25339,"isla":25340,"kendrick":25341,"entourage":25342,"stitches":25343,"##esthesia":25344,"headwaters":25345,"##ior":25346,"interlude":25347,"distraught":25348,"draught":25349,"1727":25350,"##basket":25351,"biased":25352,"sy":25353,"transient":25354,"triad":25355,"subgenus":25356,"adapting":25357,"kidd":25358,"shortstop":25359,"##umatic":25360,"dimly":25361,"spiked":25362,"mcleod":25363,"reprint":25364,"nellie":25365,"pretoria":25366,"windmill":25367,"##cek":25368,"singled":25369,"##mps":25370,"273":25371,"reunite":25372,"##orous":25373,"747":25374,"bankers":25375,"outlying":25376,"##omp":25377,"##ports":25378,"##tream":25379,"apologies":25380,"cosmetics":25381,"patsy":25382,"##deh":25383,"##ocks":25384,"##yson":25385,"bender":25386,"nantes":25387,"serene":25388,"##nad":25389,"lucha":25390,"mmm":25391,"323":25392,"##cius":25393,"##gli":25394,"cmll":25395,"coinage":25396,"nestor":25397,"juarez":25398,"##rook":25399,"smeared":25400,"sprayed":25401,"twitching":25402,"sterile":25403,"irina":25404,"embodied":25405,"juveniles":25406,"enveloped":25407,"miscellaneous":25408,"cancers":25409,"dq":25410,"gulped":25411,"luisa":25412,"crested":25413,"swat":25414,"donegal":25415,"ref":25416,"##anov":25417,"##acker":25418,"hearst":25419,"mercantile":25420,"##lika":25421,"doorbell":25422,"ua":25423,"vicki":25424,"##alla":25425,"##som":25426,"bilbao":25427,"psychologists":25428,"stryker":25429,"sw":25430,"horsemen":25431,"turkmenistan":25432,"wits":25433,"##national":25434,"anson":25435,"mathew":25436,"screenings":25437,"##umb":25438,"rihanna":25439,"##agne":25440,"##nessy":25441,"aisles":25442,"##iani":25443,"##osphere":25444,"hines":25445,"kenton":25446,"saskatoon":25447,"tasha":25448,"truncated":25449,"##champ":25450,"##itan":25451,"mildred":25452,"advises":25453,"fredrik":25454,"interpreting":25455,"inhibitors":25456,"##athi":25457,"spectroscopy":25458,"##hab":25459,"##kong":25460,"karim":25461,"panda":25462,"##oia":25463,"##nail":25464,"##vc":25465,"conqueror":25466,"kgb":25467,"leukemia":25468,"##dity":25469,"arrivals":25470,"cheered":25471,"pisa":25472,"phosphorus":25473,"shielded":25474,"##riated":25475,"mammal":25476,"unitarian":25477,"urgently":25478,"chopin":25479,"sanitary":25480,"##mission":25481,"spicy":25482,"drugged":25483,"hinges":25484,"##tort":25485,"tipping":25486,"trier":25487,"impoverished":25488,"westchester":25489,"##caster":25490,"267":25491,"epoch":25492,"nonstop":25493,"##gman":25494,"##khov":25495,"aromatic":25496,"centrally":25497,"cerro":25498,"##tively":25499,"##vio":25500,"billions":25501,"modulation":25502,"sedimentary":25503,"283":25504,"facilitating":25505,"outrageous":25506,"goldstein":25507,"##eak":25508,"##kt":25509,"ld":25510,"maitland":25511,"penultimate":25512,"pollard":25513,"##dance":25514,"fleets":25515,"spaceship":25516,"vertebrae":25517,"##nig":25518,"alcoholism":25519,"als":25520,"recital":25521,"##bham":25522,"##ference":25523,"##omics":25524,"m2":25525,"##bm":25526,"trois":25527,"##tropical":25528,"##в":25529,"commemorates":25530,"##meric":25531,"marge":25532,"##raction":25533,"1643":25534,"670":25535,"cosmetic":25536,"ravaged":25537,"##ige":25538,"catastrophe":25539,"eng":25540,"##shida":25541,"albrecht":25542,"arterial":25543,"bellamy":25544,"decor":25545,"harmon":25546,"##rde":25547,"bulbs":25548,"synchronized":25549,"vito":25550,"easiest":25551,"shetland":25552,"shielding":25553,"wnba":25554,"##glers":25555,"##ssar":25556,"##riam":25557,"brianna":25558,"cumbria":25559,"##aceous":25560,"##rard":25561,"cores":25562,"thayer":25563,"##nsk":25564,"brood":25565,"hilltop":25566,"luminous":25567,"carts":25568,"keynote":25569,"larkin":25570,"logos":25571,"##cta":25572,"##ا":25573,"##mund":25574,"##quay":25575,"lilith":25576,"tinted":25577,"277":25578,"wrestle":25579,"mobilization":25580,"##uses":25581,"sequential":25582,"siam":25583,"bloomfield":25584,"takahashi":25585,"274":25586,"##ieving":25587,"presenters":25588,"ringo":25589,"blazed":25590,"witty":25591,"##oven":25592,"##ignant":25593,"devastation":25594,"haydn":25595,"harmed":25596,"newt":25597,"therese":25598,"##peed":25599,"gershwin":25600,"molina":25601,"rabbis":25602,"sudanese":25603,"001":25604,"innate":25605,"restarted":25606,"##sack":25607,"##fus":25608,"slices":25609,"wb":25610,"##shah":25611,"enroll":25612,"hypothetical":25613,"hysterical":25614,"1743":25615,"fabio":25616,"indefinite":25617,"warped":25618,"##hg":25619,"exchanging":25620,"525":25621,"unsuitable":25622,"##sboro":25623,"gallo":25624,"1603":25625,"bret":25626,"cobalt":25627,"homemade":25628,"##hunter":25629,"mx":25630,"operatives":25631,"##dhar":25632,"terraces":25633,"durable":25634,"latch":25635,"pens":25636,"whorls":25637,"##ctuated":25638,"##eaux":25639,"billing":25640,"ligament":25641,"succumbed":25642,"##gly":25643,"regulators":25644,"spawn":25645,"##brick":25646,"##stead":25647,"filmfare":25648,"rochelle":25649,"##nzo":25650,"1725":25651,"circumstance":25652,"saber":25653,"supplements":25654,"##nsky":25655,"##tson":25656,"crowe":25657,"wellesley":25658,"carrot":25659,"##9th":25660,"##movable":25661,"primate":25662,"drury":25663,"sincerely":25664,"topical":25665,"##mad":25666,"##rao":25667,"callahan":25668,"kyiv":25669,"smarter":25670,"tits":25671,"undo":25672,"##yeh":25673,"announcements":25674,"anthologies":25675,"barrio":25676,"nebula":25677,"##islaus":25678,"##shaft":25679,"##tyn":25680,"bodyguards":25681,"2021":25682,"assassinate":25683,"barns":25684,"emmett":25685,"scully":25686,"##mah":25687,"##yd":25688,"##eland":25689,"##tino":25690,"##itarian":25691,"demoted":25692,"gorman":25693,"lashed":25694,"prized":25695,"adventist":25696,"writ":25697,"##gui":25698,"alla":25699,"invertebrates":25700,"##ausen":25701,"1641":25702,"amman":25703,"1742":25704,"align":25705,"healy":25706,"redistribution":25707,"##gf":25708,"##rize":25709,"insulation":25710,"##drop":25711,"adherents":25712,"hezbollah":25713,"vitro":25714,"ferns":25715,"yanking":25716,"269":25717,"php":25718,"registering":25719,"uppsala":25720,"cheerleading":25721,"confines":25722,"mischievous":25723,"tully":25724,"##ross":25725,"49th":25726,"docked":25727,"roam":25728,"stipulated":25729,"pumpkin":25730,"##bry":25731,"prompt":25732,"##ezer":25733,"blindly":25734,"shuddering":25735,"craftsmen":25736,"frail":25737,"scented":25738,"katharine":25739,"scramble":25740,"shaggy":25741,"sponge":25742,"helix":25743,"zaragoza":25744,"279":25745,"##52":25746,"43rd":25747,"backlash":25748,"fontaine":25749,"seizures":25750,"posse":25751,"cowan":25752,"nonfiction":25753,"telenovela":25754,"wwii":25755,"hammered":25756,"undone":25757,"##gpur":25758,"encircled":25759,"irs":25760,"##ivation":25761,"artefacts":25762,"oneself":25763,"searing":25764,"smallpox":25765,"##belle":25766,"##osaurus":25767,"shandong":25768,"breached":25769,"upland":25770,"blushing":25771,"rankin":25772,"infinitely":25773,"psyche":25774,"tolerated":25775,"docking":25776,"evicted":25777,"##col":25778,"unmarked":25779,"##lving":25780,"gnome":25781,"lettering":25782,"litres":25783,"musique":25784,"##oint":25785,"benevolent":25786,"##jal":25787,"blackened":25788,"##anna":25789,"mccall":25790,"racers":25791,"tingle":25792,"##ocene":25793,"##orestation":25794,"introductions":25795,"radically":25796,"292":25797,"##hiff":25798,"##باد":25799,"1610":25800,"1739":25801,"munchen":25802,"plead":25803,"##nka":25804,"condo":25805,"scissors":25806,"##sight":25807,"##tens":25808,"apprehension":25809,"##cey":25810,"##yin":25811,"hallmark":25812,"watering":25813,"formulas":25814,"sequels":25815,"##llas":25816,"aggravated":25817,"bae":25818,"commencing":25819,"##building":25820,"enfield":25821,"prohibits":25822,"marne":25823,"vedic":25824,"civilized":25825,"euclidean":25826,"jagger":25827,"beforehand":25828,"blasts":25829,"dumont":25830,"##arney":25831,"##nem":25832,"740":25833,"conversions":25834,"hierarchical":25835,"rios":25836,"simulator":25837,"##dya":25838,"##lellan":25839,"hedges":25840,"oleg":25841,"thrusts":25842,"shadowed":25843,"darby":25844,"maximize":25845,"1744":25846,"gregorian":25847,"##nded":25848,"##routed":25849,"sham":25850,"unspecified":25851,"##hog":25852,"emory":25853,"factual":25854,"##smo":25855,"##tp":25856,"fooled":25857,"##rger":25858,"ortega":25859,"wellness":25860,"marlon":25861,"##oton":25862,"##urance":25863,"casket":25864,"keating":25865,"ley":25866,"enclave":25867,"##ayan":25868,"char":25869,"influencing":25870,"jia":25871,"##chenko":25872,"412":25873,"ammonia":25874,"erebidae":25875,"incompatible":25876,"violins":25877,"cornered":25878,"##arat":25879,"grooves":25880,"astronauts":25881,"columbian":25882,"rampant":25883,"fabrication":25884,"kyushu":25885,"mahmud":25886,"vanish":25887,"##dern":25888,"mesopotamia":25889,"##lete":25890,"ict":25891,"##rgen":25892,"caspian":25893,"kenji":25894,"pitted":25895,"##vered":25896,"999":25897,"grimace":25898,"roanoke":25899,"tchaikovsky":25900,"twinned":25901,"##analysis":25902,"##awan":25903,"xinjiang":25904,"arias":25905,"clemson":25906,"kazakh":25907,"sizable":25908,"1662":25909,"##khand":25910,"##vard":25911,"plunge":25912,"tatum":25913,"vittorio":25914,"##nden":25915,"cholera":25916,"##dana":25917,"##oper":25918,"bracing":25919,"indifference":25920,"projectile":25921,"superliga":25922,"##chee":25923,"realises":25924,"upgrading":25925,"299":25926,"porte":25927,"retribution":25928,"##vies":25929,"nk":25930,"stil":25931,"##resses":25932,"ama":25933,"bureaucracy":25934,"blackberry":25935,"bosch":25936,"testosterone":25937,"collapses":25938,"greer":25939,"##pathic":25940,"ioc":25941,"fifties":25942,"malls":25943,"##erved":25944,"bao":25945,"baskets":25946,"adolescents":25947,"siegfried":25948,"##osity":25949,"##tosis":25950,"mantra":25951,"detecting":25952,"existent":25953,"fledgling":25954,"##cchi":25955,"dissatisfied":25956,"gan":25957,"telecommunication":25958,"mingled":25959,"sobbed":25960,"6000":25961,"controversies":25962,"outdated":25963,"taxis":25964,"##raus":25965,"fright":25966,"slams":25967,"##lham":25968,"##fect":25969,"##tten":25970,"detectors":25971,"fetal":25972,"tanned":25973,"##uw":25974,"fray":25975,"goth":25976,"olympian":25977,"skipping":25978,"mandates":25979,"scratches":25980,"sheng":25981,"unspoken":25982,"hyundai":25983,"tracey":25984,"hotspur":25985,"restrictive":25986,"##buch":25987,"americana":25988,"mundo":25989,"##bari":25990,"burroughs":25991,"diva":25992,"vulcan":25993,"##6th":25994,"distinctions":25995,"thumping":25996,"##ngen":25997,"mikey":25998,"sheds":25999,"fide":26000,"rescues":26001,"springsteen":26002,"vested":26003,"valuation":26004,"##ece":26005,"##ely":26006,"pinnacle":26007,"rake":26008,"sylvie":26009,"##edo":26010,"almond":26011,"quivering":26012,"##irus":26013,"alteration":26014,"faltered":26015,"##wad":26016,"51st":26017,"hydra":26018,"ticked":26019,"##kato":26020,"recommends":26021,"##dicated":26022,"antigua":26023,"arjun":26024,"stagecoach":26025,"wilfred":26026,"trickle":26027,"pronouns":26028,"##pon":26029,"aryan":26030,"nighttime":26031,"##anian":26032,"gall":26033,"pea":26034,"stitch":26035,"##hei":26036,"leung":26037,"milos":26038,"##dini":26039,"eritrea":26040,"nexus":26041,"starved":26042,"snowfall":26043,"kant":26044,"parasitic":26045,"cot":26046,"discus":26047,"hana":26048,"strikers":26049,"appleton":26050,"kitchens":26051,"##erina":26052,"##partisan":26053,"##itha":26054,"##vius":26055,"disclose":26056,"metis":26057,"##channel":26058,"1701":26059,"tesla":26060,"##vera":26061,"fitch":26062,"1735":26063,"blooded":26064,"##tila":26065,"decimal":26066,"##tang":26067,"##bai":26068,"cyclones":26069,"eun":26070,"bottled":26071,"peas":26072,"pensacola":26073,"basha":26074,"bolivian":26075,"crabs":26076,"boil":26077,"lanterns":26078,"partridge":26079,"roofed":26080,"1645":26081,"necks":26082,"##phila":26083,"opined":26084,"patting":26085,"##kla":26086,"##lland":26087,"chuckles":26088,"volta":26089,"whereupon":26090,"##nche":26091,"devout":26092,"euroleague":26093,"suicidal":26094,"##dee":26095,"inherently":26096,"involuntary":26097,"knitting":26098,"nasser":26099,"##hide":26100,"puppets":26101,"colourful":26102,"courageous":26103,"southend":26104,"stills":26105,"miraculous":26106,"hodgson":26107,"richer":26108,"rochdale":26109,"ethernet":26110,"greta":26111,"uniting":26112,"prism":26113,"umm":26114,"##haya":26115,"##itical":26116,"##utation":26117,"deterioration":26118,"pointe":26119,"prowess":26120,"##ropriation":26121,"lids":26122,"scranton":26123,"billings":26124,"subcontinent":26125,"##koff":26126,"##scope":26127,"brute":26128,"kellogg":26129,"psalms":26130,"degraded":26131,"##vez":26132,"stanisław":26133,"##ructured":26134,"ferreira":26135,"pun":26136,"astonishing":26137,"gunnar":26138,"##yat":26139,"arya":26140,"prc":26141,"gottfried":26142,"##tight":26143,"excursion":26144,"##ographer":26145,"dina":26146,"##quil":26147,"##nare":26148,"huffington":26149,"illustrious":26150,"wilbur":26151,"gundam":26152,"verandah":26153,"##zard":26154,"naacp":26155,"##odle":26156,"constructive":26157,"fjord":26158,"kade":26159,"##naud":26160,"generosity":26161,"thrilling":26162,"baseline":26163,"cayman":26164,"frankish":26165,"plastics":26166,"accommodations":26167,"zoological":26168,"##fting":26169,"cedric":26170,"qb":26171,"motorized":26172,"##dome":26173,"##otted":26174,"squealed":26175,"tackled":26176,"canucks":26177,"budgets":26178,"situ":26179,"asthma":26180,"dail":26181,"gabled":26182,"grasslands":26183,"whimpered":26184,"writhing":26185,"judgments":26186,"##65":26187,"minnie":26188,"pv":26189,"##carbon":26190,"bananas":26191,"grille":26192,"domes":26193,"monique":26194,"odin":26195,"maguire":26196,"markham":26197,"tierney":26198,"##estra":26199,"##chua":26200,"libel":26201,"poke":26202,"speedy":26203,"atrium":26204,"laval":26205,"notwithstanding":26206,"##edly":26207,"fai":26208,"kala":26209,"##sur":26210,"robb":26211,"##sma":26212,"listings":26213,"luz":26214,"supplementary":26215,"tianjin":26216,"##acing":26217,"enzo":26218,"jd":26219,"ric":26220,"scanner":26221,"croats":26222,"transcribed":26223,"##49":26224,"arden":26225,"cv":26226,"##hair":26227,"##raphy":26228,"##lver":26229,"##uy":26230,"357":26231,"seventies":26232,"staggering":26233,"alam":26234,"horticultural":26235,"hs":26236,"regression":26237,"timbers":26238,"blasting":26239,"##ounded":26240,"montagu":26241,"manipulating":26242,"##cit":26243,"catalytic":26244,"1550":26245,"troopers":26246,"##meo":26247,"condemnation":26248,"fitzpatrick":26249,"##oire":26250,"##roved":26251,"inexperienced":26252,"1670":26253,"castes":26254,"##lative":26255,"outing":26256,"314":26257,"dubois":26258,"flicking":26259,"quarrel":26260,"ste":26261,"learners":26262,"1625":26263,"iq":26264,"whistled":26265,"##class":26266,"282":26267,"classify":26268,"tariffs":26269,"temperament":26270,"355":26271,"folly":26272,"liszt":26273,"##yles":26274,"immersed":26275,"jordanian":26276,"ceasefire":26277,"apparel":26278,"extras":26279,"maru":26280,"fished":26281,"##bio":26282,"harta":26283,"stockport":26284,"assortment":26285,"craftsman":26286,"paralysis":26287,"transmitters":26288,"##cola":26289,"blindness":26290,"##wk":26291,"fatally":26292,"proficiency":26293,"solemnly":26294,"##orno":26295,"repairing":26296,"amore":26297,"groceries":26298,"ultraviolet":26299,"##chase":26300,"schoolhouse":26301,"##tua":26302,"resurgence":26303,"nailed":26304,"##otype":26305,"##×":26306,"ruse":26307,"saliva":26308,"diagrams":26309,"##tructing":26310,"albans":26311,"rann":26312,"thirties":26313,"1b":26314,"antennas":26315,"hilarious":26316,"cougars":26317,"paddington":26318,"stats":26319,"##eger":26320,"breakaway":26321,"ipod":26322,"reza":26323,"authorship":26324,"prohibiting":26325,"scoffed":26326,"##etz":26327,"##ttle":26328,"conscription":26329,"defected":26330,"trondheim":26331,"##fires":26332,"ivanov":26333,"keenan":26334,"##adan":26335,"##ciful":26336,"##fb":26337,"##slow":26338,"locating":26339,"##ials":26340,"##tford":26341,"cadiz":26342,"basalt":26343,"blankly":26344,"interned":26345,"rags":26346,"rattling":26347,"##tick":26348,"carpathian":26349,"reassured":26350,"sync":26351,"bum":26352,"guildford":26353,"iss":26354,"staunch":26355,"##onga":26356,"astronomers":26357,"sera":26358,"sofie":26359,"emergencies":26360,"susquehanna":26361,"##heard":26362,"duc":26363,"mastery":26364,"vh1":26365,"williamsburg":26366,"bayer":26367,"buckled":26368,"craving":26369,"##khan":26370,"##rdes":26371,"bloomington":26372,"##write":26373,"alton":26374,"barbecue":26375,"##bians":26376,"justine":26377,"##hri":26378,"##ndt":26379,"delightful":26380,"smartphone":26381,"newtown":26382,"photon":26383,"retrieval":26384,"peugeot":26385,"hissing":26386,"##monium":26387,"##orough":26388,"flavors":26389,"lighted":26390,"relaunched":26391,"tainted":26392,"##games":26393,"##lysis":26394,"anarchy":26395,"microscopic":26396,"hopping":26397,"adept":26398,"evade":26399,"evie":26400,"##beau":26401,"inhibit":26402,"sinn":26403,"adjustable":26404,"hurst":26405,"intuition":26406,"wilton":26407,"cisco":26408,"44th":26409,"lawful":26410,"lowlands":26411,"stockings":26412,"thierry":26413,"##dalen":26414,"##hila":26415,"##nai":26416,"fates":26417,"prank":26418,"tb":26419,"maison":26420,"lobbied":26421,"provocative":26422,"1724":26423,"4a":26424,"utopia":26425,"##qual":26426,"carbonate":26427,"gujarati":26428,"purcell":26429,"##rford":26430,"curtiss":26431,"##mei":26432,"overgrown":26433,"arenas":26434,"mediation":26435,"swallows":26436,"##rnik":26437,"respectful":26438,"turnbull":26439,"##hedron":26440,"##hope":26441,"alyssa":26442,"ozone":26443,"##ʻi":26444,"ami":26445,"gestapo":26446,"johansson":26447,"snooker":26448,"canteen":26449,"cuff":26450,"declines":26451,"empathy":26452,"stigma":26453,"##ags":26454,"##iner":26455,"##raine":26456,"taxpayers":26457,"gui":26458,"volga":26459,"##wright":26460,"##copic":26461,"lifespan":26462,"overcame":26463,"tattooed":26464,"enactment":26465,"giggles":26466,"##ador":26467,"##camp":26468,"barrington":26469,"bribe":26470,"obligatory":26471,"orbiting":26472,"peng":26473,"##enas":26474,"elusive":26475,"sucker":26476,"##vating":26477,"cong":26478,"hardship":26479,"empowered":26480,"anticipating":26481,"estrada":26482,"cryptic":26483,"greasy":26484,"detainees":26485,"planck":26486,"sudbury":26487,"plaid":26488,"dod":26489,"marriott":26490,"kayla":26491,"##ears":26492,"##vb":26493,"##zd":26494,"mortally":26495,"##hein":26496,"cognition":26497,"radha":26498,"319":26499,"liechtenstein":26500,"meade":26501,"richly":26502,"argyle":26503,"harpsichord":26504,"liberalism":26505,"trumpets":26506,"lauded":26507,"tyrant":26508,"salsa":26509,"tiled":26510,"lear":26511,"promoters":26512,"reused":26513,"slicing":26514,"trident":26515,"##chuk":26516,"##gami":26517,"##lka":26518,"cantor":26519,"checkpoint":26520,"##points":26521,"gaul":26522,"leger":26523,"mammalian":26524,"##tov":26525,"##aar":26526,"##schaft":26527,"doha":26528,"frenchman":26529,"nirvana":26530,"##vino":26531,"delgado":26532,"headlining":26533,"##eron":26534,"##iography":26535,"jug":26536,"tko":26537,"1649":26538,"naga":26539,"intersections":26540,"##jia":26541,"benfica":26542,"nawab":26543,"##suka":26544,"ashford":26545,"gulp":26546,"##deck":26547,"##vill":26548,"##rug":26549,"brentford":26550,"frazier":26551,"pleasures":26552,"dunne":26553,"potsdam":26554,"shenzhen":26555,"dentistry":26556,"##tec":26557,"flanagan":26558,"##dorff":26559,"##hear":26560,"chorale":26561,"dinah":26562,"prem":26563,"quezon":26564,"##rogated":26565,"relinquished":26566,"sutra":26567,"terri":26568,"##pani":26569,"flaps":26570,"##rissa":26571,"poly":26572,"##rnet":26573,"homme":26574,"aback":26575,"##eki":26576,"linger":26577,"womb":26578,"##kson":26579,"##lewood":26580,"doorstep":26581,"orthodoxy":26582,"threaded":26583,"westfield":26584,"##rval":26585,"dioceses":26586,"fridays":26587,"subsided":26588,"##gata":26589,"loyalists":26590,"##biotic":26591,"##ettes":26592,"letterman":26593,"lunatic":26594,"prelate":26595,"tenderly":26596,"invariably":26597,"souza":26598,"thug":26599,"winslow":26600,"##otide":26601,"furlongs":26602,"gogh":26603,"jeopardy":26604,"##runa":26605,"pegasus":26606,"##umble":26607,"humiliated":26608,"standalone":26609,"tagged":26610,"##roller":26611,"freshmen":26612,"klan":26613,"##bright":26614,"attaining":26615,"initiating":26616,"transatlantic":26617,"logged":26618,"viz":26619,"##uance":26620,"1723":26621,"combatants":26622,"intervening":26623,"stephane":26624,"chieftain":26625,"despised":26626,"grazed":26627,"317":26628,"cdc":26629,"galveston":26630,"godzilla":26631,"macro":26632,"simulate":26633,"##planes":26634,"parades":26635,"##esses":26636,"960":26637,"##ductive":26638,"##unes":26639,"equator":26640,"overdose":26641,"##cans":26642,"##hosh":26643,"##lifting":26644,"joshi":26645,"epstein":26646,"sonora":26647,"treacherous":26648,"aquatics":26649,"manchu":26650,"responsive":26651,"##sation":26652,"supervisory":26653,"##christ":26654,"##llins":26655,"##ibar":26656,"##balance":26657,"##uso":26658,"kimball":26659,"karlsruhe":26660,"mab":26661,"##emy":26662,"ignores":26663,"phonetic":26664,"reuters":26665,"spaghetti":26666,"820":26667,"almighty":26668,"danzig":26669,"rumbling":26670,"tombstone":26671,"designations":26672,"lured":26673,"outset":26674,"##felt":26675,"supermarkets":26676,"##wt":26677,"grupo":26678,"kei":26679,"kraft":26680,"susanna":26681,"##blood":26682,"comprehension":26683,"genealogy":26684,"##aghan":26685,"##verted":26686,"redding":26687,"##ythe":26688,"1722":26689,"bowing":26690,"##pore":26691,"##roi":26692,"lest":26693,"sharpened":26694,"fulbright":26695,"valkyrie":26696,"sikhs":26697,"##unds":26698,"swans":26699,"bouquet":26700,"merritt":26701,"##tage":26702,"##venting":26703,"commuted":26704,"redhead":26705,"clerks":26706,"leasing":26707,"cesare":26708,"dea":26709,"hazy":26710,"##vances":26711,"fledged":26712,"greenfield":26713,"servicemen":26714,"##gical":26715,"armando":26716,"blackout":26717,"dt":26718,"sagged":26719,"downloadable":26720,"intra":26721,"potion":26722,"pods":26723,"##4th":26724,"##mism":26725,"xp":26726,"attendants":26727,"gambia":26728,"stale":26729,"##ntine":26730,"plump":26731,"asteroids":26732,"rediscovered":26733,"buds":26734,"flea":26735,"hive":26736,"##neas":26737,"1737":26738,"classifications":26739,"debuts":26740,"##eles":26741,"olympus":26742,"scala":26743,"##eurs":26744,"##gno":26745,"##mute":26746,"hummed":26747,"sigismund":26748,"visuals":26749,"wiggled":26750,"await":26751,"pilasters":26752,"clench":26753,"sulfate":26754,"##ances":26755,"bellevue":26756,"enigma":26757,"trainee":26758,"snort":26759,"##sw":26760,"clouded":26761,"denim":26762,"##rank":26763,"##rder":26764,"churning":26765,"hartman":26766,"lodges":26767,"riches":26768,"sima":26769,"##missible":26770,"accountable":26771,"socrates":26772,"regulates":26773,"mueller":26774,"##cr":26775,"1702":26776,"avoids":26777,"solids":26778,"himalayas":26779,"nutrient":26780,"pup":26781,"##jevic":26782,"squat":26783,"fades":26784,"nec":26785,"##lates":26786,"##pina":26787,"##rona":26788,"##ου":26789,"privateer":26790,"tequila":26791,"##gative":26792,"##mpton":26793,"apt":26794,"hornet":26795,"immortals":26796,"##dou":26797,"asturias":26798,"cleansing":26799,"dario":26800,"##rries":26801,"##anta":26802,"etymology":26803,"servicing":26804,"zhejiang":26805,"##venor":26806,"##nx":26807,"horned":26808,"erasmus":26809,"rayon":26810,"relocating":26811,"£10":26812,"##bags":26813,"escalated":26814,"promenade":26815,"stubble":26816,"2010s":26817,"artisans":26818,"axial":26819,"liquids":26820,"mora":26821,"sho":26822,"yoo":26823,"##tsky":26824,"bundles":26825,"oldies":26826,"##nally":26827,"notification":26828,"bastion":26829,"##ths":26830,"sparkle":26831,"##lved":26832,"1728":26833,"leash":26834,"pathogen":26835,"highs":26836,"##hmi":26837,"immature":26838,"880":26839,"gonzaga":26840,"ignatius":26841,"mansions":26842,"monterrey":26843,"sweets":26844,"bryson":26845,"##loe":26846,"polled":26847,"regatta":26848,"brightest":26849,"pei":26850,"rosy":26851,"squid":26852,"hatfield":26853,"payroll":26854,"addict":26855,"meath":26856,"cornerback":26857,"heaviest":26858,"lodging":26859,"##mage":26860,"capcom":26861,"rippled":26862,"##sily":26863,"barnet":26864,"mayhem":26865,"ymca":26866,"snuggled":26867,"rousseau":26868,"##cute":26869,"blanchard":26870,"284":26871,"fragmented":26872,"leighton":26873,"chromosomes":26874,"risking":26875,"##md":26876,"##strel":26877,"##utter":26878,"corinne":26879,"coyotes":26880,"cynical":26881,"hiroshi":26882,"yeomanry":26883,"##ractive":26884,"ebook":26885,"grading":26886,"mandela":26887,"plume":26888,"agustin":26889,"magdalene":26890,"##rkin":26891,"bea":26892,"femme":26893,"trafford":26894,"##coll":26895,"##lun":26896,"##tance":26897,"52nd":26898,"fourier":26899,"upton":26900,"##mental":26901,"camilla":26902,"gust":26903,"iihf":26904,"islamabad":26905,"longevity":26906,"##kala":26907,"feldman":26908,"netting":26909,"##rization":26910,"endeavour":26911,"foraging":26912,"mfa":26913,"orr":26914,"##open":26915,"greyish":26916,"contradiction":26917,"graz":26918,"##ruff":26919,"handicapped":26920,"marlene":26921,"tweed":26922,"oaxaca":26923,"spp":26924,"campos":26925,"miocene":26926,"pri":26927,"configured":26928,"cooks":26929,"pluto":26930,"cozy":26931,"pornographic":26932,"##entes":26933,"70th":26934,"fairness":26935,"glided":26936,"jonny":26937,"lynne":26938,"rounding":26939,"sired":26940,"##emon":26941,"##nist":26942,"remade":26943,"uncover":26944,"##mack":26945,"complied":26946,"lei":26947,"newsweek":26948,"##jured":26949,"##parts":26950,"##enting":26951,"##pg":26952,"293":26953,"finer":26954,"guerrillas":26955,"athenian":26956,"deng":26957,"disused":26958,"stepmother":26959,"accuse":26960,"gingerly":26961,"seduction":26962,"521":26963,"confronting":26964,"##walker":26965,"##going":26966,"gora":26967,"nostalgia":26968,"sabres":26969,"virginity":26970,"wrenched":26971,"##minated":26972,"syndication":26973,"wielding":26974,"eyre":26975,"##56":26976,"##gnon":26977,"##igny":26978,"behaved":26979,"taxpayer":26980,"sweeps":26981,"##growth":26982,"childless":26983,"gallant":26984,"##ywood":26985,"amplified":26986,"geraldine":26987,"scrape":26988,"##ffi":26989,"babylonian":26990,"fresco":26991,"##rdan":26992,"##kney":26993,"##position":26994,"1718":26995,"restricting":26996,"tack":26997,"fukuoka":26998,"osborn":26999,"selector":27000,"partnering":27001,"##dlow":27002,"318":27003,"gnu":27004,"kia":27005,"tak":27006,"whitley":27007,"gables":27008,"##54":27009,"##mania":27010,"mri":27011,"softness":27012,"immersion":27013,"##bots":27014,"##evsky":27015,"1713":27016,"chilling":27017,"insignificant":27018,"pcs":27019,"##uis":27020,"elites":27021,"lina":27022,"purported":27023,"supplemental":27024,"teaming":27025,"##americana":27026,"##dding":27027,"##inton":27028,"proficient":27029,"rouen":27030,"##nage":27031,"##rret":27032,"niccolo":27033,"selects":27034,"##bread":27035,"fluffy":27036,"1621":27037,"gruff":27038,"knotted":27039,"mukherjee":27040,"polgara":27041,"thrash":27042,"nicholls":27043,"secluded":27044,"smoothing":27045,"thru":27046,"corsica":27047,"loaf":27048,"whitaker":27049,"inquiries":27050,"##rrier":27051,"##kam":27052,"indochina":27053,"289":27054,"marlins":27055,"myles":27056,"peking":27057,"##tea":27058,"extracts":27059,"pastry":27060,"superhuman":27061,"connacht":27062,"vogel":27063,"##ditional":27064,"##het":27065,"##udged":27066,"##lash":27067,"gloss":27068,"quarries":27069,"refit":27070,"teaser":27071,"##alic":27072,"##gaon":27073,"20s":27074,"materialized":27075,"sling":27076,"camped":27077,"pickering":27078,"tung":27079,"tracker":27080,"pursuant":27081,"##cide":27082,"cranes":27083,"soc":27084,"##cini":27085,"##typical":27086,"##viere":27087,"anhalt":27088,"overboard":27089,"workout":27090,"chores":27091,"fares":27092,"orphaned":27093,"stains":27094,"##logie":27095,"fenton":27096,"surpassing":27097,"joyah":27098,"triggers":27099,"##itte":27100,"grandmaster":27101,"##lass":27102,"##lists":27103,"clapping":27104,"fraudulent":27105,"ledger":27106,"nagasaki":27107,"##cor":27108,"##nosis":27109,"##tsa":27110,"eucalyptus":27111,"tun":27112,"##icio":27113,"##rney":27114,"##tara":27115,"dax":27116,"heroism":27117,"ina":27118,"wrexham":27119,"onboard":27120,"unsigned":27121,"##dates":27122,"moshe":27123,"galley":27124,"winnie":27125,"droplets":27126,"exiles":27127,"praises":27128,"watered":27129,"noodles":27130,"##aia":27131,"fein":27132,"adi":27133,"leland":27134,"multicultural":27135,"stink":27136,"bingo":27137,"comets":27138,"erskine":27139,"modernized":27140,"canned":27141,"constraint":27142,"domestically":27143,"chemotherapy":27144,"featherweight":27145,"stifled":27146,"##mum":27147,"darkly":27148,"irresistible":27149,"refreshing":27150,"hasty":27151,"isolate":27152,"##oys":27153,"kitchener":27154,"planners":27155,"##wehr":27156,"cages":27157,"yarn":27158,"implant":27159,"toulon":27160,"elects":27161,"childbirth":27162,"yue":27163,"##lind":27164,"##lone":27165,"cn":27166,"rightful":27167,"sportsman":27168,"junctions":27169,"remodeled":27170,"specifies":27171,"##rgh":27172,"291":27173,"##oons":27174,"complimented":27175,"##urgent":27176,"lister":27177,"ot":27178,"##logic":27179,"bequeathed":27180,"cheekbones":27181,"fontana":27182,"gabby":27183,"##dial":27184,"amadeus":27185,"corrugated":27186,"maverick":27187,"resented":27188,"triangles":27189,"##hered":27190,"##usly":27191,"nazareth":27192,"tyrol":27193,"1675":27194,"assent":27195,"poorer":27196,"sectional":27197,"aegean":27198,"##cous":27199,"296":27200,"nylon":27201,"ghanaian":27202,"##egorical":27203,"##weig":27204,"cushions":27205,"forbid":27206,"fusiliers":27207,"obstruction":27208,"somerville":27209,"##scia":27210,"dime":27211,"earrings":27212,"elliptical":27213,"leyte":27214,"oder":27215,"polymers":27216,"timmy":27217,"atm":27218,"midtown":27219,"piloted":27220,"settles":27221,"continual":27222,"externally":27223,"mayfield":27224,"##uh":27225,"enrichment":27226,"henson":27227,"keane":27228,"persians":27229,"1733":27230,"benji":27231,"braden":27232,"pep":27233,"324":27234,"##efe":27235,"contenders":27236,"pepsi":27237,"valet":27238,"##isches":27239,"298":27240,"##asse":27241,"##earing":27242,"goofy":27243,"stroll":27244,"##amen":27245,"authoritarian":27246,"occurrences":27247,"adversary":27248,"ahmedabad":27249,"tangent":27250,"toppled":27251,"dorchester":27252,"1672":27253,"modernism":27254,"marxism":27255,"islamist":27256,"charlemagne":27257,"exponential":27258,"racks":27259,"unicode":27260,"brunette":27261,"mbc":27262,"pic":27263,"skirmish":27264,"##bund":27265,"##lad":27266,"##powered":27267,"##yst":27268,"hoisted":27269,"messina":27270,"shatter":27271,"##ctum":27272,"jedi":27273,"vantage":27274,"##music":27275,"##neil":27276,"clemens":27277,"mahmoud":27278,"corrupted":27279,"authentication":27280,"lowry":27281,"nils":27282,"##washed":27283,"omnibus":27284,"wounding":27285,"jillian":27286,"##itors":27287,"##opped":27288,"serialized":27289,"narcotics":27290,"handheld":27291,"##arm":27292,"##plicity":27293,"intersecting":27294,"stimulating":27295,"##onis":27296,"crate":27297,"fellowships":27298,"hemingway":27299,"casinos":27300,"climatic":27301,"fordham":27302,"copeland":27303,"drip":27304,"beatty":27305,"leaflets":27306,"robber":27307,"brothel":27308,"madeira":27309,"##hedral":27310,"sphinx":27311,"ultrasound":27312,"##vana":27313,"valor":27314,"forbade":27315,"leonid":27316,"villas":27317,"##aldo":27318,"duane":27319,"marquez":27320,"##cytes":27321,"disadvantaged":27322,"forearms":27323,"kawasaki":27324,"reacts":27325,"consular":27326,"lax":27327,"uncles":27328,"uphold":27329,"##hopper":27330,"concepcion":27331,"dorsey":27332,"lass":27333,"##izan":27334,"arching":27335,"passageway":27336,"1708":27337,"researches":27338,"tia":27339,"internationals":27340,"##graphs":27341,"##opers":27342,"distinguishes":27343,"javanese":27344,"divert":27345,"##uven":27346,"plotted":27347,"##listic":27348,"##rwin":27349,"##erik":27350,"##tify":27351,"affirmative":27352,"signifies":27353,"validation":27354,"##bson":27355,"kari":27356,"felicity":27357,"georgina":27358,"zulu":27359,"##eros":27360,"##rained":27361,"##rath":27362,"overcoming":27363,"##dot":27364,"argyll":27365,"##rbin":27366,"1734":27367,"chiba":27368,"ratification":27369,"windy":27370,"earls":27371,"parapet":27372,"##marks":27373,"hunan":27374,"pristine":27375,"astrid":27376,"punta":27377,"##gart":27378,"brodie":27379,"##kota":27380,"##oder":27381,"malaga":27382,"minerva":27383,"rouse":27384,"##phonic":27385,"bellowed":27386,"pagoda":27387,"portals":27388,"reclamation":27389,"##gur":27390,"##odies":27391,"##⁄₄":27392,"parentheses":27393,"quoting":27394,"allergic":27395,"palette":27396,"showcases":27397,"benefactor":27398,"heartland":27399,"nonlinear":27400,"##tness":27401,"bladed":27402,"cheerfully":27403,"scans":27404,"##ety":27405,"##hone":27406,"1666":27407,"girlfriends":27408,"pedersen":27409,"hiram":27410,"sous":27411,"##liche":27412,"##nator":27413,"1683":27414,"##nery":27415,"##orio":27416,"##umen":27417,"bobo":27418,"primaries":27419,"smiley":27420,"##cb":27421,"unearthed":27422,"uniformly":27423,"fis":27424,"metadata":27425,"1635":27426,"ind":27427,"##oted":27428,"recoil":27429,"##titles":27430,"##tura":27431,"##ια":27432,"406":27433,"hilbert":27434,"jamestown":27435,"mcmillan":27436,"tulane":27437,"seychelles":27438,"##frid":27439,"antics":27440,"coli":27441,"fated":27442,"stucco":27443,"##grants":27444,"1654":27445,"bulky":27446,"accolades":27447,"arrays":27448,"caledonian":27449,"carnage":27450,"optimism":27451,"puebla":27452,"##tative":27453,"##cave":27454,"enforcing":27455,"rotherham":27456,"seo":27457,"dunlop":27458,"aeronautics":27459,"chimed":27460,"incline":27461,"zoning":27462,"archduke":27463,"hellenistic":27464,"##oses":27465,"##sions":27466,"candi":27467,"thong":27468,"##ople":27469,"magnate":27470,"rustic":27471,"##rsk":27472,"projective":27473,"slant":27474,"##offs":27475,"danes":27476,"hollis":27477,"vocalists":27478,"##ammed":27479,"congenital":27480,"contend":27481,"gesellschaft":27482,"##ocating":27483,"##pressive":27484,"douglass":27485,"quieter":27486,"##cm":27487,"##kshi":27488,"howled":27489,"salim":27490,"spontaneously":27491,"townsville":27492,"buena":27493,"southport":27494,"##bold":27495,"kato":27496,"1638":27497,"faerie":27498,"stiffly":27499,"##vus":27500,"##rled":27501,"297":27502,"flawless":27503,"realising":27504,"taboo":27505,"##7th":27506,"bytes":27507,"straightening":27508,"356":27509,"jena":27510,"##hid":27511,"##rmin":27512,"cartwright":27513,"berber":27514,"bertram":27515,"soloists":27516,"411":27517,"noses":27518,"417":27519,"coping":27520,"fission":27521,"hardin":27522,"inca":27523,"##cen":27524,"1717":27525,"mobilized":27526,"vhf":27527,"##raf":27528,"biscuits":27529,"curate":27530,"##85":27531,"##anial":27532,"331":27533,"gaunt":27534,"neighbourhoods":27535,"1540":27536,"##abas":27537,"blanca":27538,"bypassed":27539,"sockets":27540,"behold":27541,"coincidentally":27542,"##bane":27543,"nara":27544,"shave":27545,"splinter":27546,"terrific":27547,"##arion":27548,"##erian":27549,"commonplace":27550,"juris":27551,"redwood":27552,"waistband":27553,"boxed":27554,"caitlin":27555,"fingerprints":27556,"jennie":27557,"naturalized":27558,"##ired":27559,"balfour":27560,"craters":27561,"jody":27562,"bungalow":27563,"hugely":27564,"quilt":27565,"glitter":27566,"pigeons":27567,"undertaker":27568,"bulging":27569,"constrained":27570,"goo":27571,"##sil":27572,"##akh":27573,"assimilation":27574,"reworked":27575,"##person":27576,"persuasion":27577,"##pants":27578,"felicia":27579,"##cliff":27580,"##ulent":27581,"1732":27582,"explodes":27583,"##dun":27584,"##inium":27585,"##zic":27586,"lyman":27587,"vulture":27588,"hog":27589,"overlook":27590,"begs":27591,"northwards":27592,"ow":27593,"spoil":27594,"##urer":27595,"fatima":27596,"favorably":27597,"accumulate":27598,"sargent":27599,"sorority":27600,"corresponded":27601,"dispersal":27602,"kochi":27603,"toned":27604,"##imi":27605,"##lita":27606,"internacional":27607,"newfound":27608,"##agger":27609,"##lynn":27610,"##rigue":27611,"booths":27612,"peanuts":27613,"##eborg":27614,"medicare":27615,"muriel":27616,"nur":27617,"##uram":27618,"crates":27619,"millennia":27620,"pajamas":27621,"worsened":27622,"##breakers":27623,"jimi":27624,"vanuatu":27625,"yawned":27626,"##udeau":27627,"carousel":27628,"##hony":27629,"hurdle":27630,"##ccus":27631,"##mounted":27632,"##pod":27633,"rv":27634,"##eche":27635,"airship":27636,"ambiguity":27637,"compulsion":27638,"recapture":27639,"##claiming":27640,"arthritis":27641,"##osomal":27642,"1667":27643,"asserting":27644,"ngc":27645,"sniffing":27646,"dade":27647,"discontent":27648,"glendale":27649,"ported":27650,"##amina":27651,"defamation":27652,"rammed":27653,"##scent":27654,"fling":27655,"livingstone":27656,"##fleet":27657,"875":27658,"##ppy":27659,"apocalyptic":27660,"comrade":27661,"lcd":27662,"##lowe":27663,"cessna":27664,"eine":27665,"persecuted":27666,"subsistence":27667,"demi":27668,"hoop":27669,"reliefs":27670,"710":27671,"coptic":27672,"progressing":27673,"stemmed":27674,"perpetrators":27675,"1665":27676,"priestess":27677,"##nio":27678,"dobson":27679,"ebony":27680,"rooster":27681,"itf":27682,"tortricidae":27683,"##bbon":27684,"##jian":27685,"cleanup":27686,"##jean":27687,"##øy":27688,"1721":27689,"eighties":27690,"taxonomic":27691,"holiness":27692,"##hearted":27693,"##spar":27694,"antilles":27695,"showcasing":27696,"stabilized":27697,"##nb":27698,"gia":27699,"mascara":27700,"michelangelo":27701,"dawned":27702,"##uria":27703,"##vinsky":27704,"extinguished":27705,"fitz":27706,"grotesque":27707,"£100":27708,"##fera":27709,"##loid":27710,"##mous":27711,"barges":27712,"neue":27713,"throbbed":27714,"cipher":27715,"johnnie":27716,"##a1":27717,"##mpt":27718,"outburst":27719,"##swick":27720,"spearheaded":27721,"administrations":27722,"c1":27723,"heartbreak":27724,"pixels":27725,"pleasantly":27726,"##enay":27727,"lombardy":27728,"plush":27729,"##nsed":27730,"bobbie":27731,"##hly":27732,"reapers":27733,"tremor":27734,"xiang":27735,"minogue":27736,"substantive":27737,"hitch":27738,"barak":27739,"##wyl":27740,"kwan":27741,"##encia":27742,"910":27743,"obscene":27744,"elegance":27745,"indus":27746,"surfer":27747,"bribery":27748,"conserve":27749,"##hyllum":27750,"##masters":27751,"horatio":27752,"##fat":27753,"apes":27754,"rebound":27755,"psychotic":27756,"##pour":27757,"iteration":27758,"##mium":27759,"##vani":27760,"botanic":27761,"horribly":27762,"antiques":27763,"dispose":27764,"paxton":27765,"##hli":27766,"##wg":27767,"timeless":27768,"1704":27769,"disregard":27770,"engraver":27771,"hounds":27772,"##bau":27773,"##version":27774,"looted":27775,"uno":27776,"facilitates":27777,"groans":27778,"masjid":27779,"rutland":27780,"antibody":27781,"disqualification":27782,"decatur":27783,"footballers":27784,"quake":27785,"slacks":27786,"48th":27787,"rein":27788,"scribe":27789,"stabilize":27790,"commits":27791,"exemplary":27792,"tho":27793,"##hort":27794,"##chison":27795,"pantry":27796,"traversed":27797,"##hiti":27798,"disrepair":27799,"identifiable":27800,"vibrated":27801,"baccalaureate":27802,"##nnis":27803,"csa":27804,"interviewing":27805,"##iensis":27806,"##raße":27807,"greaves":27808,"wealthiest":27809,"343":27810,"classed":27811,"jogged":27812,"£5":27813,"##58":27814,"##atal":27815,"illuminating":27816,"knicks":27817,"respecting":27818,"##uno":27819,"scrubbed":27820,"##iji":27821,"##dles":27822,"kruger":27823,"moods":27824,"growls":27825,"raider":27826,"silvia":27827,"chefs":27828,"kam":27829,"vr":27830,"cree":27831,"percival":27832,"##terol":27833,"gunter":27834,"counterattack":27835,"defiant":27836,"henan":27837,"ze":27838,"##rasia":27839,"##riety":27840,"equivalence":27841,"submissions":27842,"##fra":27843,"##thor":27844,"bautista":27845,"mechanically":27846,"##heater":27847,"cornice":27848,"herbal":27849,"templar":27850,"##mering":27851,"outputs":27852,"ruining":27853,"ligand":27854,"renumbered":27855,"extravagant":27856,"mika":27857,"blockbuster":27858,"eta":27859,"insurrection":27860,"##ilia":27861,"darkening":27862,"ferocious":27863,"pianos":27864,"strife":27865,"kinship":27866,"##aer":27867,"melee":27868,"##anor":27869,"##iste":27870,"##may":27871,"##oue":27872,"decidedly":27873,"weep":27874,"##jad":27875,"##missive":27876,"##ppel":27877,"354":27878,"puget":27879,"unease":27880,"##gnant":27881,"1629":27882,"hammering":27883,"kassel":27884,"ob":27885,"wessex":27886,"##lga":27887,"bromwich":27888,"egan":27889,"paranoia":27890,"utilization":27891,"##atable":27892,"##idad":27893,"contradictory":27894,"provoke":27895,"##ols":27896,"##ouring":27897,"##tangled":27898,"knesset":27899,"##very":27900,"##lette":27901,"plumbing":27902,"##sden":27903,"##¹":27904,"greensboro":27905,"occult":27906,"sniff":27907,"338":27908,"zev":27909,"beaming":27910,"gamer":27911,"haggard":27912,"mahal":27913,"##olt":27914,"##pins":27915,"mendes":27916,"utmost":27917,"briefing":27918,"gunnery":27919,"##gut":27920,"##pher":27921,"##zh":27922,"##rok":27923,"1679":27924,"khalifa":27925,"sonya":27926,"##boot":27927,"principals":27928,"urbana":27929,"wiring":27930,"##liffe":27931,"##minating":27932,"##rrado":27933,"dahl":27934,"nyu":27935,"skepticism":27936,"np":27937,"townspeople":27938,"ithaca":27939,"lobster":27940,"somethin":27941,"##fur":27942,"##arina":27943,"##−1":27944,"freighter":27945,"zimmerman":27946,"biceps":27947,"contractual":27948,"##herton":27949,"amend":27950,"hurrying":27951,"subconscious":27952,"##anal":27953,"336":27954,"meng":27955,"clermont":27956,"spawning":27957,"##eia":27958,"##lub":27959,"dignitaries":27960,"impetus":27961,"snacks":27962,"spotting":27963,"twigs":27964,"##bilis":27965,"##cz":27966,"##ouk":27967,"libertadores":27968,"nic":27969,"skylar":27970,"##aina":27971,"##firm":27972,"gustave":27973,"asean":27974,"##anum":27975,"dieter":27976,"legislatures":27977,"flirt":27978,"bromley":27979,"trolls":27980,"umar":27981,"##bbies":27982,"##tyle":27983,"blah":27984,"parc":27985,"bridgeport":27986,"crank":27987,"negligence":27988,"##nction":27989,"46th":27990,"constantin":27991,"molded":27992,"bandages":27993,"seriousness":27994,"00pm":27995,"siegel":27996,"carpets":27997,"compartments":27998,"upbeat":27999,"statehood":28000,"##dner":28001,"##edging":28002,"marko":28003,"730":28004,"platt":28005,"##hane":28006,"paving":28007,"##iy":28008,"1738":28009,"abbess":28010,"impatience":28011,"limousine":28012,"nbl":28013,"##talk":28014,"441":28015,"lucille":28016,"mojo":28017,"nightfall":28018,"robbers":28019,"##nais":28020,"karel":28021,"brisk":28022,"calves":28023,"replicate":28024,"ascribed":28025,"telescopes":28026,"##olf":28027,"intimidated":28028,"##reen":28029,"ballast":28030,"specialization":28031,"##sit":28032,"aerodynamic":28033,"caliphate":28034,"rainer":28035,"visionary":28036,"##arded":28037,"epsilon":28038,"##aday":28039,"##onte":28040,"aggregation":28041,"auditory":28042,"boosted":28043,"reunification":28044,"kathmandu":28045,"loco":28046,"robyn":28047,"402":28048,"acknowledges":28049,"appointing":28050,"humanoid":28051,"newell":28052,"redeveloped":28053,"restraints":28054,"##tained":28055,"barbarians":28056,"chopper":28057,"1609":28058,"italiana":28059,"##lez":28060,"##lho":28061,"investigates":28062,"wrestlemania":28063,"##anies":28064,"##bib":28065,"690":28066,"##falls":28067,"creaked":28068,"dragoons":28069,"gravely":28070,"minions":28071,"stupidity":28072,"volley":28073,"##harat":28074,"##week":28075,"musik":28076,"##eries":28077,"##uously":28078,"fungal":28079,"massimo":28080,"semantics":28081,"malvern":28082,"##ahl":28083,"##pee":28084,"discourage":28085,"embryo":28086,"imperialism":28087,"1910s":28088,"profoundly":28089,"##ddled":28090,"jiangsu":28091,"sparkled":28092,"stat":28093,"##holz":28094,"sweatshirt":28095,"tobin":28096,"##iction":28097,"sneered":28098,"##cheon":28099,"##oit":28100,"brit":28101,"causal":28102,"smyth":28103,"##neuve":28104,"diffuse":28105,"perrin":28106,"silvio":28107,"##ipes":28108,"##recht":28109,"detonated":28110,"iqbal":28111,"selma":28112,"##nism":28113,"##zumi":28114,"roasted":28115,"##riders":28116,"tay":28117,"##ados":28118,"##mament":28119,"##mut":28120,"##rud":28121,"840":28122,"completes":28123,"nipples":28124,"cfa":28125,"flavour":28126,"hirsch":28127,"##laus":28128,"calderon":28129,"sneakers":28130,"moravian":28131,"##ksha":28132,"1622":28133,"rq":28134,"294":28135,"##imeters":28136,"bodo":28137,"##isance":28138,"##pre":28139,"##ronia":28140,"anatomical":28141,"excerpt":28142,"##lke":28143,"dh":28144,"kunst":28145,"##tablished":28146,"##scoe":28147,"biomass":28148,"panted":28149,"unharmed":28150,"gael":28151,"housemates":28152,"montpellier":28153,"##59":28154,"coa":28155,"rodents":28156,"tonic":28157,"hickory":28158,"singleton":28159,"##taro":28160,"451":28161,"1719":28162,"aldo":28163,"breaststroke":28164,"dempsey":28165,"och":28166,"rocco":28167,"##cuit":28168,"merton":28169,"dissemination":28170,"midsummer":28171,"serials":28172,"##idi":28173,"haji":28174,"polynomials":28175,"##rdon":28176,"gs":28177,"enoch":28178,"prematurely":28179,"shutter":28180,"taunton":28181,"£3":28182,"##grating":28183,"##inates":28184,"archangel":28185,"harassed":28186,"##asco":28187,"326":28188,"archway":28189,"dazzling":28190,"##ecin":28191,"1736":28192,"sumo":28193,"wat":28194,"##kovich":28195,"1086":28196,"honneur":28197,"##ently":28198,"##nostic":28199,"##ttal":28200,"##idon":28201,"1605":28202,"403":28203,"1716":28204,"blogger":28205,"rents":28206,"##gnan":28207,"hires":28208,"##ikh":28209,"##dant":28210,"howie":28211,"##rons":28212,"handler":28213,"retracted":28214,"shocks":28215,"1632":28216,"arun":28217,"duluth":28218,"kepler":28219,"trumpeter":28220,"##lary":28221,"peeking":28222,"seasoned":28223,"trooper":28224,"##mara":28225,"laszlo":28226,"##iciencies":28227,"##rti":28228,"heterosexual":28229,"##inatory":28230,"##ssion":28231,"indira":28232,"jogging":28233,"##inga":28234,"##lism":28235,"beit":28236,"dissatisfaction":28237,"malice":28238,"##ately":28239,"nedra":28240,"peeling":28241,"##rgeon":28242,"47th":28243,"stadiums":28244,"475":28245,"vertigo":28246,"##ains":28247,"iced":28248,"restroom":28249,"##plify":28250,"##tub":28251,"illustrating":28252,"pear":28253,"##chner":28254,"##sibility":28255,"inorganic":28256,"rappers":28257,"receipts":28258,"watery":28259,"##kura":28260,"lucinda":28261,"##oulos":28262,"reintroduced":28263,"##8th":28264,"##tched":28265,"gracefully":28266,"saxons":28267,"nutritional":28268,"wastewater":28269,"rained":28270,"favourites":28271,"bedrock":28272,"fisted":28273,"hallways":28274,"likeness":28275,"upscale":28276,"##lateral":28277,"1580":28278,"blinds":28279,"prequel":28280,"##pps":28281,"##tama":28282,"deter":28283,"humiliating":28284,"restraining":28285,"tn":28286,"vents":28287,"1659":28288,"laundering":28289,"recess":28290,"rosary":28291,"tractors":28292,"coulter":28293,"federer":28294,"##ifiers":28295,"##plin":28296,"persistence":28297,"##quitable":28298,"geschichte":28299,"pendulum":28300,"quakers":28301,"##beam":28302,"bassett":28303,"pictorial":28304,"buffet":28305,"koln":28306,"##sitor":28307,"drills":28308,"reciprocal":28309,"shooters":28310,"##57":28311,"##cton":28312,"##tees":28313,"converge":28314,"pip":28315,"dmitri":28316,"donnelly":28317,"yamamoto":28318,"aqua":28319,"azores":28320,"demographics":28321,"hypnotic":28322,"spitfire":28323,"suspend":28324,"wryly":28325,"roderick":28326,"##rran":28327,"sebastien":28328,"##asurable":28329,"mavericks":28330,"##fles":28331,"##200":28332,"himalayan":28333,"prodigy":28334,"##iance":28335,"transvaal":28336,"demonstrators":28337,"handcuffs":28338,"dodged":28339,"mcnamara":28340,"sublime":28341,"1726":28342,"crazed":28343,"##efined":28344,"##till":28345,"ivo":28346,"pondered":28347,"reconciled":28348,"shrill":28349,"sava":28350,"##duk":28351,"bal":28352,"cad":28353,"heresy":28354,"jaipur":28355,"goran":28356,"##nished":28357,"341":28358,"lux":28359,"shelly":28360,"whitehall":28361,"##hre":28362,"israelis":28363,"peacekeeping":28364,"##wled":28365,"1703":28366,"demetrius":28367,"ousted":28368,"##arians":28369,"##zos":28370,"beale":28371,"anwar":28372,"backstroke":28373,"raged":28374,"shrinking":28375,"cremated":28376,"##yck":28377,"benign":28378,"towing":28379,"wadi":28380,"darmstadt":28381,"landfill":28382,"parana":28383,"soothe":28384,"colleen":28385,"sidewalks":28386,"mayfair":28387,"tumble":28388,"hepatitis":28389,"ferrer":28390,"superstructure":28391,"##gingly":28392,"##urse":28393,"##wee":28394,"anthropological":28395,"translators":28396,"##mies":28397,"closeness":28398,"hooves":28399,"##pw":28400,"mondays":28401,"##roll":28402,"##vita":28403,"landscaping":28404,"##urized":28405,"purification":28406,"sock":28407,"thorns":28408,"thwarted":28409,"jalan":28410,"tiberius":28411,"##taka":28412,"saline":28413,"##rito":28414,"confidently":28415,"khyber":28416,"sculptors":28417,"##ij":28418,"brahms":28419,"hammersmith":28420,"inspectors":28421,"battista":28422,"fivb":28423,"fragmentation":28424,"hackney":28425,"##uls":28426,"arresting":28427,"exercising":28428,"antoinette":28429,"bedfordshire":28430,"##zily":28431,"dyed":28432,"##hema":28433,"1656":28434,"racetrack":28435,"variability":28436,"##tique":28437,"1655":28438,"austrians":28439,"deteriorating":28440,"madman":28441,"theorists":28442,"aix":28443,"lehman":28444,"weathered":28445,"1731":28446,"decreed":28447,"eruptions":28448,"1729":28449,"flaw":28450,"quinlan":28451,"sorbonne":28452,"flutes":28453,"nunez":28454,"1711":28455,"adored":28456,"downwards":28457,"fable":28458,"rasped":28459,"1712":28460,"moritz":28461,"mouthful":28462,"renegade":28463,"shivers":28464,"stunts":28465,"dysfunction":28466,"restrain":28467,"translit":28468,"327":28469,"pancakes":28470,"##avio":28471,"##cision":28472,"##tray":28473,"351":28474,"vial":28475,"##lden":28476,"bain":28477,"##maid":28478,"##oxide":28479,"chihuahua":28480,"malacca":28481,"vimes":28482,"##rba":28483,"##rnier":28484,"1664":28485,"donnie":28486,"plaques":28487,"##ually":28488,"337":28489,"bangs":28490,"floppy":28491,"huntsville":28492,"loretta":28493,"nikolay":28494,"##otte":28495,"eater":28496,"handgun":28497,"ubiquitous":28498,"##hett":28499,"eras":28500,"zodiac":28501,"1634":28502,"##omorphic":28503,"1820s":28504,"##zog":28505,"cochran":28506,"##bula":28507,"##lithic":28508,"warring":28509,"##rada":28510,"dalai":28511,"excused":28512,"blazers":28513,"mcconnell":28514,"reeling":28515,"bot":28516,"este":28517,"##abi":28518,"geese":28519,"hoax":28520,"taxon":28521,"##bla":28522,"guitarists":28523,"##icon":28524,"condemning":28525,"hunts":28526,"inversion":28527,"moffat":28528,"taekwondo":28529,"##lvis":28530,"1624":28531,"stammered":28532,"##rest":28533,"##rzy":28534,"sousa":28535,"fundraiser":28536,"marylebone":28537,"navigable":28538,"uptown":28539,"cabbage":28540,"daniela":28541,"salman":28542,"shitty":28543,"whimper":28544,"##kian":28545,"##utive":28546,"programmers":28547,"protections":28548,"rm":28549,"##rmi":28550,"##rued":28551,"forceful":28552,"##enes":28553,"fuss":28554,"##tao":28555,"##wash":28556,"brat":28557,"oppressive":28558,"reykjavik":28559,"spartak":28560,"ticking":28561,"##inkles":28562,"##kiewicz":28563,"adolph":28564,"horst":28565,"maui":28566,"protege":28567,"straighten":28568,"cpc":28569,"landau":28570,"concourse":28571,"clements":28572,"resultant":28573,"##ando":28574,"imaginative":28575,"joo":28576,"reactivated":28577,"##rem":28578,"##ffled":28579,"##uising":28580,"consultative":28581,"##guide":28582,"flop":28583,"kaitlyn":28584,"mergers":28585,"parenting":28586,"somber":28587,"##vron":28588,"supervise":28589,"vidhan":28590,"##imum":28591,"courtship":28592,"exemplified":28593,"harmonies":28594,"medallist":28595,"refining":28596,"##rrow":28597,"##ка":28598,"amara":28599,"##hum":28600,"780":28601,"goalscorer":28602,"sited":28603,"overshadowed":28604,"rohan":28605,"displeasure":28606,"secretive":28607,"multiplied":28608,"osman":28609,"##orth":28610,"engravings":28611,"padre":28612,"##kali":28613,"##veda":28614,"miniatures":28615,"mis":28616,"##yala":28617,"clap":28618,"pali":28619,"rook":28620,"##cana":28621,"1692":28622,"57th":28623,"antennae":28624,"astro":28625,"oskar":28626,"1628":28627,"bulldog":28628,"crotch":28629,"hackett":28630,"yucatan":28631,"##sure":28632,"amplifiers":28633,"brno":28634,"ferrara":28635,"migrating":28636,"##gree":28637,"thanking":28638,"turing":28639,"##eza":28640,"mccann":28641,"ting":28642,"andersson":28643,"onslaught":28644,"gaines":28645,"ganga":28646,"incense":28647,"standardization":28648,"##mation":28649,"sentai":28650,"scuba":28651,"stuffing":28652,"turquoise":28653,"waivers":28654,"alloys":28655,"##vitt":28656,"regaining":28657,"vaults":28658,"##clops":28659,"##gizing":28660,"digger":28661,"furry":28662,"memorabilia":28663,"probing":28664,"##iad":28665,"payton":28666,"rec":28667,"deutschland":28668,"filippo":28669,"opaque":28670,"seamen":28671,"zenith":28672,"afrikaans":28673,"##filtration":28674,"disciplined":28675,"inspirational":28676,"##merie":28677,"banco":28678,"confuse":28679,"grafton":28680,"tod":28681,"##dgets":28682,"championed":28683,"simi":28684,"anomaly":28685,"biplane":28686,"##ceptive":28687,"electrode":28688,"##para":28689,"1697":28690,"cleavage":28691,"crossbow":28692,"swirl":28693,"informant":28694,"##lars":28695,"##osta":28696,"afi":28697,"bonfire":28698,"spec":28699,"##oux":28700,"lakeside":28701,"slump":28702,"##culus":28703,"##lais":28704,"##qvist":28705,"##rrigan":28706,"1016":28707,"facades":28708,"borg":28709,"inwardly":28710,"cervical":28711,"xl":28712,"pointedly":28713,"050":28714,"stabilization":28715,"##odon":28716,"chests":28717,"1699":28718,"hacked":28719,"ctv":28720,"orthogonal":28721,"suzy":28722,"##lastic":28723,"gaulle":28724,"jacobite":28725,"rearview":28726,"##cam":28727,"##erted":28728,"ashby":28729,"##drik":28730,"##igate":28731,"##mise":28732,"##zbek":28733,"affectionately":28734,"canine":28735,"disperse":28736,"latham":28737,"##istles":28738,"##ivar":28739,"spielberg":28740,"##orin":28741,"##idium":28742,"ezekiel":28743,"cid":28744,"##sg":28745,"durga":28746,"middletown":28747,"##cina":28748,"customized":28749,"frontiers":28750,"harden":28751,"##etano":28752,"##zzy":28753,"1604":28754,"bolsheviks":28755,"##66":28756,"coloration":28757,"yoko":28758,"##bedo":28759,"briefs":28760,"slabs":28761,"debra":28762,"liquidation":28763,"plumage":28764,"##oin":28765,"blossoms":28766,"dementia":28767,"subsidy":28768,"1611":28769,"proctor":28770,"relational":28771,"jerseys":28772,"parochial":28773,"ter":28774,"##ici":28775,"esa":28776,"peshawar":28777,"cavalier":28778,"loren":28779,"cpi":28780,"idiots":28781,"shamrock":28782,"1646":28783,"dutton":28784,"malabar":28785,"mustache":28786,"##endez":28787,"##ocytes":28788,"referencing":28789,"terminates":28790,"marche":28791,"yarmouth":28792,"##sop":28793,"acton":28794,"mated":28795,"seton":28796,"subtly":28797,"baptised":28798,"beige":28799,"extremes":28800,"jolted":28801,"kristina":28802,"telecast":28803,"##actic":28804,"safeguard":28805,"waldo":28806,"##baldi":28807,"##bular":28808,"endeavors":28809,"sloppy":28810,"subterranean":28811,"##ensburg":28812,"##itung":28813,"delicately":28814,"pigment":28815,"tq":28816,"##scu":28817,"1626":28818,"##ound":28819,"collisions":28820,"coveted":28821,"herds":28822,"##personal":28823,"##meister":28824,"##nberger":28825,"chopra":28826,"##ricting":28827,"abnormalities":28828,"defective":28829,"galician":28830,"lucie":28831,"##dilly":28832,"alligator":28833,"likened":28834,"##genase":28835,"burundi":28836,"clears":28837,"complexion":28838,"derelict":28839,"deafening":28840,"diablo":28841,"fingered":28842,"champaign":28843,"dogg":28844,"enlist":28845,"isotope":28846,"labeling":28847,"mrna":28848,"##erre":28849,"brilliance":28850,"marvelous":28851,"##ayo":28852,"1652":28853,"crawley":28854,"ether":28855,"footed":28856,"dwellers":28857,"deserts":28858,"hamish":28859,"rubs":28860,"warlock":28861,"skimmed":28862,"##lizer":28863,"870":28864,"buick":28865,"embark":28866,"heraldic":28867,"irregularities":28868,"##ajan":28869,"kiara":28870,"##kulam":28871,"##ieg":28872,"antigen":28873,"kowalski":28874,"##lge":28875,"oakley":28876,"visitation":28877,"##mbit":28878,"vt":28879,"##suit":28880,"1570":28881,"murderers":28882,"##miento":28883,"##rites":28884,"chimneys":28885,"##sling":28886,"condemn":28887,"custer":28888,"exchequer":28889,"havre":28890,"##ghi":28891,"fluctuations":28892,"##rations":28893,"dfb":28894,"hendricks":28895,"vaccines":28896,"##tarian":28897,"nietzsche":28898,"biking":28899,"juicy":28900,"##duced":28901,"brooding":28902,"scrolling":28903,"selangor":28904,"##ragan":28905,"352":28906,"annum":28907,"boomed":28908,"seminole":28909,"sugarcane":28910,"##dna":28911,"departmental":28912,"dismissing":28913,"innsbruck":28914,"arteries":28915,"ashok":28916,"batavia":28917,"daze":28918,"kun":28919,"overtook":28920,"##rga":28921,"##tlan":28922,"beheaded":28923,"gaddafi":28924,"holm":28925,"electronically":28926,"faulty":28927,"galilee":28928,"fractures":28929,"kobayashi":28930,"##lized":28931,"gunmen":28932,"magma":28933,"aramaic":28934,"mala":28935,"eastenders":28936,"inference":28937,"messengers":28938,"bf":28939,"##qu":28940,"407":28941,"bathrooms":28942,"##vere":28943,"1658":28944,"flashbacks":28945,"ideally":28946,"misunderstood":28947,"##jali":28948,"##weather":28949,"mendez":28950,"##grounds":28951,"505":28952,"uncanny":28953,"##iii":28954,"1709":28955,"friendships":28956,"##nbc":28957,"sacrament":28958,"accommodated":28959,"reiterated":28960,"logistical":28961,"pebbles":28962,"thumped":28963,"##escence":28964,"administering":28965,"decrees":28966,"drafts":28967,"##flight":28968,"##cased":28969,"##tula":28970,"futuristic":28971,"picket":28972,"intimidation":28973,"winthrop":28974,"##fahan":28975,"interfered":28976,"339":28977,"afar":28978,"francoise":28979,"morally":28980,"uta":28981,"cochin":28982,"croft":28983,"dwarfs":28984,"##bruck":28985,"##dents":28986,"##nami":28987,"biker":28988,"##hner":28989,"##meral":28990,"nano":28991,"##isen":28992,"##ometric":28993,"##pres":28994,"##ан":28995,"brightened":28996,"meek":28997,"parcels":28998,"securely":28999,"gunners":29000,"##jhl":29001,"##zko":29002,"agile":29003,"hysteria":29004,"##lten":29005,"##rcus":29006,"bukit":29007,"champs":29008,"chevy":29009,"cuckoo":29010,"leith":29011,"sadler":29012,"theologians":29013,"welded":29014,"##section":29015,"1663":29016,"jj":29017,"plurality":29018,"xander":29019,"##rooms":29020,"##formed":29021,"shredded":29022,"temps":29023,"intimately":29024,"pau":29025,"tormented":29026,"##lok":29027,"##stellar":29028,"1618":29029,"charred":29030,"ems":29031,"essen":29032,"##mmel":29033,"alarms":29034,"spraying":29035,"ascot":29036,"blooms":29037,"twinkle":29038,"##abia":29039,"##apes":29040,"internment":29041,"obsidian":29042,"##chaft":29043,"snoop":29044,"##dav":29045,"##ooping":29046,"malibu":29047,"##tension":29048,"quiver":29049,"##itia":29050,"hays":29051,"mcintosh":29052,"travers":29053,"walsall":29054,"##ffie":29055,"1623":29056,"beverley":29057,"schwarz":29058,"plunging":29059,"structurally":29060,"m3":29061,"rosenthal":29062,"vikram":29063,"##tsk":29064,"770":29065,"ghz":29066,"##onda":29067,"##tiv":29068,"chalmers":29069,"groningen":29070,"pew":29071,"reckon":29072,"unicef":29073,"##rvis":29074,"55th":29075,"##gni":29076,"1651":29077,"sulawesi":29078,"avila":29079,"cai":29080,"metaphysical":29081,"screwing":29082,"turbulence":29083,"##mberg":29084,"augusto":29085,"samba":29086,"56th":29087,"baffled":29088,"momentary":29089,"toxin":29090,"##urian":29091,"##wani":29092,"aachen":29093,"condoms":29094,"dali":29095,"steppe":29096,"##3d":29097,"##app":29098,"##oed":29099,"##year":29100,"adolescence":29101,"dauphin":29102,"electrically":29103,"inaccessible":29104,"microscopy":29105,"nikita":29106,"##ega":29107,"atv":29108,"##cel":29109,"##enter":29110,"##oles":29111,"##oteric":29112,"##ы":29113,"accountants":29114,"punishments":29115,"wrongly":29116,"bribes":29117,"adventurous":29118,"clinch":29119,"flinders":29120,"southland":29121,"##hem":29122,"##kata":29123,"gough":29124,"##ciency":29125,"lads":29126,"soared":29127,"##ה":29128,"undergoes":29129,"deformation":29130,"outlawed":29131,"rubbish":29132,"##arus":29133,"##mussen":29134,"##nidae":29135,"##rzburg":29136,"arcs":29137,"##ingdon":29138,"##tituted":29139,"1695":29140,"wheelbase":29141,"wheeling":29142,"bombardier":29143,"campground":29144,"zebra":29145,"##lices":29146,"##oj":29147,"##bain":29148,"lullaby":29149,"##ecure":29150,"donetsk":29151,"wylie":29152,"grenada":29153,"##arding":29154,"##ης":29155,"squinting":29156,"eireann":29157,"opposes":29158,"##andra":29159,"maximal":29160,"runes":29161,"##broken":29162,"##cuting":29163,"##iface":29164,"##ror":29165,"##rosis":29166,"additive":29167,"britney":29168,"adultery":29169,"triggering":29170,"##drome":29171,"detrimental":29172,"aarhus":29173,"containment":29174,"jc":29175,"swapped":29176,"vichy":29177,"##ioms":29178,"madly":29179,"##oric":29180,"##rag":29181,"brant":29182,"##ckey":29183,"##trix":29184,"1560":29185,"1612":29186,"broughton":29187,"rustling":29188,"##stems":29189,"##uder":29190,"asbestos":29191,"mentoring":29192,"##nivorous":29193,"finley":29194,"leaps":29195,"##isan":29196,"apical":29197,"pry":29198,"slits":29199,"substitutes":29200,"##dict":29201,"intuitive":29202,"fantasia":29203,"insistent":29204,"unreasonable":29205,"##igen":29206,"##vna":29207,"domed":29208,"hannover":29209,"margot":29210,"ponder":29211,"##zziness":29212,"impromptu":29213,"jian":29214,"lc":29215,"rampage":29216,"stemming":29217,"##eft":29218,"andrey":29219,"gerais":29220,"whichever":29221,"amnesia":29222,"appropriated":29223,"anzac":29224,"clicks":29225,"modifying":29226,"ultimatum":29227,"cambrian":29228,"maids":29229,"verve":29230,"yellowstone":29231,"##mbs":29232,"conservatoire":29233,"##scribe":29234,"adherence":29235,"dinners":29236,"spectra":29237,"imperfect":29238,"mysteriously":29239,"sidekick":29240,"tatar":29241,"tuba":29242,"##aks":29243,"##ifolia":29244,"distrust":29245,"##athan":29246,"##zle":29247,"c2":29248,"ronin":29249,"zac":29250,"##pse":29251,"celaena":29252,"instrumentalist":29253,"scents":29254,"skopje":29255,"##mbling":29256,"comical":29257,"compensated":29258,"vidal":29259,"condor":29260,"intersect":29261,"jingle":29262,"wavelengths":29263,"##urrent":29264,"mcqueen":29265,"##izzly":29266,"carp":29267,"weasel":29268,"422":29269,"kanye":29270,"militias":29271,"postdoctoral":29272,"eugen":29273,"gunslinger":29274,"##ɛ":29275,"faux":29276,"hospice":29277,"##for":29278,"appalled":29279,"derivation":29280,"dwarves":29281,"##elis":29282,"dilapidated":29283,"##folk":29284,"astoria":29285,"philology":29286,"##lwyn":29287,"##otho":29288,"##saka":29289,"inducing":29290,"philanthropy":29291,"##bf":29292,"##itative":29293,"geek":29294,"markedly":29295,"sql":29296,"##yce":29297,"bessie":29298,"indices":29299,"rn":29300,"##flict":29301,"495":29302,"frowns":29303,"resolving":29304,"weightlifting":29305,"tugs":29306,"cleric":29307,"contentious":29308,"1653":29309,"mania":29310,"rms":29311,"##miya":29312,"##reate":29313,"##ruck":29314,"##tucket":29315,"bien":29316,"eels":29317,"marek":29318,"##ayton":29319,"##cence":29320,"discreet":29321,"unofficially":29322,"##ife":29323,"leaks":29324,"##bber":29325,"1705":29326,"332":29327,"dung":29328,"compressor":29329,"hillsborough":29330,"pandit":29331,"shillings":29332,"distal":29333,"##skin":29334,"381":29335,"##tat":29336,"##you":29337,"nosed":29338,"##nir":29339,"mangrove":29340,"undeveloped":29341,"##idia":29342,"textures":29343,"##inho":29344,"##500":29345,"##rise":29346,"ae":29347,"irritating":29348,"nay":29349,"amazingly":29350,"bancroft":29351,"apologetic":29352,"compassionate":29353,"kata":29354,"symphonies":29355,"##lovic":29356,"airspace":29357,"##lch":29358,"930":29359,"gifford":29360,"precautions":29361,"fulfillment":29362,"sevilla":29363,"vulgar":29364,"martinique":29365,"##urities":29366,"looting":29367,"piccolo":29368,"tidy":29369,"##dermott":29370,"quadrant":29371,"armchair":29372,"incomes":29373,"mathematicians":29374,"stampede":29375,"nilsson":29376,"##inking":29377,"##scan":29378,"foo":29379,"quarterfinal":29380,"##ostal":29381,"shang":29382,"shouldered":29383,"squirrels":29384,"##owe":29385,"344":29386,"vinegar":29387,"##bner":29388,"##rchy":29389,"##systems":29390,"delaying":29391,"##trics":29392,"ars":29393,"dwyer":29394,"rhapsody":29395,"sponsoring":29396,"##gration":29397,"bipolar":29398,"cinder":29399,"starters":29400,"##olio":29401,"##urst":29402,"421":29403,"signage":29404,"##nty":29405,"aground":29406,"figurative":29407,"mons":29408,"acquaintances":29409,"duets":29410,"erroneously":29411,"soyuz":29412,"elliptic":29413,"recreated":29414,"##cultural":29415,"##quette":29416,"##ssed":29417,"##tma":29418,"##zcz":29419,"moderator":29420,"scares":29421,"##itaire":29422,"##stones":29423,"##udence":29424,"juniper":29425,"sighting":29426,"##just":29427,"##nsen":29428,"britten":29429,"calabria":29430,"ry":29431,"bop":29432,"cramer":29433,"forsyth":29434,"stillness":29435,"##л":29436,"airmen":29437,"gathers":29438,"unfit":29439,"##umber":29440,"##upt":29441,"taunting":29442,"##rip":29443,"seeker":29444,"streamlined":29445,"##bution":29446,"holster":29447,"schumann":29448,"tread":29449,"vox":29450,"##gano":29451,"##onzo":29452,"strive":29453,"dil":29454,"reforming":29455,"covent":29456,"newbury":29457,"predicting":29458,"##orro":29459,"decorate":29460,"tre":29461,"##puted":29462,"andover":29463,"ie":29464,"asahi":29465,"dept":29466,"dunkirk":29467,"gills":29468,"##tori":29469,"buren":29470,"huskies":29471,"##stis":29472,"##stov":29473,"abstracts":29474,"bets":29475,"loosen":29476,"##opa":29477,"1682":29478,"yearning":29479,"##glio":29480,"##sir":29481,"berman":29482,"effortlessly":29483,"enamel":29484,"napoli":29485,"persist":29486,"##peration":29487,"##uez":29488,"attache":29489,"elisa":29490,"b1":29491,"invitations":29492,"##kic":29493,"accelerating":29494,"reindeer":29495,"boardwalk":29496,"clutches":29497,"nelly":29498,"polka":29499,"starbucks":29500,"##kei":29501,"adamant":29502,"huey":29503,"lough":29504,"unbroken":29505,"adventurer":29506,"embroidery":29507,"inspecting":29508,"stanza":29509,"##ducted":29510,"naia":29511,"taluka":29512,"##pone":29513,"##roids":29514,"chases":29515,"deprivation":29516,"florian":29517,"##jing":29518,"##ppet":29519,"earthly":29520,"##lib":29521,"##ssee":29522,"colossal":29523,"foreigner":29524,"vet":29525,"freaks":29526,"patrice":29527,"rosewood":29528,"triassic":29529,"upstate":29530,"##pkins":29531,"dominates":29532,"ata":29533,"chants":29534,"ks":29535,"vo":29536,"##400":29537,"##bley":29538,"##raya":29539,"##rmed":29540,"555":29541,"agra":29542,"infiltrate":29543,"##ailing":29544,"##ilation":29545,"##tzer":29546,"##uppe":29547,"##werk":29548,"binoculars":29549,"enthusiast":29550,"fujian":29551,"squeak":29552,"##avs":29553,"abolitionist":29554,"almeida":29555,"boredom":29556,"hampstead":29557,"marsden":29558,"rations":29559,"##ands":29560,"inflated":29561,"334":29562,"bonuses":29563,"rosalie":29564,"patna":29565,"##rco":29566,"329":29567,"detachments":29568,"penitentiary":29569,"54th":29570,"flourishing":29571,"woolf":29572,"##dion":29573,"##etched":29574,"papyrus":29575,"##lster":29576,"##nsor":29577,"##toy":29578,"bobbed":29579,"dismounted":29580,"endelle":29581,"inhuman":29582,"motorola":29583,"tbs":29584,"wince":29585,"wreath":29586,"##ticus":29587,"hideout":29588,"inspections":29589,"sanjay":29590,"disgrace":29591,"infused":29592,"pudding":29593,"stalks":29594,"##urbed":29595,"arsenic":29596,"leases":29597,"##hyl":29598,"##rrard":29599,"collarbone":29600,"##waite":29601,"##wil":29602,"dowry":29603,"##bant":29604,"##edance":29605,"genealogical":29606,"nitrate":29607,"salamanca":29608,"scandals":29609,"thyroid":29610,"necessitated":29611,"##!":29612,"##\"":29613,"###":29614,"##$":29615,"##%":29616,"##&":29617,"##'":29618,"##(":29619,"##)":29620,"##*":29621,"##+":29622,"##,":29623,"##-":29624,"##.":29625,"##/":29626,"##:":29627,"##;":29628,"##<":29629,"##=":29630,"##>":29631,"##?":29632,"##@":29633,"##[":29634,"##\\":29635,"##]":29636,"##^":29637,"##_":29638,"##`":29639,"##{":29640,"##|":29641,"##}":29642,"##~":29643,"##¡":29644,"##¢":29645,"##£":29646,"##¤":29647,"##¥":29648,"##¦":29649,"##§":29650,"##¨":29651,"##©":29652,"##ª":29653,"##«":29654,"##¬":29655,"##®":29656,"##±":29657,"##´":29658,"##µ":29659,"##¶":29660,"##·":29661,"##º":29662,"##»":29663,"##¼":29664,"##¾":29665,"##¿":29666,"##æ":29667,"##ð":29668,"##÷":29669,"##þ":29670,"##đ":29671,"##ħ":29672,"##ŋ":29673,"##œ":29674,"##ƒ":29675,"##ɐ":29676,"##ɑ":29677,"##ɒ":29678,"##ɔ":29679,"##ɕ":29680,"##ə":29681,"##ɡ":29682,"##ɣ":29683,"##ɨ":29684,"##ɪ":29685,"##ɫ":29686,"##ɬ":29687,"##ɯ":29688,"##ɲ":29689,"##ɴ":29690,"##ɹ":29691,"##ɾ":29692,"##ʀ":29693,"##ʁ":29694,"##ʂ":29695,"##ʃ":29696,"##ʉ":29697,"##ʊ":29698,"##ʋ":29699,"##ʌ":29700,"##ʎ":29701,"##ʐ":29702,"##ʑ":29703,"##ʒ":29704,"##ʔ":29705,"##ʰ":29706,"##ʲ":29707,"##ʳ":29708,"##ʷ":29709,"##ʸ":29710,"##ʻ":29711,"##ʼ":29712,"##ʾ":29713,"##ʿ":29714,"##ˈ":29715,"##ˡ":29716,"##ˢ":29717,"##ˣ":29718,"##ˤ":29719,"##β":29720,"##γ":29721,"##δ":29722,"##ε":29723,"##ζ":29724,"##θ":29725,"##κ":29726,"##λ":29727,"##μ":29728,"##ξ":29729,"##ο":29730,"##π":29731,"##ρ":29732,"##σ":29733,"##τ":29734,"##υ":29735,"##φ":29736,"##χ":29737,"##ψ":29738,"##ω":29739,"##б":29740,"##г":29741,"##д":29742,"##ж":29743,"##з":29744,"##м":29745,"##п":29746,"##с":29747,"##у":29748,"##ф":29749,"##х":29750,"##ц":29751,"##ч":29752,"##ш":29753,"##щ":29754,"##ъ":29755,"##э":29756,"##ю":29757,"##ђ":29758,"##є":29759,"##і":29760,"##ј":29761,"##љ":29762,"##њ":29763,"##ћ":29764,"##ӏ":29765,"##ա":29766,"##բ":29767,"##գ":29768,"##դ":29769,"##ե":29770,"##թ":29771,"##ի":29772,"##լ":29773,"##կ":29774,"##հ":29775,"##մ":29776,"##յ":29777,"##ն":29778,"##ո":29779,"##պ":29780,"##ս":29781,"##վ":29782,"##տ":29783,"##ր":29784,"##ւ":29785,"##ք":29786,"##־":29787,"##א":29788,"##ב":29789,"##ג":29790,"##ד":29791,"##ו":29792,"##ז":29793,"##ח":29794,"##ט":29795,"##י":29796,"##ך":29797,"##כ":29798,"##ל":29799,"##ם":29800,"##מ":29801,"##ן":29802,"##נ":29803,"##ס":29804,"##ע":29805,"##ף":29806,"##פ":29807,"##ץ":29808,"##צ":29809,"##ק":29810,"##ר":29811,"##ש":29812,"##ת":29813,"##،":29814,"##ء":29815,"##ب":29816,"##ت":29817,"##ث":29818,"##ج":29819,"##ح":29820,"##خ":29821,"##ذ":29822,"##ز":29823,"##س":29824,"##ش":29825,"##ص":29826,"##ض":29827,"##ط":29828,"##ظ":29829,"##ع":29830,"##غ":29831,"##ـ":29832,"##ف":29833,"##ق":29834,"##ك":29835,"##و":29836,"##ى":29837,"##ٹ":29838,"##پ":29839,"##چ":29840,"##ک":29841,"##گ":29842,"##ں":29843,"##ھ":29844,"##ہ":29845,"##ے":29846,"##अ":29847,"##आ":29848,"##उ":29849,"##ए":29850,"##क":29851,"##ख":29852,"##ग":29853,"##च":29854,"##ज":29855,"##ट":29856,"##ड":29857,"##ण":29858,"##त":29859,"##थ":29860,"##द":29861,"##ध":29862,"##न":29863,"##प":29864,"##ब":29865,"##भ":29866,"##म":29867,"##य":29868,"##र":29869,"##ल":29870,"##व":29871,"##श":29872,"##ष":29873,"##स":29874,"##ह":29875,"##ा":29876,"##ि":29877,"##ी":29878,"##ो":29879,"##।":29880,"##॥":29881,"##ং":29882,"##অ":29883,"##আ":29884,"##ই":29885,"##উ":29886,"##এ":29887,"##ও":29888,"##ক":29889,"##খ":29890,"##গ":29891,"##চ":29892,"##ছ":29893,"##জ":29894,"##ট":29895,"##ড":29896,"##ণ":29897,"##ত":29898,"##থ":29899,"##দ":29900,"##ধ":29901,"##ন":29902,"##প":29903,"##ব":29904,"##ভ":29905,"##ম":29906,"##য":29907,"##র":29908,"##ল":29909,"##শ":29910,"##ষ":29911,"##স":29912,"##হ":29913,"##া":29914,"##ি":29915,"##ী":29916,"##ে":29917,"##க":29918,"##ச":29919,"##ட":29920,"##த":29921,"##ந":29922,"##ன":29923,"##ப":29924,"##ம":29925,"##ய":29926,"##ர":29927,"##ல":29928,"##ள":29929,"##வ":29930,"##ா":29931,"##ி":29932,"##ு":29933,"##ே":29934,"##ை":29935,"##ನ":29936,"##ರ":29937,"##ಾ":29938,"##ක":29939,"##ය":29940,"##ර":29941,"##ල":29942,"##ව":29943,"##ා":29944,"##ก":29945,"##ง":29946,"##ต":29947,"##ท":29948,"##น":29949,"##พ":29950,"##ม":29951,"##ย":29952,"##ร":29953,"##ล":29954,"##ว":29955,"##ส":29956,"##อ":29957,"##า":29958,"##เ":29959,"##་":29960,"##།":29961,"##ག":29962,"##ང":29963,"##ད":29964,"##ན":29965,"##པ":29966,"##བ":29967,"##མ":29968,"##འ":29969,"##ར":29970,"##ལ":29971,"##ས":29972,"##မ":29973,"##ა":29974,"##ბ":29975,"##გ":29976,"##დ":29977,"##ე":29978,"##ვ":29979,"##თ":29980,"##ი":29981,"##კ":29982,"##ლ":29983,"##მ":29984,"##ნ":29985,"##ო":29986,"##რ":29987,"##ს":29988,"##ტ":29989,"##უ":29990,"##ᄀ":29991,"##ᄂ":29992,"##ᄃ":29993,"##ᄅ":29994,"##ᄆ":29995,"##ᄇ":29996,"##ᄉ":29997,"##ᄊ":29998,"##ᄋ":29999,"##ᄌ":30000,"##ᄎ":30001,"##ᄏ":30002,"##ᄐ":30003,"##ᄑ":30004,"##ᄒ":30005,"##ᅡ":30006,"##ᅢ":30007,"##ᅥ":30008,"##ᅦ":30009,"##ᅧ":30010,"##ᅩ":30011,"##ᅪ":30012,"##ᅭ":30013,"##ᅮ":30014,"##ᅯ":30015,"##ᅲ":30016,"##ᅳ":30017,"##ᅴ":30018,"##ᅵ":30019,"##ᆨ":30020,"##ᆫ":30021,"##ᆯ":30022,"##ᆷ":30023,"##ᆸ":30024,"##ᆼ":30025,"##ᴬ":30026,"##ᴮ":30027,"##ᴰ":30028,"##ᴵ":30029,"##ᴺ":30030,"##ᵀ":30031,"##ᵃ":30032,"##ᵇ":30033,"##ᵈ":30034,"##ᵉ":30035,"##ᵍ":30036,"##ᵏ":30037,"##ᵐ":30038,"##ᵒ":30039,"##ᵖ":30040,"##ᵗ":30041,"##ᵘ":30042,"##ᵣ":30043,"##ᵤ":30044,"##ᵥ":30045,"##ᶜ":30046,"##ᶠ":30047,"##‐":30048,"##‑":30049,"##‒":30050,"##–":30051,"##—":30052,"##―":30053,"##‖":30054,"##‘":30055,"##’":30056,"##‚":30057,"##“":30058,"##”":30059,"##„":30060,"##†":30061,"##‡":30062,"##•":30063,"##…":30064,"##‰":30065,"##′":30066,"##″":30067,"##›":30068,"##‿":30069,"##⁄":30070,"##⁰":30071,"##ⁱ":30072,"##⁴":30073,"##⁵":30074,"##⁶":30075,"##⁷":30076,"##⁸":30077,"##⁹":30078,"##⁻":30079,"##ⁿ":30080,"##₅":30081,"##₆":30082,"##₇":30083,"##₈":30084,"##₉":30085,"##₊":30086,"##₍":30087,"##₎":30088,"##ₐ":30089,"##ₑ":30090,"##ₒ":30091,"##ₓ":30092,"##ₕ":30093,"##ₖ":30094,"##ₗ":30095,"##ₘ":30096,"##ₚ":30097,"##ₛ":30098,"##ₜ":30099,"##₤":30100,"##₩":30101,"##€":30102,"##₱":30103,"##₹":30104,"##ℓ":30105,"##№":30106,"##ℝ":30107,"##™":30108,"##⅓":30109,"##⅔":30110,"##←":30111,"##↑":30112,"##→":30113,"##↓":30114,"##↔":30115,"##↦":30116,"##⇄":30117,"##⇌":30118,"##⇒":30119,"##∂":30120,"##∅":30121,"##∆":30122,"##∇":30123,"##∈":30124,"##∗":30125,"##∘":30126,"##√":30127,"##∞":30128,"##∧":30129,"##∨":30130,"##∩":30131,"##∪":30132,"##≈":30133,"##≡":30134,"##≤":30135,"##≥":30136,"##⊂":30137,"##⊆":30138,"##⊕":30139,"##⊗":30140,"##⋅":30141,"##─":30142,"##│":30143,"##■":30144,"##▪":30145,"##●":30146,"##★":30147,"##☆":30148,"##☉":30149,"##♠":30150,"##♣":30151,"##♥":30152,"##♦":30153,"##♯":30154,"##⟨":30155,"##⟩":30156,"##ⱼ":30157,"##⺩":30158,"##⺼":30159,"##⽥":30160,"##、":30161,"##。":30162,"##〈":30163,"##〉":30164,"##《":30165,"##》":30166,"##「":30167,"##」":30168,"##『":30169,"##』":30170,"##〜":30171,"##あ":30172,"##い":30173,"##う":30174,"##え":30175,"##お":30176,"##か":30177,"##き":30178,"##く":30179,"##け":30180,"##こ":30181,"##さ":30182,"##し":30183,"##す":30184,"##せ":30185,"##そ":30186,"##た":30187,"##ち":30188,"##っ":30189,"##つ":30190,"##て":30191,"##と":30192,"##な":30193,"##に":30194,"##ぬ":30195,"##ね":30196,"##の":30197,"##は":30198,"##ひ":30199,"##ふ":30200,"##へ":30201,"##ほ":30202,"##ま":30203,"##み":30204,"##む":30205,"##め":30206,"##も":30207,"##や":30208,"##ゆ":30209,"##よ":30210,"##ら":30211,"##り":30212,"##る":30213,"##れ":30214,"##ろ":30215,"##を":30216,"##ん":30217,"##ァ":30218,"##ア":30219,"##ィ":30220,"##イ":30221,"##ウ":30222,"##ェ":30223,"##エ":30224,"##オ":30225,"##カ":30226,"##キ":30227,"##ク":30228,"##ケ":30229,"##コ":30230,"##サ":30231,"##シ":30232,"##ス":30233,"##セ":30234,"##タ":30235,"##チ":30236,"##ッ":30237,"##ツ":30238,"##テ":30239,"##ト":30240,"##ナ":30241,"##ニ":30242,"##ノ":30243,"##ハ":30244,"##ヒ":30245,"##フ":30246,"##ヘ":30247,"##ホ":30248,"##マ":30249,"##ミ":30250,"##ム":30251,"##メ":30252,"##モ":30253,"##ャ":30254,"##ュ":30255,"##ョ":30256,"##ラ":30257,"##リ":30258,"##ル":30259,"##レ":30260,"##ロ":30261,"##ワ":30262,"##ン":30263,"##・":30264,"##ー":30265,"##一":30266,"##三":30267,"##上":30268,"##下":30269,"##不":30270,"##世":30271,"##中":30272,"##主":30273,"##久":30274,"##之":30275,"##也":30276,"##事":30277,"##二":30278,"##五":30279,"##井":30280,"##京":30281,"##人":30282,"##亻":30283,"##仁":30284,"##介":30285,"##代":30286,"##仮":30287,"##伊":30288,"##会":30289,"##佐":30290,"##侍":30291,"##保":30292,"##信":30293,"##健":30294,"##元":30295,"##光":30296,"##八":30297,"##公":30298,"##内":30299,"##出":30300,"##分":30301,"##前":30302,"##劉":30303,"##力":30304,"##加":30305,"##勝":30306,"##北":30307,"##区":30308,"##十":30309,"##千":30310,"##南":30311,"##博":30312,"##原":30313,"##口":30314,"##古":30315,"##史":30316,"##司":30317,"##合":30318,"##吉":30319,"##同":30320,"##名":30321,"##和":30322,"##囗":30323,"##四":30324,"##国":30325,"##國":30326,"##土":30327,"##地":30328,"##坂":30329,"##城":30330,"##堂":30331,"##場":30332,"##士":30333,"##夏":30334,"##外":30335,"##大":30336,"##天":30337,"##太":30338,"##夫":30339,"##奈":30340,"##女":30341,"##子":30342,"##学":30343,"##宀":30344,"##宇":30345,"##安":30346,"##宗":30347,"##定":30348,"##宣":30349,"##宮":30350,"##家":30351,"##宿":30352,"##寺":30353,"##將":30354,"##小":30355,"##尚":30356,"##山":30357,"##岡":30358,"##島":30359,"##崎":30360,"##川":30361,"##州":30362,"##巿":30363,"##帝":30364,"##平":30365,"##年":30366,"##幸":30367,"##广":30368,"##弘":30369,"##張":30370,"##彳":30371,"##後":30372,"##御":30373,"##德":30374,"##心":30375,"##忄":30376,"##志":30377,"##忠":30378,"##愛":30379,"##成":30380,"##我":30381,"##戦":30382,"##戸":30383,"##手":30384,"##扌":30385,"##政":30386,"##文":30387,"##新":30388,"##方":30389,"##日":30390,"##明":30391,"##星":30392,"##春":30393,"##昭":30394,"##智":30395,"##曲":30396,"##書":30397,"##月":30398,"##有":30399,"##朝":30400,"##木":30401,"##本":30402,"##李":30403,"##村":30404,"##東":30405,"##松":30406,"##林":30407,"##森":30408,"##楊":30409,"##樹":30410,"##橋":30411,"##歌":30412,"##止":30413,"##正":30414,"##武":30415,"##比":30416,"##氏":30417,"##民":30418,"##水":30419,"##氵":30420,"##氷":30421,"##永":30422,"##江":30423,"##沢":30424,"##河":30425,"##治":30426,"##法":30427,"##海":30428,"##清":30429,"##漢":30430,"##瀬":30431,"##火":30432,"##版":30433,"##犬":30434,"##王":30435,"##生":30436,"##田":30437,"##男":30438,"##疒":30439,"##発":30440,"##白":30441,"##的":30442,"##皇":30443,"##目":30444,"##相":30445,"##省":30446,"##真":30447,"##石":30448,"##示":30449,"##社":30450,"##神":30451,"##福":30452,"##禾":30453,"##秀":30454,"##秋":30455,"##空":30456,"##立":30457,"##章":30458,"##竹":30459,"##糹":30460,"##美":30461,"##義":30462,"##耳":30463,"##良":30464,"##艹":30465,"##花":30466,"##英":30467,"##華":30468,"##葉":30469,"##藤":30470,"##行":30471,"##街":30472,"##西":30473,"##見":30474,"##訁":30475,"##語":30476,"##谷":30477,"##貝":30478,"##貴":30479,"##車":30480,"##軍":30481,"##辶":30482,"##道":30483,"##郎":30484,"##郡":30485,"##部":30486,"##都":30487,"##里":30488,"##野":30489,"##金":30490,"##鈴":30491,"##镇":30492,"##長":30493,"##門":30494,"##間":30495,"##阝":30496,"##阿":30497,"##陳":30498,"##陽":30499,"##雄":30500,"##青":30501,"##面":30502,"##風":30503,"##食":30504,"##香":30505,"##馬":30506,"##高":30507,"##龍":30508,"##龸":30509,"##fi":30510,"##fl":30511,"##!":30512,"##(":30513,"##)":30514,"##,":30515,"##-":30516,"##.":30517,"##/":30518,"##:":30519,"##?":30520,"##~":30521}}} \ No newline at end of file diff --git a/bun.lock b/bun.lock deleted file mode 100644 index c31b3865..00000000 --- a/bun.lock +++ /dev/null @@ -1,2037 +0,0 @@ -{ - "lockfileVersion": 1, - "configVersion": 0, - "workspaces": { - "": { - "name": "@soulcraft/brainy", - "dependencies": { - "@aws-sdk/client-s3": "^3.540.0", - "@azure/identity": "^4.0.0", - "@azure/storage-blob": "^12.17.0", - "@google-cloud/storage": "^7.14.0", - "@msgpack/msgpack": "^3.1.2", - "@types/js-yaml": "^4.0.9", - "boxen": "^8.0.1", - "chalk": "^5.3.0", - "chardet": "^2.0.0", - "cli-table3": "^0.6.5", - "commander": "^11.1.0", - "csv-parse": "^6.1.0", - "exifr": "^7.1.3", - "inquirer": "^12.9.3", - "js-yaml": "^4.1.0", - "mammoth": "^1.11.0", - "mime": "^4.1.0", - "onnxruntime-web": "^1.22.0", - "ora": "^8.2.0", - "pdfjs-dist": "^4.0.379", - "probe-image-size": "^7.2.3", - "prompts": "^2.4.2", - "roaring-wasm": "^1.1.0", - "uuid": "^9.0.1", - "ws": "^8.18.3", - "xlsx": "^0.18.5", - }, - "devDependencies": { - "@rollup/plugin-commonjs": "^28.0.6", - "@rollup/plugin-node-resolve": "^16.0.1", - "@rollup/plugin-replace": "^6.0.2", - "@rollup/plugin-terser": "^0.4.4", - "@testcontainers/redis": "^11.5.1", - "@types/mime": "^3.0.4", - "@types/node": "^20.11.30", - "@types/probe-image-size": "^7.2.5", - "@types/uuid": "^10.0.0", - "@types/ws": "^8.18.1", - "@typescript-eslint/eslint-plugin": "^8.0.0", - "@typescript-eslint/parser": "^8.0.0", - "@vitest/coverage-v8": "^3.2.4", - "jspdf": "^3.0.3", - "minio": "^8.0.5", - "standard-version": "^9.5.0", - "testcontainers": "^11.5.1", - "tsx": "^4.19.2", - "typescript": "^5.4.5", - "vitest": "^3.2.4", - }, - }, - }, - "overrides": { - "boolean": "3.2.0", - }, - "packages": { - "@ampproject/remapping": ["@ampproject/remapping@2.3.0", "", { "dependencies": { "@jridgewell/gen-mapping": "^0.3.5", "@jridgewell/trace-mapping": "^0.3.24" } }, "sha512-30iZtAPgz+LTIYoeivqYo853f02jBYSd5uGnGpkFV0M3xOt9aN73erkgYAmZU43x4VfqcnLxW9Kpg3R5LC4YYw=="], - - "@aws-crypto/crc32": ["@aws-crypto/crc32@5.2.0", "", { "dependencies": { "@aws-crypto/util": "^5.2.0", "@aws-sdk/types": "^3.222.0", "tslib": "^2.6.2" } }, "sha512-nLbCWqQNgUiwwtFsen1AdzAtvuLRsQS8rYgMuxCrdKf9kOssamGLuPwyTY9wyYblNr9+1XM8v6zoDTPPSIeANg=="], - - "@aws-crypto/crc32c": ["@aws-crypto/crc32c@5.2.0", "", { "dependencies": { "@aws-crypto/util": "^5.2.0", "@aws-sdk/types": "^3.222.0", "tslib": "^2.6.2" } }, "sha512-+iWb8qaHLYKrNvGRbiYRHSdKRWhto5XlZUEBwDjYNf+ly5SVYG6zEoYIdxvf5R3zyeP16w4PLBn3rH1xc74Rag=="], - - "@aws-crypto/sha1-browser": ["@aws-crypto/sha1-browser@5.2.0", "", { "dependencies": { "@aws-crypto/supports-web-crypto": "^5.2.0", "@aws-crypto/util": "^5.2.0", "@aws-sdk/types": "^3.222.0", "@aws-sdk/util-locate-window": "^3.0.0", "@smithy/util-utf8": "^2.0.0", "tslib": "^2.6.2" } }, "sha512-OH6lveCFfcDjX4dbAvCFSYUjJZjDr/3XJ3xHtjn3Oj5b9RjojQo8npoLeA/bNwkOkrSQ0wgrHzXk4tDRxGKJeg=="], - - "@aws-crypto/sha256-browser": ["@aws-crypto/sha256-browser@5.2.0", "", { "dependencies": { "@aws-crypto/sha256-js": "^5.2.0", "@aws-crypto/supports-web-crypto": "^5.2.0", "@aws-crypto/util": "^5.2.0", "@aws-sdk/types": "^3.222.0", "@aws-sdk/util-locate-window": "^3.0.0", "@smithy/util-utf8": "^2.0.0", "tslib": "^2.6.2" } }, "sha512-AXfN/lGotSQwu6HNcEsIASo7kWXZ5HYWvfOmSNKDsEqC4OashTp8alTmaz+F7TC2L083SFv5RdB+qU3Vs1kZqw=="], - - "@aws-crypto/sha256-js": ["@aws-crypto/sha256-js@5.2.0", "", { "dependencies": { "@aws-crypto/util": "^5.2.0", "@aws-sdk/types": "^3.222.0", "tslib": "^2.6.2" } }, "sha512-FFQQyu7edu4ufvIZ+OadFpHHOt+eSTBaYaki44c+akjg7qZg9oOQeLlk77F6tSYqjDAFClrHJk9tMf0HdVyOvA=="], - - "@aws-crypto/supports-web-crypto": ["@aws-crypto/supports-web-crypto@5.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-iAvUotm021kM33eCdNfwIN//F77/IADDSs58i+MDaOqFrVjZo9bAal0NK7HurRuWLLpF1iLX7gbWrjHjeo+YFg=="], - - "@aws-crypto/util": ["@aws-crypto/util@5.2.0", "", { "dependencies": { "@aws-sdk/types": "^3.222.0", "@smithy/util-utf8": "^2.0.0", "tslib": "^2.6.2" } }, "sha512-4RkU9EsI6ZpBve5fseQlGNUWKMa1RLPQ1dnjnQoe07ldfIzcsGb5hC5W0Dm7u423KWzawlrpbjXBrXCEv9zazQ=="], - - "@aws-sdk/client-s3": ["@aws-sdk/client-s3@3.954.0", "", { "dependencies": { "@aws-crypto/sha1-browser": "5.2.0", "@aws-crypto/sha256-browser": "5.2.0", "@aws-crypto/sha256-js": "5.2.0", "@aws-sdk/core": "3.954.0", "@aws-sdk/credential-provider-node": "3.954.0", "@aws-sdk/middleware-bucket-endpoint": "3.953.0", "@aws-sdk/middleware-expect-continue": "3.953.0", "@aws-sdk/middleware-flexible-checksums": "3.954.0", "@aws-sdk/middleware-host-header": "3.953.0", "@aws-sdk/middleware-location-constraint": "3.953.0", "@aws-sdk/middleware-logger": "3.953.0", "@aws-sdk/middleware-recursion-detection": "3.953.0", "@aws-sdk/middleware-sdk-s3": "3.954.0", "@aws-sdk/middleware-ssec": "3.953.0", "@aws-sdk/middleware-user-agent": "3.954.0", "@aws-sdk/region-config-resolver": "3.953.0", "@aws-sdk/signature-v4-multi-region": "3.954.0", "@aws-sdk/types": "3.953.0", "@aws-sdk/util-endpoints": "3.953.0", "@aws-sdk/util-user-agent-browser": "3.953.0", "@aws-sdk/util-user-agent-node": "3.954.0", "@smithy/config-resolver": "^4.4.4", "@smithy/core": "^3.19.0", "@smithy/eventstream-serde-browser": "^4.2.6", "@smithy/eventstream-serde-config-resolver": "^4.3.6", "@smithy/eventstream-serde-node": "^4.2.6", "@smithy/fetch-http-handler": "^5.3.7", "@smithy/hash-blob-browser": "^4.2.7", "@smithy/hash-node": "^4.2.6", "@smithy/hash-stream-node": "^4.2.6", "@smithy/invalid-dependency": "^4.2.6", "@smithy/md5-js": "^4.2.6", "@smithy/middleware-content-length": "^4.2.6", "@smithy/middleware-endpoint": "^4.4.0", "@smithy/middleware-retry": "^4.4.16", "@smithy/middleware-serde": "^4.2.7", "@smithy/middleware-stack": "^4.2.6", "@smithy/node-config-provider": "^4.3.6", "@smithy/node-http-handler": "^4.4.6", "@smithy/protocol-http": "^5.3.6", "@smithy/smithy-client": "^4.10.1", "@smithy/types": "^4.10.0", "@smithy/url-parser": "^4.2.6", "@smithy/util-base64": "^4.3.0", "@smithy/util-body-length-browser": "^4.2.0", "@smithy/util-body-length-node": "^4.2.1", "@smithy/util-defaults-mode-browser": "^4.3.15", "@smithy/util-defaults-mode-node": "^4.2.18", "@smithy/util-endpoints": "^3.2.6", "@smithy/util-middleware": "^4.2.6", "@smithy/util-retry": "^4.2.6", "@smithy/util-stream": "^4.5.7", "@smithy/util-utf8": "^4.2.0", "@smithy/util-waiter": "^4.2.6", "tslib": "^2.6.2" } }, "sha512-DoeySsljzjuWRzqoETLszHGKKOOWlzuGZh3oAF7TkYRsrwbuYYmttrWomb9koogaF0S5YSPwCMCUbKbpF0lbTA=="], - - "@aws-sdk/client-sso": ["@aws-sdk/client-sso@3.954.0", "", { "dependencies": { "@aws-crypto/sha256-browser": "5.2.0", "@aws-crypto/sha256-js": "5.2.0", "@aws-sdk/core": "3.954.0", "@aws-sdk/middleware-host-header": "3.953.0", "@aws-sdk/middleware-logger": "3.953.0", "@aws-sdk/middleware-recursion-detection": "3.953.0", "@aws-sdk/middleware-user-agent": "3.954.0", "@aws-sdk/region-config-resolver": "3.953.0", "@aws-sdk/types": "3.953.0", "@aws-sdk/util-endpoints": "3.953.0", "@aws-sdk/util-user-agent-browser": "3.953.0", "@aws-sdk/util-user-agent-node": "3.954.0", "@smithy/config-resolver": "^4.4.4", "@smithy/core": "^3.19.0", "@smithy/fetch-http-handler": "^5.3.7", "@smithy/hash-node": "^4.2.6", "@smithy/invalid-dependency": "^4.2.6", "@smithy/middleware-content-length": "^4.2.6", "@smithy/middleware-endpoint": "^4.4.0", "@smithy/middleware-retry": "^4.4.16", "@smithy/middleware-serde": "^4.2.7", "@smithy/middleware-stack": "^4.2.6", "@smithy/node-config-provider": "^4.3.6", "@smithy/node-http-handler": "^4.4.6", "@smithy/protocol-http": "^5.3.6", "@smithy/smithy-client": "^4.10.1", "@smithy/types": "^4.10.0", "@smithy/url-parser": "^4.2.6", "@smithy/util-base64": "^4.3.0", "@smithy/util-body-length-browser": "^4.2.0", "@smithy/util-body-length-node": "^4.2.1", "@smithy/util-defaults-mode-browser": "^4.3.15", "@smithy/util-defaults-mode-node": "^4.2.18", "@smithy/util-endpoints": "^3.2.6", "@smithy/util-middleware": "^4.2.6", "@smithy/util-retry": "^4.2.6", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-FVyMAvlFhLK68DHWB1lSkCRTm25xl38bIZDd+jKt5+yDolCrG5+n9aIN8AA8jNO1HNGhZuMjSIQm9r5rGmJH8g=="], - - "@aws-sdk/core": ["@aws-sdk/core@3.954.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@aws-sdk/xml-builder": "3.953.0", "@smithy/core": "^3.19.0", "@smithy/node-config-provider": "^4.3.6", "@smithy/property-provider": "^4.2.6", "@smithy/protocol-http": "^5.3.6", "@smithy/signature-v4": "^5.3.6", "@smithy/smithy-client": "^4.10.1", "@smithy/types": "^4.10.0", "@smithy/util-base64": "^4.3.0", "@smithy/util-middleware": "^4.2.6", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-5oYO5RP+mvCNXNj8XnF9jZo0EP0LTseYOJVNQYcii1D9DJqzHL3HJWurYh7cXxz7G7eDyvVYA01O9Xpt34TdoA=="], - - "@aws-sdk/credential-provider-env": ["@aws-sdk/credential-provider-env@3.954.0", "", { "dependencies": { "@aws-sdk/core": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/property-provider": "^4.2.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-2HNkqBjfsvyoRuPAiFh86JBFMFyaCNhL4VyH6XqwTGKZffjG7hdBmzXPy7AT7G3oFh1k/1Zc27v0qxaKoK7mBA=="], - - "@aws-sdk/credential-provider-http": ["@aws-sdk/credential-provider-http@3.954.0", "", { "dependencies": { "@aws-sdk/core": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/fetch-http-handler": "^5.3.7", "@smithy/node-http-handler": "^4.4.6", "@smithy/property-provider": "^4.2.6", "@smithy/protocol-http": "^5.3.6", "@smithy/smithy-client": "^4.10.1", "@smithy/types": "^4.10.0", "@smithy/util-stream": "^4.5.7", "tslib": "^2.6.2" } }, "sha512-CrWD5300+NE1OYRnSVDxoG7G0b5cLIZb7yp+rNQ5Jq/kqnTmyJXpVAsivq+bQIDaGzPXhadzpAMIoo7K/aHaag=="], - - "@aws-sdk/credential-provider-ini": ["@aws-sdk/credential-provider-ini@3.954.0", "", { "dependencies": { "@aws-sdk/core": "3.954.0", "@aws-sdk/credential-provider-env": "3.954.0", "@aws-sdk/credential-provider-http": "3.954.0", "@aws-sdk/credential-provider-login": "3.954.0", "@aws-sdk/credential-provider-process": "3.954.0", "@aws-sdk/credential-provider-sso": "3.954.0", "@aws-sdk/credential-provider-web-identity": "3.954.0", "@aws-sdk/nested-clients": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/credential-provider-imds": "^4.2.6", "@smithy/property-provider": "^4.2.6", "@smithy/shared-ini-file-loader": "^4.4.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-WAFD8pVwRSoBsuXcoD+s/hrdsP9Z0PNUedSgkOGExuJVAabpM2cIIMzYNsdHio9XFZUSqHkv8mF5mQXuIZvuzg=="], - - "@aws-sdk/credential-provider-login": ["@aws-sdk/credential-provider-login@3.954.0", "", { "dependencies": { "@aws-sdk/core": "3.954.0", "@aws-sdk/nested-clients": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/property-provider": "^4.2.6", "@smithy/protocol-http": "^5.3.6", "@smithy/shared-ini-file-loader": "^4.4.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-EYqaBWwdVbVK7prmsmgTWLPptoWREplPkFMFscOpVmseDvf/0IjYNbNLLtfuhy/6L7ZBGI9wat2k4u0MRivvxA=="], - - "@aws-sdk/credential-provider-node": ["@aws-sdk/credential-provider-node@3.954.0", "", { "dependencies": { "@aws-sdk/credential-provider-env": "3.954.0", "@aws-sdk/credential-provider-http": "3.954.0", "@aws-sdk/credential-provider-ini": "3.954.0", "@aws-sdk/credential-provider-process": "3.954.0", "@aws-sdk/credential-provider-sso": "3.954.0", "@aws-sdk/credential-provider-web-identity": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/credential-provider-imds": "^4.2.6", "@smithy/property-provider": "^4.2.6", "@smithy/shared-ini-file-loader": "^4.4.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-UPBjw7Lnly5i+/rES8Z5U+nPaumzEUYOE/wrHkxyH6JjwFWn8w7R07fE5Z5cgYlIq1U1lQ7sxYwB3wHPpQ65Aw=="], - - "@aws-sdk/credential-provider-process": ["@aws-sdk/credential-provider-process@3.954.0", "", { "dependencies": { "@aws-sdk/core": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/property-provider": "^4.2.6", "@smithy/shared-ini-file-loader": "^4.4.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-Y1/0O2LgbKM8iIgcVj/GNEQW6p90LVTCOzF2CI1pouoKqxmZ/1F7F66WHoa6XUOfKaCRj/R6nuMR3om9ThaM5A=="], - - "@aws-sdk/credential-provider-sso": ["@aws-sdk/credential-provider-sso@3.954.0", "", { "dependencies": { "@aws-sdk/client-sso": "3.954.0", "@aws-sdk/core": "3.954.0", "@aws-sdk/token-providers": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/property-provider": "^4.2.6", "@smithy/shared-ini-file-loader": "^4.4.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-UXxGfkp/plFRdyidMLvNul5zoLKmHhVQOCrD2OgR/lg9jNqNmJ7abF+Qu8abo902iDkhU21Qj4M398cx6l8Kng=="], - - "@aws-sdk/credential-provider-web-identity": ["@aws-sdk/credential-provider-web-identity@3.954.0", "", { "dependencies": { "@aws-sdk/core": "3.954.0", "@aws-sdk/nested-clients": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/property-provider": "^4.2.6", "@smithy/shared-ini-file-loader": "^4.4.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-XEyf1T08q1tG4zkTS4Dnf1cAQyrJUo/xlvi6XNpqGhY3bOmKUYE2h/K6eITIdytDL9VuCpWYQ6YRcIVtL29E0w=="], - - "@aws-sdk/middleware-bucket-endpoint": ["@aws-sdk/middleware-bucket-endpoint@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@aws-sdk/util-arn-parser": "3.953.0", "@smithy/node-config-provider": "^4.3.6", "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "@smithy/util-config-provider": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-YHVRIOowtGIl/L2WuS83FgRlm31tU0aL1yryWaFtF+AFjA5BIeiFkxIZqaRGxJpJvFEBdohsyq6Ipv5mgWfezg=="], - - "@aws-sdk/middleware-expect-continue": ["@aws-sdk/middleware-expect-continue@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-BQTVXrypQ0rbb7au/Hk4IS5GaJZlwk6O44Rjk6Kxb0IvGQhSurNTuesFiJx1sLbf+w+T31saPtODcfQQERqhCQ=="], - - "@aws-sdk/middleware-flexible-checksums": ["@aws-sdk/middleware-flexible-checksums@3.954.0", "", { "dependencies": { "@aws-crypto/crc32": "5.2.0", "@aws-crypto/crc32c": "5.2.0", "@aws-crypto/util": "5.2.0", "@aws-sdk/core": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/is-array-buffer": "^4.2.0", "@smithy/node-config-provider": "^4.3.6", "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "@smithy/util-middleware": "^4.2.6", "@smithy/util-stream": "^4.5.7", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-hHOPDJyxucNodkgapLhA0VdwDBwVYN9DX20aA6j+3nwutAlZ5skaV7Bw0W3YC7Fh/ieDKKhcSZulONd4lVTwMg=="], - - "@aws-sdk/middleware-host-header": ["@aws-sdk/middleware-host-header@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-jTGhfkONav+r4E6HLOrl5SzBqDmPByUYCkyB/c/3TVb8jX3wAZx8/q9bphKpCh+G5ARi3IdbSisgkZrJYqQ19Q=="], - - "@aws-sdk/middleware-location-constraint": ["@aws-sdk/middleware-location-constraint@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-h0urrbteIQEybyIISaJfQLZ/+/lJPRzPWAQT4epvzfgv/4MKZI7K83dK7SfTwAooVKFBHiCMok2Cf0iHDt07Kw=="], - - "@aws-sdk/middleware-logger": ["@aws-sdk/middleware-logger@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-PlWdVYgcuptkIC0ZKqVUhWNtSHXJSx7U9V8J7dJjRmsXC40X7zpEycvrkzDMJjeTDGcCceYbyYAg/4X1lkcIMw=="], - - "@aws-sdk/middleware-recursion-detection": ["@aws-sdk/middleware-recursion-detection@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@aws/lambda-invoke-store": "^0.2.2", "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-cmIJx0gWeesUKK4YwgE+VQL3mpACr3/J24fbwnc1Z5tntC86b+HQFzU5vsBDw6lLwyD46dBgWdsXFh1jL+ZaFw=="], - - "@aws-sdk/middleware-sdk-s3": ["@aws-sdk/middleware-sdk-s3@3.954.0", "", { "dependencies": { "@aws-sdk/core": "3.954.0", "@aws-sdk/types": "3.953.0", "@aws-sdk/util-arn-parser": "3.953.0", "@smithy/core": "^3.19.0", "@smithy/node-config-provider": "^4.3.6", "@smithy/protocol-http": "^5.3.6", "@smithy/signature-v4": "^5.3.6", "@smithy/smithy-client": "^4.10.1", "@smithy/types": "^4.10.0", "@smithy/util-config-provider": "^4.2.0", "@smithy/util-middleware": "^4.2.6", "@smithy/util-stream": "^4.5.7", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-274CNmnRjknmfFb2o0Azxic54fnujaA8AYSeRUOho3lN48TVzx85eAFWj2kLgvUJO88pE3jBDPWboKQiQdXeUQ=="], - - "@aws-sdk/middleware-ssec": ["@aws-sdk/middleware-ssec@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-OrhG1kcQ9zZh3NS3RovR028N0+UndQ957zF1k5HPLeFLwFwQN1uPOufzzPzAyXIIKtR69ARFsQI4mstZS4DMvw=="], - - "@aws-sdk/middleware-user-agent": ["@aws-sdk/middleware-user-agent@3.954.0", "", { "dependencies": { "@aws-sdk/core": "3.954.0", "@aws-sdk/types": "3.953.0", "@aws-sdk/util-endpoints": "3.953.0", "@smithy/core": "^3.19.0", "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-5PX8JDe3dB2+MqXeGIhmgFnm2rbVsSxhz+Xyuu1oxLtbOn+a9UDA+sNBufEBjt3UxWy5qwEEY1fxdbXXayjlGg=="], - - "@aws-sdk/nested-clients": ["@aws-sdk/nested-clients@3.954.0", "", { "dependencies": { "@aws-crypto/sha256-browser": "5.2.0", "@aws-crypto/sha256-js": "5.2.0", "@aws-sdk/core": "3.954.0", "@aws-sdk/middleware-host-header": "3.953.0", "@aws-sdk/middleware-logger": "3.953.0", "@aws-sdk/middleware-recursion-detection": "3.953.0", "@aws-sdk/middleware-user-agent": "3.954.0", "@aws-sdk/region-config-resolver": "3.953.0", "@aws-sdk/types": "3.953.0", "@aws-sdk/util-endpoints": "3.953.0", "@aws-sdk/util-user-agent-browser": "3.953.0", "@aws-sdk/util-user-agent-node": "3.954.0", "@smithy/config-resolver": "^4.4.4", "@smithy/core": "^3.19.0", "@smithy/fetch-http-handler": "^5.3.7", "@smithy/hash-node": "^4.2.6", "@smithy/invalid-dependency": "^4.2.6", "@smithy/middleware-content-length": "^4.2.6", "@smithy/middleware-endpoint": "^4.4.0", "@smithy/middleware-retry": "^4.4.16", "@smithy/middleware-serde": "^4.2.7", "@smithy/middleware-stack": "^4.2.6", "@smithy/node-config-provider": "^4.3.6", "@smithy/node-http-handler": "^4.4.6", "@smithy/protocol-http": "^5.3.6", "@smithy/smithy-client": "^4.10.1", "@smithy/types": "^4.10.0", "@smithy/url-parser": "^4.2.6", "@smithy/util-base64": "^4.3.0", "@smithy/util-body-length-browser": "^4.2.0", "@smithy/util-body-length-node": "^4.2.1", "@smithy/util-defaults-mode-browser": "^4.3.15", "@smithy/util-defaults-mode-node": "^4.2.18", "@smithy/util-endpoints": "^3.2.6", "@smithy/util-middleware": "^4.2.6", "@smithy/util-retry": "^4.2.6", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-JLUhf35fTQIDPLk6G5KPggL9tV//Hjhy6+N2zZeis76LuBRNhKDq8z1CFyKhjf00vXi/tDYdn9D7y9emI+5Y/g=="], - - "@aws-sdk/region-config-resolver": ["@aws-sdk/region-config-resolver@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@smithy/config-resolver": "^4.4.4", "@smithy/node-config-provider": "^4.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-5MJgnsc+HLO+le0EK1cy92yrC7kyhGZSpaq8PcQvKs9qtXCXT5Tb6tMdkr5Y07JxYsYOV1omWBynvL6PWh08tQ=="], - - "@aws-sdk/signature-v4-multi-region": ["@aws-sdk/signature-v4-multi-region@3.954.0", "", { "dependencies": { "@aws-sdk/middleware-sdk-s3": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/protocol-http": "^5.3.6", "@smithy/signature-v4": "^5.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-GJJbUaSlGrMSRWui3Oz8ByygpQlzDGm195yTKirgGyu4tfYrFr/QWrWT42EUktY/L4Irev1pdHTuLS+AGHO1gw=="], - - "@aws-sdk/token-providers": ["@aws-sdk/token-providers@3.954.0", "", { "dependencies": { "@aws-sdk/core": "3.954.0", "@aws-sdk/nested-clients": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/property-provider": "^4.2.6", "@smithy/shared-ini-file-loader": "^4.4.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-rDyN3oQQKMOJgyQ9/LNbh4fAGAj8ePMGOAQzSP/kyzizmViI6STpBW1o/VRqiTgMNi1bvA9ZasDtfrJqcVt0iA=="], - - "@aws-sdk/types": ["@aws-sdk/types@3.953.0", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-M9Iwg9kTyqTErI0vOTVVpcnTHWzS3VplQppy8MuL02EE+mJ0BIwpWfsaAPQW+/XnVpdNpWZTsHcNE29f1+hR8g=="], - - "@aws-sdk/util-arn-parser": ["@aws-sdk/util-arn-parser@3.953.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-9hqdKkn4OvYzzaLryq2xnwcrPc8ziY34i9szUdgBfSqEC6pBxbY9/lLXmrgzfwMSL2Z7/v2go4Od0p5eukKLMQ=="], - - "@aws-sdk/util-endpoints": ["@aws-sdk/util-endpoints@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@smithy/types": "^4.10.0", "@smithy/url-parser": "^4.2.6", "@smithy/util-endpoints": "^3.2.6", "tslib": "^2.6.2" } }, "sha512-rjaS6jrFksopXvNg6YeN+D1lYwhcByORNlFuYesFvaQNtPOufbE5tJL4GJ3TMXyaY0uFR28N5BHHITPyWWfH/g=="], - - "@aws-sdk/util-locate-window": ["@aws-sdk/util-locate-window@3.953.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-mPxK+I1LcrgC/RSa3G5AMAn8eN2Ay0VOgw8lSRmV1jCtO+iYvNeCqOdxoJUjOW6I5BA4niIRWqVORuRP07776Q=="], - - "@aws-sdk/util-user-agent-browser": ["@aws-sdk/util-user-agent-browser@3.953.0", "", { "dependencies": { "@aws-sdk/types": "3.953.0", "@smithy/types": "^4.10.0", "bowser": "^2.11.0", "tslib": "^2.6.2" } }, "sha512-UF5NeqYesWuFao+u7LJvpV1SJCaLml5BtFZKUdTnNNMeN6jvV+dW/eQoFGpXF94RCqguX0XESmRuRRPQp+/rzQ=="], - - "@aws-sdk/util-user-agent-node": ["@aws-sdk/util-user-agent-node@3.954.0", "", { "dependencies": { "@aws-sdk/middleware-user-agent": "3.954.0", "@aws-sdk/types": "3.953.0", "@smithy/node-config-provider": "^4.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" }, "peerDependencies": { "aws-crt": ">=1.0.0" }, "optionalPeers": ["aws-crt"] }, "sha512-fB5S5VOu7OFkeNzcblQlez4AjO5hgDFaa7phYt7716YWisY3RjAaQPlxgv+G3GltHHDJIfzEC5aRxdf62B9zMg=="], - - "@aws-sdk/xml-builder": ["@aws-sdk/xml-builder@3.953.0", "", { "dependencies": { "@smithy/types": "^4.10.0", "fast-xml-parser": "5.2.5", "tslib": "^2.6.2" } }, "sha512-Zmrj21jQ2OeOJGr9spPiN00aQvXa/WUqRXcTVENhrMt+OFoSOfDFpYhUj9NQ09QmQ8KMWFoWuWW6iKurNqLvAA=="], - - "@aws/lambda-invoke-store": ["@aws/lambda-invoke-store@0.2.2", "", {}, "sha512-C0NBLsIqzDIae8HFw9YIrIBsbc0xTiOtt7fAukGPnqQ/+zZNaq+4jhuccltK0QuWHBnNm/a6kLIRA6GFiM10eg=="], - - "@azure/abort-controller": ["@azure/abort-controller@2.1.2", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-nBrLsEWm4J2u5LpAPjxADTlq3trDgVZZXHNKabeXZtpq3d3AbN/KGO82R87rdDz5/lYB024rtEf10/q0urNgsA=="], - - "@azure/core-auth": ["@azure/core-auth@1.10.1", "", { "dependencies": { "@azure/abort-controller": "^2.1.2", "@azure/core-util": "^1.13.0", "tslib": "^2.6.2" } }, "sha512-ykRMW8PjVAn+RS6ww5cmK9U2CyH9p4Q88YJwvUslfuMmN98w/2rdGRLPqJYObapBCdzBVeDgYWdJnFPFb7qzpg=="], - - "@azure/core-client": ["@azure/core-client@1.10.1", "", { "dependencies": { "@azure/abort-controller": "^2.1.2", "@azure/core-auth": "^1.10.0", "@azure/core-rest-pipeline": "^1.22.0", "@azure/core-tracing": "^1.3.0", "@azure/core-util": "^1.13.0", "@azure/logger": "^1.3.0", "tslib": "^2.6.2" } }, "sha512-Nh5PhEOeY6PrnxNPsEHRr9eimxLwgLlpmguQaHKBinFYA/RU9+kOYVOQqOrTsCL+KSxrLLl1gD8Dk5BFW/7l/w=="], - - "@azure/core-http-compat": ["@azure/core-http-compat@2.3.1", "", { "dependencies": { "@azure/abort-controller": "^2.1.2", "@azure/core-client": "^1.10.0", "@azure/core-rest-pipeline": "^1.22.0" } }, "sha512-az9BkXND3/d5VgdRRQVkiJb2gOmDU8Qcq4GvjtBmDICNiQ9udFmDk4ZpSB5Qq1OmtDJGlQAfBaS4palFsazQ5g=="], - - "@azure/core-lro": ["@azure/core-lro@2.7.2", "", { "dependencies": { "@azure/abort-controller": "^2.0.0", "@azure/core-util": "^1.2.0", "@azure/logger": "^1.0.0", "tslib": "^2.6.2" } }, "sha512-0YIpccoX8m/k00O7mDDMdJpbr6mf1yWo2dfmxt5A8XVZVVMz2SSKaEbMCeJRvgQ0IaSlqhjT47p4hVIRRy90xw=="], - - "@azure/core-paging": ["@azure/core-paging@1.6.2", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-YKWi9YuCU04B55h25cnOYZHxXYtEvQEbKST5vqRga7hWY9ydd3FZHdeQF8pyh+acWZvppw13M/LMGx0LABUVMA=="], - - "@azure/core-rest-pipeline": ["@azure/core-rest-pipeline@1.22.2", "", { "dependencies": { "@azure/abort-controller": "^2.1.2", "@azure/core-auth": "^1.10.0", "@azure/core-tracing": "^1.3.0", "@azure/core-util": "^1.13.0", "@azure/logger": "^1.3.0", "@typespec/ts-http-runtime": "^0.3.0", "tslib": "^2.6.2" } }, "sha512-MzHym+wOi8CLUlKCQu12de0nwcq9k9Kuv43j4Wa++CsCpJwps2eeBQwD2Bu8snkxTtDKDx4GwjuR9E8yC8LNrg=="], - - "@azure/core-tracing": ["@azure/core-tracing@1.3.1", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-9MWKevR7Hz8kNzzPLfX4EAtGM2b8mr50HPDBvio96bURP/9C+HjdH3sBlLSNNrvRAr5/k/svoH457gB5IKpmwQ=="], - - "@azure/core-util": ["@azure/core-util@1.13.1", "", { "dependencies": { "@azure/abort-controller": "^2.1.2", "@typespec/ts-http-runtime": "^0.3.0", "tslib": "^2.6.2" } }, "sha512-XPArKLzsvl0Hf0CaGyKHUyVgF7oDnhKoP85Xv6M4StF/1AhfORhZudHtOyf2s+FcbuQ9dPRAjB8J2KvRRMUK2A=="], - - "@azure/core-xml": ["@azure/core-xml@1.5.0", "", { "dependencies": { "fast-xml-parser": "^5.0.7", "tslib": "^2.8.1" } }, "sha512-D/sdlJBMJfx7gqoj66PKVmhDDaU6TKA49ptcolxdas29X7AfvLTmfAGLjAcIMBK7UZ2o4lygHIqVckOlQU3xWw=="], - - "@azure/identity": ["@azure/identity@4.13.0", "", { "dependencies": { "@azure/abort-controller": "^2.0.0", "@azure/core-auth": "^1.9.0", "@azure/core-client": "^1.9.2", "@azure/core-rest-pipeline": "^1.17.0", "@azure/core-tracing": "^1.0.0", "@azure/core-util": "^1.11.0", "@azure/logger": "^1.0.0", "@azure/msal-browser": "^4.2.0", "@azure/msal-node": "^3.5.0", "open": "^10.1.0", "tslib": "^2.2.0" } }, "sha512-uWC0fssc+hs1TGGVkkghiaFkkS7NkTxfnCH+Hdg+yTehTpMcehpok4PgUKKdyCH+9ldu6FhiHRv84Ntqj1vVcw=="], - - "@azure/logger": ["@azure/logger@1.3.0", "", { "dependencies": { "@typespec/ts-http-runtime": "^0.3.0", "tslib": "^2.6.2" } }, "sha512-fCqPIfOcLE+CGqGPd66c8bZpwAji98tZ4JI9i/mlTNTlsIWslCfpg48s/ypyLxZTump5sypjrKn2/kY7q8oAbA=="], - - "@azure/msal-browser": ["@azure/msal-browser@4.27.0", "", { "dependencies": { "@azure/msal-common": "15.13.3" } }, "sha512-bZ8Pta6YAbdd0o0PEaL1/geBsPrLEnyY/RDWqvF1PP9RUH8EMLvUMGoZFYS6jSlUan6KZ9IMTLCnwpWWpQRK/w=="], - - "@azure/msal-common": ["@azure/msal-common@15.13.3", "", {}, "sha512-shSDU7Ioecya+Aob5xliW9IGq1Ui8y4EVSdWGyI1Gbm4Vg61WpP95LuzcY214/wEjSn6w4PZYD4/iVldErHayQ=="], - - "@azure/msal-node": ["@azure/msal-node@3.8.4", "", { "dependencies": { "@azure/msal-common": "15.13.3", "jsonwebtoken": "^9.0.0", "uuid": "^8.3.0" } }, "sha512-lvuAwsDpPDE/jSuVQOBMpLbXuVuLsPNRwWCyK3/6bPlBk0fGWegqoZ0qjZclMWyQ2JNvIY3vHY7hoFmFmFQcOw=="], - - "@azure/storage-blob": ["@azure/storage-blob@12.29.1", "", { "dependencies": { "@azure/abort-controller": "^2.1.2", "@azure/core-auth": "^1.9.0", "@azure/core-client": "^1.9.3", "@azure/core-http-compat": "^2.2.0", "@azure/core-lro": "^2.2.0", "@azure/core-paging": "^1.6.2", "@azure/core-rest-pipeline": "^1.19.1", "@azure/core-tracing": "^1.2.0", "@azure/core-util": "^1.11.0", "@azure/core-xml": "^1.4.5", "@azure/logger": "^1.1.4", "@azure/storage-common": "^12.1.1", "events": "^3.0.0", "tslib": "^2.8.1" } }, "sha512-7ktyY0rfTM0vo7HvtK6E3UvYnI9qfd6Oz6z/+92VhGRveWng3kJwMKeUpqmW/NmwcDNbxHpSlldG+vsUnRFnBg=="], - - "@azure/storage-common": ["@azure/storage-common@12.1.1", "", { "dependencies": { "@azure/abort-controller": "^2.1.2", "@azure/core-auth": "^1.9.0", "@azure/core-http-compat": "^2.2.0", "@azure/core-rest-pipeline": "^1.19.1", "@azure/core-tracing": "^1.2.0", "@azure/core-util": "^1.11.0", "@azure/logger": "^1.1.4", "events": "^3.3.0", "tslib": "^2.8.1" } }, "sha512-eIOH1pqFwI6UmVNnDQvmFeSg0XppuzDLFeUNO/Xht7ODAzRLgGDh7h550pSxoA+lPDxBl1+D2m/KG3jWzCUjTg=="], - - "@babel/code-frame": ["@babel/code-frame@7.27.1", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.27.1", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-cjQ7ZlQ0Mv3b47hABuTevyTuYN4i+loJKGeV9flcCgIK37cCXRh+L1bd3iBHlynerhQ7BhCkn2BPbQUL+rGqFg=="], - - "@babel/helper-string-parser": ["@babel/helper-string-parser@7.27.1", "", {}, "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA=="], - - "@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.28.5", "", {}, "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q=="], - - "@babel/parser": ["@babel/parser@7.28.5", "", { "dependencies": { "@babel/types": "^7.28.5" }, "bin": { "parser": "bin/babel-parser.js" } }, "sha512-KKBU1VGYR7ORr3At5HAtUQ+TV3SzRCXmA/8OdDZiLDBIZxVyzXuztPjfLd3BV1PRAQGCMWWSHYhL0F8d5uHBDQ=="], - - "@babel/runtime": ["@babel/runtime@7.28.4", "", {}, "sha512-Q/N6JNWvIvPnLDvjlE1OUBLPQHH6l3CltCEsHIujp45zQUSSh8K+gHnaEX45yAT1nyngnINhvWtzN+Nb9D8RAQ=="], - - "@babel/types": ["@babel/types@7.28.5", "", { "dependencies": { "@babel/helper-string-parser": "^7.27.1", "@babel/helper-validator-identifier": "^7.28.5" } }, "sha512-qQ5m48eI/MFLQ5PxQj4PFaprjyCTLI37ElWMmNs0K8Lk3dVeOdNpB3ks8jc7yM5CDmVC73eMVk/trk3fgmrUpA=="], - - "@balena/dockerignore": ["@balena/dockerignore@1.0.2", "", {}, "sha512-wMue2Sy4GAVTk6Ic4tJVcnfdau+gx2EnG7S+uAEe+TWJFqE4YoWN4/H8MSLj4eYJKxGg26lZwboEniNiNwZQ6Q=="], - - "@bcoe/v8-coverage": ["@bcoe/v8-coverage@1.0.2", "", {}, "sha512-6zABk/ECA/QYSCQ1NGiVwwbQerUCZ+TQbp64Q3AgmfNvurHH0j8TtXa1qbShXA6qqkpAj4V5W8pP6mLe1mcMqA=="], - - "@colors/colors": ["@colors/colors@1.5.0", "", {}, "sha512-ooWCrlZP11i8GImSjTHYHLkvFDP48nS4+204nGb1RiX/WXYHmJA2III9/e2DWVabCESdW7hBAEzHRqUn9OUVvQ=="], - - "@esbuild/aix-ppc64": ["@esbuild/aix-ppc64@0.27.2", "", { "os": "aix", "cpu": "ppc64" }, "sha512-GZMB+a0mOMZs4MpDbj8RJp4cw+w1WV5NYD6xzgvzUJ5Ek2jerwfO2eADyI6ExDSUED+1X8aMbegahsJi+8mgpw=="], - - "@esbuild/android-arm": ["@esbuild/android-arm@0.27.2", "", { "os": "android", "cpu": "arm" }, "sha512-DVNI8jlPa7Ujbr1yjU2PfUSRtAUZPG9I1RwW4F4xFB1Imiu2on0ADiI/c3td+KmDtVKNbi+nffGDQMfcIMkwIA=="], - - "@esbuild/android-arm64": ["@esbuild/android-arm64@0.27.2", "", { "os": "android", "cpu": "arm64" }, "sha512-pvz8ZZ7ot/RBphf8fv60ljmaoydPU12VuXHImtAs0XhLLw+EXBi2BLe3OYSBslR4rryHvweW5gmkKFwTiFy6KA=="], - - "@esbuild/android-x64": ["@esbuild/android-x64@0.27.2", "", { "os": "android", "cpu": "x64" }, "sha512-z8Ank4Byh4TJJOh4wpz8g2vDy75zFL0TlZlkUkEwYXuPSgX8yzep596n6mT7905kA9uHZsf/o2OJZubl2l3M7A=="], - - "@esbuild/darwin-arm64": ["@esbuild/darwin-arm64@0.27.2", "", { "os": "darwin", "cpu": "arm64" }, "sha512-davCD2Zc80nzDVRwXTcQP/28fiJbcOwvdolL0sOiOsbwBa72kegmVU0Wrh1MYrbuCL98Omp5dVhQFWRKR2ZAlg=="], - - "@esbuild/darwin-x64": ["@esbuild/darwin-x64@0.27.2", "", { "os": "darwin", "cpu": "x64" }, "sha512-ZxtijOmlQCBWGwbVmwOF/UCzuGIbUkqB1faQRf5akQmxRJ1ujusWsb3CVfk/9iZKr2L5SMU5wPBi1UWbvL+VQA=="], - - "@esbuild/freebsd-arm64": ["@esbuild/freebsd-arm64@0.27.2", "", { "os": "freebsd", "cpu": "arm64" }, "sha512-lS/9CN+rgqQ9czogxlMcBMGd+l8Q3Nj1MFQwBZJyoEKI50XGxwuzznYdwcav6lpOGv5BqaZXqvBSiB/kJ5op+g=="], - - "@esbuild/freebsd-x64": ["@esbuild/freebsd-x64@0.27.2", "", { "os": "freebsd", "cpu": "x64" }, "sha512-tAfqtNYb4YgPnJlEFu4c212HYjQWSO/w/h/lQaBK7RbwGIkBOuNKQI9tqWzx7Wtp7bTPaGC6MJvWI608P3wXYA=="], - - "@esbuild/linux-arm": ["@esbuild/linux-arm@0.27.2", "", { "os": "linux", "cpu": "arm" }, "sha512-vWfq4GaIMP9AIe4yj1ZUW18RDhx6EPQKjwe7n8BbIecFtCQG4CfHGaHuh7fdfq+y3LIA2vGS/o9ZBGVxIDi9hw=="], - - "@esbuild/linux-arm64": ["@esbuild/linux-arm64@0.27.2", "", { "os": "linux", "cpu": "arm64" }, "sha512-hYxN8pr66NsCCiRFkHUAsxylNOcAQaxSSkHMMjcpx0si13t1LHFphxJZUiGwojB1a/Hd5OiPIqDdXONia6bhTw=="], - - "@esbuild/linux-ia32": ["@esbuild/linux-ia32@0.27.2", "", { "os": "linux", "cpu": "ia32" }, "sha512-MJt5BRRSScPDwG2hLelYhAAKh9imjHK5+NE/tvnRLbIqUWa+0E9N4WNMjmp/kXXPHZGqPLxggwVhz7QP8CTR8w=="], - - "@esbuild/linux-loong64": ["@esbuild/linux-loong64@0.27.2", "", { "os": "linux", "cpu": "none" }, "sha512-lugyF1atnAT463aO6KPshVCJK5NgRnU4yb3FUumyVz+cGvZbontBgzeGFO1nF+dPueHD367a2ZXe1NtUkAjOtg=="], - - "@esbuild/linux-mips64el": ["@esbuild/linux-mips64el@0.27.2", "", { "os": "linux", "cpu": "none" }, "sha512-nlP2I6ArEBewvJ2gjrrkESEZkB5mIoaTswuqNFRv/WYd+ATtUpe9Y09RnJvgvdag7he0OWgEZWhviS1OTOKixw=="], - - "@esbuild/linux-ppc64": ["@esbuild/linux-ppc64@0.27.2", "", { "os": "linux", "cpu": "ppc64" }, "sha512-C92gnpey7tUQONqg1n6dKVbx3vphKtTHJaNG2Ok9lGwbZil6DrfyecMsp9CrmXGQJmZ7iiVXvvZH6Ml5hL6XdQ=="], - - "@esbuild/linux-riscv64": ["@esbuild/linux-riscv64@0.27.2", "", { "os": "linux", "cpu": "none" }, "sha512-B5BOmojNtUyN8AXlK0QJyvjEZkWwy/FKvakkTDCziX95AowLZKR6aCDhG7LeF7uMCXEJqwa8Bejz5LTPYm8AvA=="], - - "@esbuild/linux-s390x": ["@esbuild/linux-s390x@0.27.2", "", { "os": "linux", "cpu": "s390x" }, "sha512-p4bm9+wsPwup5Z8f4EpfN63qNagQ47Ua2znaqGH6bqLlmJ4bx97Y9JdqxgGZ6Y8xVTixUnEkoKSHcpRlDnNr5w=="], - - "@esbuild/linux-x64": ["@esbuild/linux-x64@0.27.2", "", { "os": "linux", "cpu": "x64" }, "sha512-uwp2Tip5aPmH+NRUwTcfLb+W32WXjpFejTIOWZFw/v7/KnpCDKG66u4DLcurQpiYTiYwQ9B7KOeMJvLCu/OvbA=="], - - "@esbuild/netbsd-arm64": ["@esbuild/netbsd-arm64@0.27.2", "", { "os": "none", "cpu": "arm64" }, "sha512-Kj6DiBlwXrPsCRDeRvGAUb/LNrBASrfqAIok+xB0LxK8CHqxZ037viF13ugfsIpePH93mX7xfJp97cyDuTZ3cw=="], - - "@esbuild/netbsd-x64": ["@esbuild/netbsd-x64@0.27.2", "", { "os": "none", "cpu": "x64" }, "sha512-HwGDZ0VLVBY3Y+Nw0JexZy9o/nUAWq9MlV7cahpaXKW6TOzfVno3y3/M8Ga8u8Yr7GldLOov27xiCnqRZf0tCA=="], - - "@esbuild/openbsd-arm64": ["@esbuild/openbsd-arm64@0.27.2", "", { "os": "openbsd", "cpu": "arm64" }, "sha512-DNIHH2BPQ5551A7oSHD0CKbwIA/Ox7+78/AWkbS5QoRzaqlev2uFayfSxq68EkonB+IKjiuxBFoV8ESJy8bOHA=="], - - "@esbuild/openbsd-x64": ["@esbuild/openbsd-x64@0.27.2", "", { "os": "openbsd", "cpu": "x64" }, "sha512-/it7w9Nb7+0KFIzjalNJVR5bOzA9Vay+yIPLVHfIQYG/j+j9VTH84aNB8ExGKPU4AzfaEvN9/V4HV+F+vo8OEg=="], - - "@esbuild/openharmony-arm64": ["@esbuild/openharmony-arm64@0.27.2", "", { "os": "none", "cpu": "arm64" }, "sha512-LRBbCmiU51IXfeXk59csuX/aSaToeG7w48nMwA6049Y4J4+VbWALAuXcs+qcD04rHDuSCSRKdmY63sruDS5qag=="], - - "@esbuild/sunos-x64": ["@esbuild/sunos-x64@0.27.2", "", { "os": "sunos", "cpu": "x64" }, "sha512-kMtx1yqJHTmqaqHPAzKCAkDaKsffmXkPHThSfRwZGyuqyIeBvf08KSsYXl+abf5HDAPMJIPnbBfXvP2ZC2TfHg=="], - - "@esbuild/win32-arm64": ["@esbuild/win32-arm64@0.27.2", "", { "os": "win32", "cpu": "arm64" }, "sha512-Yaf78O/B3Kkh+nKABUF++bvJv5Ijoy9AN1ww904rOXZFLWVc5OLOfL56W+C8F9xn5JQZa3UX6m+IktJnIb1Jjg=="], - - "@esbuild/win32-ia32": ["@esbuild/win32-ia32@0.27.2", "", { "os": "win32", "cpu": "ia32" }, "sha512-Iuws0kxo4yusk7sw70Xa2E2imZU5HoixzxfGCdxwBdhiDgt9vX9VUCBhqcwY7/uh//78A1hMkkROMJq9l27oLQ=="], - - "@esbuild/win32-x64": ["@esbuild/win32-x64@0.27.2", "", { "os": "win32", "cpu": "x64" }, "sha512-sRdU18mcKf7F+YgheI/zGf5alZatMUTKj/jNS6l744f9u3WFu4v7twcUI9vu4mknF4Y9aDlblIie0IM+5xxaqQ=="], - - "@eslint-community/eslint-utils": ["@eslint-community/eslint-utils@4.9.0", "", { "dependencies": { "eslint-visitor-keys": "^3.4.3" }, "peerDependencies": { "eslint": "^6.0.0 || ^7.0.0 || >=8.0.0" } }, "sha512-ayVFHdtZ+hsq1t2Dy24wCmGXGe4q9Gu3smhLYALJrr473ZH27MsnSL+LKUlimp4BWJqMDMLmPpx/Q9R3OAlL4g=="], - - "@eslint-community/regexpp": ["@eslint-community/regexpp@4.12.2", "", {}, "sha512-EriSTlt5OC9/7SXkRSCAhfSxxoSUgBm33OH+IkwbdpgoqsSsUg7y3uh+IICI/Qg4BBWr3U2i39RpmycbxMq4ew=="], - - "@eslint/config-array": ["@eslint/config-array@0.21.1", "", { "dependencies": { "@eslint/object-schema": "^2.1.7", "debug": "^4.3.1", "minimatch": "^3.1.2" } }, "sha512-aw1gNayWpdI/jSYVgzN5pL0cfzU02GT3NBpeT/DXbx1/1x7ZKxFPd9bwrzygx/qiwIQiJ1sw/zD8qY/kRvlGHA=="], - - "@eslint/config-helpers": ["@eslint/config-helpers@0.4.2", "", { "dependencies": { "@eslint/core": "^0.17.0" } }, "sha512-gBrxN88gOIf3R7ja5K9slwNayVcZgK6SOUORm2uBzTeIEfeVaIhOpCtTox3P6R7o2jLFwLFTLnC7kU/RGcYEgw=="], - - "@eslint/core": ["@eslint/core@0.17.0", "", { "dependencies": { "@types/json-schema": "^7.0.15" } }, "sha512-yL/sLrpmtDaFEiUj1osRP4TI2MDz1AddJL+jZ7KSqvBuliN4xqYY54IfdN8qD8Toa6g1iloph1fxQNkjOxrrpQ=="], - - "@eslint/eslintrc": ["@eslint/eslintrc@3.3.3", "", { "dependencies": { "ajv": "^6.12.4", "debug": "^4.3.2", "espree": "^10.0.1", "globals": "^14.0.0", "ignore": "^5.2.0", "import-fresh": "^3.2.1", "js-yaml": "^4.1.1", "minimatch": "^3.1.2", "strip-json-comments": "^3.1.1" } }, "sha512-Kr+LPIUVKz2qkx1HAMH8q1q6azbqBAsXJUxBl/ODDuVPX45Z9DfwB8tPjTi6nNZ8BuM3nbJxC5zCAg5elnBUTQ=="], - - "@eslint/js": ["@eslint/js@9.39.2", "", {}, "sha512-q1mjIoW1VX4IvSocvM/vbTiveKC4k9eLrajNEuSsmjymSDEbpGddtpfOoN7YGAqBK3NG+uqo8ia4PDTt8buCYA=="], - - "@eslint/object-schema": ["@eslint/object-schema@2.1.7", "", {}, "sha512-VtAOaymWVfZcmZbp6E2mympDIHvyjXs/12LqWYjVw6qjrfF+VK+fyG33kChz3nnK+SU5/NeHOqrTEHS8sXO3OA=="], - - "@eslint/plugin-kit": ["@eslint/plugin-kit@0.4.1", "", { "dependencies": { "@eslint/core": "^0.17.0", "levn": "^0.4.1" } }, "sha512-43/qtrDUokr7LJqoF2c3+RInu/t4zfrpYdoSDfYyhg52rwLV6TnOvdG4fXm7IkSB3wErkcmJS9iEhjVtOSEjjA=="], - - "@google-cloud/paginator": ["@google-cloud/paginator@5.0.2", "", { "dependencies": { "arrify": "^2.0.0", "extend": "^3.0.2" } }, "sha512-DJS3s0OVH4zFDB1PzjxAsHqJT6sKVbRwwML0ZBP9PbU7Yebtu/7SWMRzvO2J3nUi9pRNITCfu4LJeooM2w4pjg=="], - - "@google-cloud/projectify": ["@google-cloud/projectify@4.0.0", "", {}, "sha512-MmaX6HeSvyPbWGwFq7mXdo0uQZLGBYCwziiLIGq5JVX+/bdI3SAq6bP98trV5eTWfLuvsMcIC1YJOF2vfteLFA=="], - - "@google-cloud/promisify": ["@google-cloud/promisify@4.0.0", "", {}, "sha512-Orxzlfb9c67A15cq2JQEyVc7wEsmFBmHjZWZYQMUyJ1qivXyMwdyNOs9odi79hze+2zqdTtu1E19IM/FtqZ10g=="], - - "@google-cloud/storage": ["@google-cloud/storage@7.18.0", "", { "dependencies": { "@google-cloud/paginator": "^5.0.0", "@google-cloud/projectify": "^4.0.0", "@google-cloud/promisify": "<4.1.0", "abort-controller": "^3.0.0", "async-retry": "^1.3.3", "duplexify": "^4.1.3", "fast-xml-parser": "^4.4.1", "gaxios": "^6.0.2", "google-auth-library": "^9.6.3", "html-entities": "^2.5.2", "mime": "^3.0.0", "p-limit": "^3.0.1", "retry-request": "^7.0.0", "teeny-request": "^9.0.0", "uuid": "^8.0.0" } }, "sha512-r3ZwDMiz4nwW6R922Z1pwpePxyRwE5GdevYX63hRmAQUkUQJcBH/79EnQPDv5cOv1mFBgevdNWQfi3tie3dHrQ=="], - - "@grpc/grpc-js": ["@grpc/grpc-js@1.14.3", "", { "dependencies": { "@grpc/proto-loader": "^0.8.0", "@js-sdsl/ordered-map": "^4.4.2" } }, "sha512-Iq8QQQ/7X3Sac15oB6p0FmUg/klxQvXLeileoqrTRGJYLV+/9tubbr9ipz0GKHjmXVsgFPo/+W+2cA8eNcR+XA=="], - - "@grpc/proto-loader": ["@grpc/proto-loader@0.7.15", "", { "dependencies": { "lodash.camelcase": "^4.3.0", "long": "^5.0.0", "protobufjs": "^7.2.5", "yargs": "^17.7.2" }, "bin": { "proto-loader-gen-types": "build/bin/proto-loader-gen-types.js" } }, "sha512-tMXdRCfYVixjuFK+Hk0Q1s38gV9zDiDJfWL3h1rv4Qc39oILCu1TRTDt7+fGUI8K4G1Fj125Hx/ru3azECWTyQ=="], - - "@humanfs/core": ["@humanfs/core@0.19.1", "", {}, "sha512-5DyQ4+1JEUzejeK1JGICcideyfUbGixgS9jNgex5nqkW+cY7WZhxBigmieN5Qnw9ZosSNVC9KQKyb+GUaGyKUA=="], - - "@humanfs/node": ["@humanfs/node@0.16.7", "", { "dependencies": { "@humanfs/core": "^0.19.1", "@humanwhocodes/retry": "^0.4.0" } }, "sha512-/zUx+yOsIrG4Y43Eh2peDeKCxlRt/gET6aHfaKpuq267qXdYDFViVHfMaLyygZOnl0kGWxFIgsBy8QFuTLUXEQ=="], - - "@humanwhocodes/module-importer": ["@humanwhocodes/module-importer@1.0.1", "", {}, "sha512-bxveV4V8v5Yb4ncFTT3rPSgZBOpCkjfK0y4oVVVJwIuDVBRMDXrPyXRL988i5ap9m9bnyEEjWfm5WkBmtffLfA=="], - - "@humanwhocodes/retry": ["@humanwhocodes/retry@0.4.3", "", {}, "sha512-bV0Tgo9K4hfPCek+aMAn81RppFKv2ySDQeMoSZuvTASywNTnVJCArCZE2FWqpvIatKu7VMRLWlR1EazvVhDyhQ=="], - - "@hutson/parse-repository-url": ["@hutson/parse-repository-url@3.0.2", "", {}, "sha512-H9XAx3hc0BQHY6l+IFSWHDySypcXsvsuLhgYLUGywmJ5pswRVQJUHpOsobnLYp2ZUaUlKiKDrgWWhosOwAEM8Q=="], - - "@inquirer/ansi": ["@inquirer/ansi@1.0.2", "", {}, "sha512-S8qNSZiYzFd0wAcyG5AXCvUHC5Sr7xpZ9wZ2py9XR88jUz8wooStVx5M6dRzczbBWjic9NP7+rY0Xi7qqK/aMQ=="], - - "@inquirer/checkbox": ["@inquirer/checkbox@4.3.2", "", { "dependencies": { "@inquirer/ansi": "^1.0.2", "@inquirer/core": "^10.3.2", "@inquirer/figures": "^1.0.15", "@inquirer/type": "^3.0.10", "yoctocolors-cjs": "^2.1.3" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-VXukHf0RR1doGe6Sm4F0Em7SWYLTHSsbGfJdS9Ja2bX5/D5uwVOEjr07cncLROdBvmnvCATYEWlHqYmXv2IlQA=="], - - "@inquirer/confirm": ["@inquirer/confirm@5.1.21", "", { "dependencies": { "@inquirer/core": "^10.3.2", "@inquirer/type": "^3.0.10" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-KR8edRkIsUayMXV+o3Gv+q4jlhENF9nMYUZs9PA2HzrXeHI8M5uDag70U7RJn9yyiMZSbtF5/UexBtAVtZGSbQ=="], - - "@inquirer/core": ["@inquirer/core@10.3.2", "", { "dependencies": { "@inquirer/ansi": "^1.0.2", "@inquirer/figures": "^1.0.15", "@inquirer/type": "^3.0.10", "cli-width": "^4.1.0", "mute-stream": "^2.0.0", "signal-exit": "^4.1.0", "wrap-ansi": "^6.2.0", "yoctocolors-cjs": "^2.1.3" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-43RTuEbfP8MbKzedNqBrlhhNKVwoK//vUFNW3Q3vZ88BLcrs4kYpGg+B2mm5p2K/HfygoCxuKwJJiv8PbGmE0A=="], - - "@inquirer/editor": ["@inquirer/editor@4.2.23", "", { "dependencies": { "@inquirer/core": "^10.3.2", "@inquirer/external-editor": "^1.0.3", "@inquirer/type": "^3.0.10" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-aLSROkEwirotxZ1pBaP8tugXRFCxW94gwrQLxXfrZsKkfjOYC1aRvAZuhpJOb5cu4IBTJdsCigUlf2iCOu4ZDQ=="], - - "@inquirer/expand": ["@inquirer/expand@4.0.23", "", { "dependencies": { "@inquirer/core": "^10.3.2", "@inquirer/type": "^3.0.10", "yoctocolors-cjs": "^2.1.3" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-nRzdOyFYnpeYTTR2qFwEVmIWypzdAx/sIkCMeTNTcflFOovfqUk+HcFhQQVBftAh9gmGrpFj6QcGEqrDMDOiew=="], - - "@inquirer/external-editor": ["@inquirer/external-editor@1.0.3", "", { "dependencies": { "chardet": "^2.1.1", "iconv-lite": "^0.7.0" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-RWbSrDiYmO4LbejWY7ttpxczuwQyZLBUyygsA9Nsv95hpzUWwnNTVQmAq3xuh7vNwCp07UTmE5i11XAEExx4RA=="], - - "@inquirer/figures": ["@inquirer/figures@1.0.15", "", {}, "sha512-t2IEY+unGHOzAaVM5Xx6DEWKeXlDDcNPeDyUpsRc6CUhBfU3VQOEl+Vssh7VNp1dR8MdUJBWhuObjXCsVpjN5g=="], - - "@inquirer/input": ["@inquirer/input@4.3.1", "", { "dependencies": { "@inquirer/core": "^10.3.2", "@inquirer/type": "^3.0.10" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-kN0pAM4yPrLjJ1XJBjDxyfDduXOuQHrBB8aLDMueuwUGn+vNpF7Gq7TvyVxx8u4SHlFFj4trmj+a2cbpG4Jn1g=="], - - "@inquirer/number": ["@inquirer/number@3.0.23", "", { "dependencies": { "@inquirer/core": "^10.3.2", "@inquirer/type": "^3.0.10" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-5Smv0OK7K0KUzUfYUXDXQc9jrf8OHo4ktlEayFlelCjwMXz0299Y8OrI+lj7i4gCBY15UObk76q0QtxjzFcFcg=="], - - "@inquirer/password": ["@inquirer/password@4.0.23", "", { "dependencies": { "@inquirer/ansi": "^1.0.2", "@inquirer/core": "^10.3.2", "@inquirer/type": "^3.0.10" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-zREJHjhT5vJBMZX/IUbyI9zVtVfOLiTO66MrF/3GFZYZ7T4YILW5MSkEYHceSii/KtRk+4i3RE7E1CUXA2jHcA=="], - - "@inquirer/prompts": ["@inquirer/prompts@7.10.1", "", { "dependencies": { "@inquirer/checkbox": "^4.3.2", "@inquirer/confirm": "^5.1.21", "@inquirer/editor": "^4.2.23", "@inquirer/expand": "^4.0.23", "@inquirer/input": "^4.3.1", "@inquirer/number": "^3.0.23", "@inquirer/password": "^4.0.23", "@inquirer/rawlist": "^4.1.11", "@inquirer/search": "^3.2.2", "@inquirer/select": "^4.4.2" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-Dx/y9bCQcXLI5ooQ5KyvA4FTgeo2jYj/7plWfV5Ak5wDPKQZgudKez2ixyfz7tKXzcJciTxqLeK7R9HItwiByg=="], - - "@inquirer/rawlist": ["@inquirer/rawlist@4.1.11", "", { "dependencies": { "@inquirer/core": "^10.3.2", "@inquirer/type": "^3.0.10", "yoctocolors-cjs": "^2.1.3" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-+LLQB8XGr3I5LZN/GuAHo+GpDJegQwuPARLChlMICNdwW7OwV2izlCSCxN6cqpL0sMXmbKbFcItJgdQq5EBXTw=="], - - "@inquirer/search": ["@inquirer/search@3.2.2", "", { "dependencies": { "@inquirer/core": "^10.3.2", "@inquirer/figures": "^1.0.15", "@inquirer/type": "^3.0.10", "yoctocolors-cjs": "^2.1.3" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-p2bvRfENXCZdWF/U2BXvnSI9h+tuA8iNqtUKb9UWbmLYCRQxd8WkvwWvYn+3NgYaNwdUkHytJMGG4MMLucI1kA=="], - - "@inquirer/select": ["@inquirer/select@4.4.2", "", { "dependencies": { "@inquirer/ansi": "^1.0.2", "@inquirer/core": "^10.3.2", "@inquirer/figures": "^1.0.15", "@inquirer/type": "^3.0.10", "yoctocolors-cjs": "^2.1.3" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-l4xMuJo55MAe+N7Qr4rX90vypFwCajSakx59qe/tMaC1aEHWLyw68wF4o0A4SLAY4E0nd+Vt+EyskeDIqu1M6w=="], - - "@inquirer/type": ["@inquirer/type@3.0.10", "", { "peerDependencies": { "@types/node": ">=18" } }, "sha512-BvziSRxfz5Ov8ch0z/n3oijRSEcEsHnhggm4xFZe93DHcUCTlutlq9Ox4SVENAfcRD22UQq7T/atg9Wr3k09eA=="], - - "@isaacs/cliui": ["@isaacs/cliui@8.0.2", "", { "dependencies": { "string-width": "^5.1.2", "string-width-cjs": "npm:string-width@^4.2.0", "strip-ansi": "^7.0.1", "strip-ansi-cjs": "npm:strip-ansi@^6.0.1", "wrap-ansi": "^8.1.0", "wrap-ansi-cjs": "npm:wrap-ansi@^7.0.0" } }, "sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA=="], - - "@istanbuljs/schema": ["@istanbuljs/schema@0.1.3", "", {}, "sha512-ZXRY4jNvVgSVQ8DL3LTcakaAtXwTVUxE81hslsyD2AtoXW/wVob10HkOJ1X/pAlcI7D+2YoZKg5do8G/w6RYgA=="], - - "@jridgewell/gen-mapping": ["@jridgewell/gen-mapping@0.3.13", "", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.0", "@jridgewell/trace-mapping": "^0.3.24" } }, "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA=="], - - "@jridgewell/resolve-uri": ["@jridgewell/resolve-uri@3.1.2", "", {}, "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw=="], - - "@jridgewell/source-map": ["@jridgewell/source-map@0.3.11", "", { "dependencies": { "@jridgewell/gen-mapping": "^0.3.5", "@jridgewell/trace-mapping": "^0.3.25" } }, "sha512-ZMp1V8ZFcPG5dIWnQLr3NSI1MiCU7UETdS/A0G8V/XWHvJv3ZsFqutJn1Y5RPmAPX6F3BiE397OqveU/9NCuIA=="], - - "@jridgewell/sourcemap-codec": ["@jridgewell/sourcemap-codec@1.5.5", "", {}, "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og=="], - - "@jridgewell/trace-mapping": ["@jridgewell/trace-mapping@0.3.31", "", { "dependencies": { "@jridgewell/resolve-uri": "^3.1.0", "@jridgewell/sourcemap-codec": "^1.4.14" } }, "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw=="], - - "@js-sdsl/ordered-map": ["@js-sdsl/ordered-map@4.4.2", "", {}, "sha512-iUKgm52T8HOE/makSxjqoWhe95ZJA1/G1sYsGev2JDKUSS14KAgg1LHb+Ba+IPow0xflbnSkOsZcO08C7w1gYw=="], - - "@msgpack/msgpack": ["@msgpack/msgpack@3.1.2", "", {}, "sha512-JEW4DEtBzfe8HvUYecLU9e6+XJnKDlUAIve8FvPzF3Kzs6Xo/KuZkZJsDH0wJXl/qEZbeeE7edxDNY3kMs39hQ=="], - - "@napi-rs/canvas": ["@napi-rs/canvas@0.1.84", "", { "optionalDependencies": { "@napi-rs/canvas-android-arm64": "0.1.84", "@napi-rs/canvas-darwin-arm64": "0.1.84", "@napi-rs/canvas-darwin-x64": "0.1.84", "@napi-rs/canvas-linux-arm-gnueabihf": "0.1.84", "@napi-rs/canvas-linux-arm64-gnu": "0.1.84", "@napi-rs/canvas-linux-arm64-musl": "0.1.84", "@napi-rs/canvas-linux-riscv64-gnu": "0.1.84", "@napi-rs/canvas-linux-x64-gnu": "0.1.84", "@napi-rs/canvas-linux-x64-musl": "0.1.84", "@napi-rs/canvas-win32-x64-msvc": "0.1.84" } }, "sha512-88FTNFs4uuiFKP0tUrPsEXhpe9dg7za9ILZJE08pGdUveMIDeana1zwfVkqRHJDPJFAmGY3dXmJ99dzsy57YnA=="], - - "@napi-rs/canvas-android-arm64": ["@napi-rs/canvas-android-arm64@0.1.84", "", { "os": "android", "cpu": "arm64" }, "sha512-pdvuqvj3qtwVryqgpAGornJLV6Ezpk39V6wT4JCnRVGy8I3Tk1au8qOalFGrx/r0Ig87hWslysPpHBxVpBMIww=="], - - "@napi-rs/canvas-darwin-arm64": ["@napi-rs/canvas-darwin-arm64@0.1.84", "", { "os": "darwin", "cpu": "arm64" }, "sha512-A8IND3Hnv0R6abc6qCcCaOCujTLMmGxtucMTZ5vbQUrEN/scxi378MyTLtyWg+MRr6bwQJ6v/orqMS9datIcww=="], - - "@napi-rs/canvas-darwin-x64": ["@napi-rs/canvas-darwin-x64@0.1.84", "", { "os": "darwin", "cpu": "x64" }, "sha512-AUW45lJhYWwnA74LaNeqhvqYKK/2hNnBBBl03KRdqeCD4tKneUSrxUqIv8d22CBweOvrAASyKN3W87WO2zEr/A=="], - - "@napi-rs/canvas-linux-arm-gnueabihf": ["@napi-rs/canvas-linux-arm-gnueabihf@0.1.84", "", { "os": "linux", "cpu": "arm" }, "sha512-8zs5ZqOrdgs4FioTxSBrkl/wHZB56bJNBqaIsfPL4ZkEQCinOkrFF7xIcXiHiKp93J3wUtbIzeVrhTIaWwqk+A=="], - - "@napi-rs/canvas-linux-arm64-gnu": ["@napi-rs/canvas-linux-arm64-gnu@0.1.84", "", { "os": "linux", "cpu": "arm64" }, "sha512-i204vtowOglJUpbAFWU5mqsJgH0lVpNk/Ml4mQtB4Lndd86oF+Otr6Mr5KQnZHqYGhlSIKiU2SYnUbhO28zGQA=="], - - "@napi-rs/canvas-linux-arm64-musl": ["@napi-rs/canvas-linux-arm64-musl@0.1.84", "", { "os": "linux", "cpu": "arm64" }, "sha512-VyZq0EEw+OILnWk7G3ZgLLPaz1ERaPP++jLjeyLMbFOF+Tr4zHzWKiKDsEV/cT7btLPZbVoR3VX+T9/QubnURQ=="], - - "@napi-rs/canvas-linux-riscv64-gnu": ["@napi-rs/canvas-linux-riscv64-gnu@0.1.84", "", { "os": "linux", "cpu": "none" }, "sha512-PSMTh8DiThvLRsbtc/a065I/ceZk17EXAATv9uNvHgkgo7wdEfTh2C3aveNkBMGByVO3tvnvD5v/YFtZL07cIg=="], - - "@napi-rs/canvas-linux-x64-gnu": ["@napi-rs/canvas-linux-x64-gnu@0.1.84", "", { "os": "linux", "cpu": "x64" }, "sha512-N1GY3noO1oqgEo3rYQIwY44kfM11vA0lDbN0orTOHfCSUZTUyiYCY0nZ197QMahZBm1aR/vYgsWpV74MMMDuNA=="], - - "@napi-rs/canvas-linux-x64-musl": ["@napi-rs/canvas-linux-x64-musl@0.1.84", "", { "os": "linux", "cpu": "x64" }, "sha512-vUZmua6ADqTWyHyei81aXIt9wp0yjeNwTH0KdhdeoBb6azHmFR8uKTukZMXfLCC3bnsW0t4lW7K78KNMknmtjg=="], - - "@napi-rs/canvas-win32-x64-msvc": ["@napi-rs/canvas-win32-x64-msvc@0.1.84", "", { "os": "win32", "cpu": "x64" }, "sha512-YSs8ncurc1xzegUMNnQUTYrdrAuaXdPMOa+iYYyAxydOtg0ppV386hyYMsy00Yip1NlTgLCseRG4sHSnjQx6og=="], - - "@pkgjs/parseargs": ["@pkgjs/parseargs@0.11.0", "", {}, "sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg=="], - - "@protobufjs/aspromise": ["@protobufjs/aspromise@1.1.2", "", {}, "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ=="], - - "@protobufjs/base64": ["@protobufjs/base64@1.1.2", "", {}, "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg=="], - - "@protobufjs/codegen": ["@protobufjs/codegen@2.0.4", "", {}, "sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg=="], - - "@protobufjs/eventemitter": ["@protobufjs/eventemitter@1.1.0", "", {}, "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q=="], - - "@protobufjs/fetch": ["@protobufjs/fetch@1.1.0", "", { "dependencies": { "@protobufjs/aspromise": "^1.1.1", "@protobufjs/inquire": "^1.1.0" } }, "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ=="], - - "@protobufjs/float": ["@protobufjs/float@1.0.2", "", {}, "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ=="], - - "@protobufjs/inquire": ["@protobufjs/inquire@1.1.0", "", {}, "sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q=="], - - "@protobufjs/path": ["@protobufjs/path@1.1.2", "", {}, "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA=="], - - "@protobufjs/pool": ["@protobufjs/pool@1.1.0", "", {}, "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw=="], - - "@protobufjs/utf8": ["@protobufjs/utf8@1.1.0", "", {}, "sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw=="], - - "@rollup/plugin-commonjs": ["@rollup/plugin-commonjs@28.0.9", "", { "dependencies": { "@rollup/pluginutils": "^5.0.1", "commondir": "^1.0.1", "estree-walker": "^2.0.2", "fdir": "^6.2.0", "is-reference": "1.2.1", "magic-string": "^0.30.3", "picomatch": "^4.0.2" }, "peerDependencies": { "rollup": "^2.68.0||^3.0.0||^4.0.0" } }, "sha512-PIR4/OHZ79romx0BVVll/PkwWpJ7e5lsqFa3gFfcrFPWwLXLV39JVUzQV9RKjWerE7B845Hqjj9VYlQeieZ2dA=="], - - "@rollup/plugin-node-resolve": ["@rollup/plugin-node-resolve@16.0.3", "", { "dependencies": { "@rollup/pluginutils": "^5.0.1", "@types/resolve": "1.20.2", "deepmerge": "^4.2.2", "is-module": "^1.0.0", "resolve": "^1.22.1" }, "peerDependencies": { "rollup": "^2.78.0||^3.0.0||^4.0.0" } }, "sha512-lUYM3UBGuM93CnMPG1YocWu7X802BrNF3jW2zny5gQyLQgRFJhV1Sq0Zi74+dh/6NBx1DxFC4b4GXg9wUCG5Qg=="], - - "@rollup/plugin-replace": ["@rollup/plugin-replace@6.0.3", "", { "dependencies": { "@rollup/pluginutils": "^5.0.1", "magic-string": "^0.30.3" }, "peerDependencies": { "rollup": "^1.20.0||^2.0.0||^3.0.0||^4.0.0" } }, "sha512-J4RZarRvQAm5IF0/LwUUg+obsm+xZhYnbMXmXROyoSE1ATJe3oXSb9L5MMppdxP2ylNSjv6zFBwKYjcKMucVfA=="], - - "@rollup/plugin-terser": ["@rollup/plugin-terser@0.4.4", "", { "dependencies": { "serialize-javascript": "^6.0.1", "smob": "^1.0.0", "terser": "^5.17.4" }, "peerDependencies": { "rollup": "^2.0.0||^3.0.0||^4.0.0" } }, "sha512-XHeJC5Bgvs8LfukDwWZp7yeqin6ns8RTl2B9avbejt6tZqsqvVoWI7ZTQrcNsfKEDWBTnTxM8nMDkO2IFFbd0A=="], - - "@rollup/pluginutils": ["@rollup/pluginutils@5.3.0", "", { "dependencies": { "@types/estree": "^1.0.0", "estree-walker": "^2.0.2", "picomatch": "^4.0.2" }, "peerDependencies": { "rollup": "^1.20.0||^2.0.0||^3.0.0||^4.0.0" } }, "sha512-5EdhGZtnu3V88ces7s53hhfK5KSASnJZv8Lulpc04cWO3REESroJXg73DFsOmgbU2BhwV0E20bu2IDZb3VKW4Q=="], - - "@rollup/rollup-android-arm-eabi": ["@rollup/rollup-android-arm-eabi@4.53.5", "", { "os": "android", "cpu": "arm" }, "sha512-iDGS/h7D8t7tvZ1t6+WPK04KD0MwzLZrG0se1hzBjSi5fyxlsiggoJHwh18PCFNn7tG43OWb6pdZ6Y+rMlmyNQ=="], - - "@rollup/rollup-android-arm64": ["@rollup/rollup-android-arm64@4.53.5", "", { "os": "android", "cpu": "arm64" }, "sha512-wrSAViWvZHBMMlWk6EJhvg8/rjxzyEhEdgfMMjREHEq11EtJ6IP6yfcCH57YAEca2Oe3FNCE9DSTgU70EIGmVw=="], - - "@rollup/rollup-darwin-arm64": ["@rollup/rollup-darwin-arm64@4.53.5", "", { "os": "darwin", "cpu": "arm64" }, "sha512-S87zZPBmRO6u1YXQLwpveZm4JfPpAa6oHBX7/ghSiGH3rz/KDgAu1rKdGutV+WUI6tKDMbaBJomhnT30Y2t4VQ=="], - - "@rollup/rollup-darwin-x64": ["@rollup/rollup-darwin-x64@4.53.5", "", { "os": "darwin", "cpu": "x64" }, "sha512-YTbnsAaHo6VrAczISxgpTva8EkfQus0VPEVJCEaboHtZRIb6h6j0BNxRBOwnDciFTZLDPW5r+ZBmhL/+YpTZgA=="], - - "@rollup/rollup-freebsd-arm64": ["@rollup/rollup-freebsd-arm64@4.53.5", "", { "os": "freebsd", "cpu": "arm64" }, "sha512-1T8eY2J8rKJWzaznV7zedfdhD1BqVs1iqILhmHDq/bqCUZsrMt+j8VCTHhP0vdfbHK3e1IQ7VYx3jlKqwlf+vw=="], - - "@rollup/rollup-freebsd-x64": ["@rollup/rollup-freebsd-x64@4.53.5", "", { "os": "freebsd", "cpu": "x64" }, "sha512-sHTiuXyBJApxRn+VFMaw1U+Qsz4kcNlxQ742snICYPrY+DDL8/ZbaC4DVIB7vgZmp3jiDaKA0WpBdP0aqPJoBQ=="], - - "@rollup/rollup-linux-arm-gnueabihf": ["@rollup/rollup-linux-arm-gnueabihf@4.53.5", "", { "os": "linux", "cpu": "arm" }, "sha512-dV3T9MyAf0w8zPVLVBptVlzaXxka6xg1f16VAQmjg+4KMSTWDvhimI/Y6mp8oHwNrmnmVl9XxJ/w/mO4uIQONA=="], - - "@rollup/rollup-linux-arm-musleabihf": ["@rollup/rollup-linux-arm-musleabihf@4.53.5", "", { "os": "linux", "cpu": "arm" }, "sha512-wIGYC1x/hyjP+KAu9+ewDI+fi5XSNiUi9Bvg6KGAh2TsNMA3tSEs+Sh6jJ/r4BV/bx/CyWu2ue9kDnIdRyafcQ=="], - - "@rollup/rollup-linux-arm64-gnu": ["@rollup/rollup-linux-arm64-gnu@4.53.5", "", { "os": "linux", "cpu": "arm64" }, "sha512-Y+qVA0D9d0y2FRNiG9oM3Hut/DgODZbU9I8pLLPwAsU0tUKZ49cyV1tzmB/qRbSzGvY8lpgGkJuMyuhH7Ma+Vg=="], - - "@rollup/rollup-linux-arm64-musl": ["@rollup/rollup-linux-arm64-musl@4.53.5", "", { "os": "linux", "cpu": "arm64" }, "sha512-juaC4bEgJsyFVfqhtGLz8mbopaWD+WeSOYr5E16y+1of6KQjc0BpwZLuxkClqY1i8sco+MdyoXPNiCkQou09+g=="], - - "@rollup/rollup-linux-loong64-gnu": ["@rollup/rollup-linux-loong64-gnu@4.53.5", "", { "os": "linux", "cpu": "none" }, "sha512-rIEC0hZ17A42iXtHX+EPJVL/CakHo+tT7W0pbzdAGuWOt2jxDFh7A/lRhsNHBcqL4T36+UiAgwO8pbmn3dE8wA=="], - - "@rollup/rollup-linux-ppc64-gnu": ["@rollup/rollup-linux-ppc64-gnu@4.53.5", "", { "os": "linux", "cpu": "ppc64" }, "sha512-T7l409NhUE552RcAOcmJHj3xyZ2h7vMWzcwQI0hvn5tqHh3oSoclf9WgTl+0QqffWFG8MEVZZP1/OBglKZx52Q=="], - - "@rollup/rollup-linux-riscv64-gnu": ["@rollup/rollup-linux-riscv64-gnu@4.53.5", "", { "os": "linux", "cpu": "none" }, "sha512-7OK5/GhxbnrMcxIFoYfhV/TkknarkYC1hqUw1wU2xUN3TVRLNT5FmBv4KkheSG2xZ6IEbRAhTooTV2+R5Tk0lQ=="], - - "@rollup/rollup-linux-riscv64-musl": ["@rollup/rollup-linux-riscv64-musl@4.53.5", "", { "os": "linux", "cpu": "none" }, "sha512-GwuDBE/PsXaTa76lO5eLJTyr2k8QkPipAyOrs4V/KJufHCZBJ495VCGJol35grx9xryk4V+2zd3Ri+3v7NPh+w=="], - - "@rollup/rollup-linux-s390x-gnu": ["@rollup/rollup-linux-s390x-gnu@4.53.5", "", { "os": "linux", "cpu": "s390x" }, "sha512-IAE1Ziyr1qNfnmiQLHBURAD+eh/zH1pIeJjeShleII7Vj8kyEm2PF77o+lf3WTHDpNJcu4IXJxNO0Zluro8bOw=="], - - "@rollup/rollup-linux-x64-gnu": ["@rollup/rollup-linux-x64-gnu@4.53.5", "", { "os": "linux", "cpu": "x64" }, "sha512-Pg6E+oP7GvZ4XwgRJBuSXZjcqpIW3yCBhK4BcsANvb47qMvAbCjR6E+1a/U2WXz1JJxp9/4Dno3/iSJLcm5auw=="], - - "@rollup/rollup-linux-x64-musl": ["@rollup/rollup-linux-x64-musl@4.53.5", "", { "os": "linux", "cpu": "x64" }, "sha512-txGtluxDKTxaMDzUduGP0wdfng24y1rygUMnmlUJ88fzCCULCLn7oE5kb2+tRB+MWq1QDZT6ObT5RrR8HFRKqg=="], - - "@rollup/rollup-openharmony-arm64": ["@rollup/rollup-openharmony-arm64@4.53.5", "", { "os": "none", "cpu": "arm64" }, "sha512-3DFiLPnTxiOQV993fMc+KO8zXHTcIjgaInrqlG8zDp1TlhYl6WgrOHuJkJQ6M8zHEcntSJsUp1XFZSY8C1DYbg=="], - - "@rollup/rollup-win32-arm64-msvc": ["@rollup/rollup-win32-arm64-msvc@4.53.5", "", { "os": "win32", "cpu": "arm64" }, "sha512-nggc/wPpNTgjGg75hu+Q/3i32R00Lq1B6N1DO7MCU340MRKL3WZJMjA9U4K4gzy3dkZPXm9E1Nc81FItBVGRlA=="], - - "@rollup/rollup-win32-ia32-msvc": ["@rollup/rollup-win32-ia32-msvc@4.53.5", "", { "os": "win32", "cpu": "ia32" }, "sha512-U/54pTbdQpPLBdEzCT6NBCFAfSZMvmjr0twhnD9f4EIvlm9wy3jjQ38yQj1AGznrNO65EWQMgm/QUjuIVrYF9w=="], - - "@rollup/rollup-win32-x64-gnu": ["@rollup/rollup-win32-x64-gnu@4.53.5", "", { "os": "win32", "cpu": "x64" }, "sha512-2NqKgZSuLH9SXBBV2dWNRCZmocgSOx8OJSdpRaEcRlIfX8YrKxUT6z0F1NpvDVhOsl190UFTRh2F2WDWWCYp3A=="], - - "@rollup/rollup-win32-x64-msvc": ["@rollup/rollup-win32-x64-msvc@4.53.5", "", { "os": "win32", "cpu": "x64" }, "sha512-JRpZUhCfhZ4keB5v0fe02gQJy05GqboPOaxvjugW04RLSYYoB/9t2lx2u/tMs/Na/1NXfY8QYjgRljRpN+MjTQ=="], - - "@smithy/abort-controller": ["@smithy/abort-controller@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-P7JD4J+wxHMpGxqIg6SHno2tPkZbBUBLbPpR5/T1DEUvw/mEaINBMaPFZNM7lA+ToSCZ36j6nMHa+5kej+fhGg=="], - - "@smithy/chunked-blob-reader": ["@smithy/chunked-blob-reader@5.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-WmU0TnhEAJLWvfSeMxBNe5xtbselEO8+4wG0NtZeL8oR21WgH1xiO37El+/Y+H/Ie4SCwBy3MxYWmOYaGgZueA=="], - - "@smithy/chunked-blob-reader-native": ["@smithy/chunked-blob-reader-native@4.2.1", "", { "dependencies": { "@smithy/util-base64": "^4.3.0", "tslib": "^2.6.2" } }, "sha512-lX9Ay+6LisTfpLid2zZtIhSEjHMZoAR5hHCR4H7tBz/Zkfr5ea8RcQ7Tk4mi0P76p4cN+Btz16Ffno7YHpKXnQ=="], - - "@smithy/config-resolver": ["@smithy/config-resolver@4.4.4", "", { "dependencies": { "@smithy/node-config-provider": "^4.3.6", "@smithy/types": "^4.10.0", "@smithy/util-config-provider": "^4.2.0", "@smithy/util-endpoints": "^3.2.6", "@smithy/util-middleware": "^4.2.6", "tslib": "^2.6.2" } }, "sha512-s3U5ChS21DwU54kMmZ0UJumoS5cg0+rGVZvN6f5Lp6EbAVi0ZyP+qDSHdewfmXKUgNK1j3z45JyzulkDukrjAA=="], - - "@smithy/core": ["@smithy/core@3.19.0", "", { "dependencies": { "@smithy/middleware-serde": "^4.2.7", "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "@smithy/util-base64": "^4.3.0", "@smithy/util-body-length-browser": "^4.2.0", "@smithy/util-middleware": "^4.2.6", "@smithy/util-stream": "^4.5.7", "@smithy/util-utf8": "^4.2.0", "@smithy/uuid": "^1.1.0", "tslib": "^2.6.2" } }, "sha512-Y9oHXpBcXQgYHOcAEmxjkDilUbSTkgKjoHYed3WaYUH8jngq8lPWDBSpjHblJ9uOgBdy5mh3pzebrScDdYr29w=="], - - "@smithy/credential-provider-imds": ["@smithy/credential-provider-imds@4.2.6", "", { "dependencies": { "@smithy/node-config-provider": "^4.3.6", "@smithy/property-provider": "^4.2.6", "@smithy/types": "^4.10.0", "@smithy/url-parser": "^4.2.6", "tslib": "^2.6.2" } }, "sha512-xBmawExyTzOjbhzkZwg+vVm/khg28kG+rj2sbGlULjFd1jI70sv/cbpaR0Ev4Yfd6CpDUDRMe64cTqR//wAOyA=="], - - "@smithy/eventstream-codec": ["@smithy/eventstream-codec@4.2.6", "", { "dependencies": { "@aws-crypto/crc32": "5.2.0", "@smithy/types": "^4.10.0", "@smithy/util-hex-encoding": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-OZfsI+YRG26XZik/jKMMg37acnBSbUiK/8nETW3uM3mLj+0tMmFXdHQw1e5WEd/IHN8BGOh3te91SNDe2o4RHg=="], - - "@smithy/eventstream-serde-browser": ["@smithy/eventstream-serde-browser@4.2.6", "", { "dependencies": { "@smithy/eventstream-serde-universal": "^4.2.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-6OiaAaEbLB6dEkRbQyNzFSJv5HDvly3Mc6q/qcPd2uS/g3szR8wAIkh7UndAFKfMypNSTuZ6eCBmgCLR5LacTg=="], - - "@smithy/eventstream-serde-config-resolver": ["@smithy/eventstream-serde-config-resolver@4.3.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-xP5YXbOVRVN8A4pDnSUkEUsL9fYFU6VNhxo8tgr13YnMbf3Pn4xVr+hSyLVjS1Frfi1Uk03ET5Bwml4+0CeYEw=="], - - "@smithy/eventstream-serde-node": ["@smithy/eventstream-serde-node@4.2.6", "", { "dependencies": { "@smithy/eventstream-serde-universal": "^4.2.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-jhH7nJuaOpnTFcuZpWK9dqb6Ge2yGi1okTo0W6wkJrfwAm2vwmO74tF1v07JmrSyHBcKLQATEexclJw9K1Vj7w=="], - - "@smithy/eventstream-serde-universal": ["@smithy/eventstream-serde-universal@4.2.6", "", { "dependencies": { "@smithy/eventstream-codec": "^4.2.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-olIfZ230B64TvPD6b0tPvrEp2eB0FkyL3KvDlqF4RVmIc/kn3orzXnV6DTQdOOW5UU+M5zKY3/BU47X420/oPw=="], - - "@smithy/fetch-http-handler": ["@smithy/fetch-http-handler@5.3.7", "", { "dependencies": { "@smithy/protocol-http": "^5.3.6", "@smithy/querystring-builder": "^4.2.6", "@smithy/types": "^4.10.0", "@smithy/util-base64": "^4.3.0", "tslib": "^2.6.2" } }, "sha512-fcVap4QwqmzQwQK9QU3keeEpCzTjnP9NJ171vI7GnD7nbkAIcP9biZhDUx88uRH9BabSsQDS0unUps88uZvFIQ=="], - - "@smithy/hash-blob-browser": ["@smithy/hash-blob-browser@4.2.7", "", { "dependencies": { "@smithy/chunked-blob-reader": "^5.2.0", "@smithy/chunked-blob-reader-native": "^4.2.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-CIbCTGGX5CI7tfewBPSYD9ycp2Vb2GW5xnXD1n7GcO9mu37EN7A6DvCHM9MX7pOeS1adMn5D+1yRwI3eABVbcA=="], - - "@smithy/hash-node": ["@smithy/hash-node@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "@smithy/util-buffer-from": "^4.2.0", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-k3Dy9VNR37wfMh2/1RHkFf/e0rMyN0pjY0FdyY6ItJRjENYyVPRMwad6ZR1S9HFm6tTuIOd9pqKBmtJ4VHxvxg=="], - - "@smithy/hash-stream-node": ["@smithy/hash-stream-node@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-+3T8LkH39YIhYHsv/Ec8lF+92nykZpU+XMBvAyXF/uLcTp86pxa5oSJk1vzaRY9N++qgDLYjzJ6OVbtAgDGwfw=="], - - "@smithy/invalid-dependency": ["@smithy/invalid-dependency@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-E4t/V/q2T46RY21fpfznd1iSLTvCXKNKo4zJ1QuEFN4SE9gKfu2vb6bgq35LpufkQ+SETWIC7ZAf2GGvTlBaMQ=="], - - "@smithy/is-array-buffer": ["@smithy/is-array-buffer@4.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-DZZZBvC7sjcYh4MazJSGiWMI2L7E0oCiRHREDzIxi/M2LY79/21iXt6aPLHge82wi5LsuRF5A06Ds3+0mlh6CQ=="], - - "@smithy/md5-js": ["@smithy/md5-js@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-ZXeh8UmH31JdcNsrQ1o9v1IVuva9JFwxIc6zTMxWX7wcmWvVR7Ai9aUEw5LraNKqdkAsb06clpM2sRH4Iy55Sg=="], - - "@smithy/middleware-content-length": ["@smithy/middleware-content-length@4.2.6", "", { "dependencies": { "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-0cjqjyfj+Gls30ntq45SsBtqF3dfJQCeqQPyGz58Pk8OgrAr5YiB7ZvDzjCA94p4r6DCI4qLm7FKobqBjf515w=="], - - "@smithy/middleware-endpoint": ["@smithy/middleware-endpoint@4.4.0", "", { "dependencies": { "@smithy/core": "^3.19.0", "@smithy/middleware-serde": "^4.2.7", "@smithy/node-config-provider": "^4.3.6", "@smithy/shared-ini-file-loader": "^4.4.1", "@smithy/types": "^4.10.0", "@smithy/url-parser": "^4.2.6", "@smithy/util-middleware": "^4.2.6", "tslib": "^2.6.2" } }, "sha512-M6qWfUNny6NFNy8amrCGIb9TfOMUkHVtg9bHtEFGRgfH7A7AtPpn/fcrToGPjVDK1ECuMVvqGQOXcZxmu9K+7A=="], - - "@smithy/middleware-retry": ["@smithy/middleware-retry@4.4.16", "", { "dependencies": { "@smithy/node-config-provider": "^4.3.6", "@smithy/protocol-http": "^5.3.6", "@smithy/service-error-classification": "^4.2.6", "@smithy/smithy-client": "^4.10.1", "@smithy/types": "^4.10.0", "@smithy/util-middleware": "^4.2.6", "@smithy/util-retry": "^4.2.6", "@smithy/uuid": "^1.1.0", "tslib": "^2.6.2" } }, "sha512-XPpNhNRzm3vhYm7YCsyw3AtmWggJbg1wNGAoqb7NBYr5XA5isMRv14jgbYyUV6IvbTBFZQdf2QpeW43LrRdStQ=="], - - "@smithy/middleware-serde": ["@smithy/middleware-serde@4.2.7", "", { "dependencies": { "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-PFMVHVPgtFECeu4iZ+4SX6VOQT0+dIpm4jSPLLL6JLSkp9RohGqKBKD0cbiXdeIFS08Forp0UHI6kc0gIHenSA=="], - - "@smithy/middleware-stack": ["@smithy/middleware-stack@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-JSbALU3G+JS4kyBZPqnJ3hxIYwOVRV7r9GNQMS6j5VsQDo5+Es5nddLfr9TQlxZLNHPvKSh+XSB0OuWGfSWFcA=="], - - "@smithy/node-config-provider": ["@smithy/node-config-provider@4.3.6", "", { "dependencies": { "@smithy/property-provider": "^4.2.6", "@smithy/shared-ini-file-loader": "^4.4.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-fYEyL59Qe82Ha1p97YQTMEQPJYmBS+ux76foqluaTVWoG9Px5J53w6NvXZNE3wP7lIicLDF7Vj1Em18XTX7fsA=="], - - "@smithy/node-http-handler": ["@smithy/node-http-handler@4.4.6", "", { "dependencies": { "@smithy/abort-controller": "^4.2.6", "@smithy/protocol-http": "^5.3.6", "@smithy/querystring-builder": "^4.2.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-Gsb9jf4ido5BhPfani4ggyrKDd3ZK+vTFWmUaZeFg5G3E5nhFmqiTzAIbHqmPs1sARuJawDiGMGR/nY+Gw6+aQ=="], - - "@smithy/property-provider": ["@smithy/property-provider@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-a/tGSLPtaia2krbRdwR4xbZKO8lU67DjMk/jfY4QKt4PRlKML+2tL/gmAuhNdFDioO6wOq0sXkfnddNFH9mNUA=="], - - "@smithy/protocol-http": ["@smithy/protocol-http@5.3.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-qLRZzP2+PqhE3OSwvY2jpBbP0WKTZ9opTsn+6IWYI0SKVpbG+imcfNxXPq9fj5XeaUTr7odpsNpK6dmoiM1gJQ=="], - - "@smithy/querystring-builder": ["@smithy/querystring-builder@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "@smithy/util-uri-escape": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-MeM9fTAiD3HvoInK/aA8mgJaKQDvm8N0dKy6EiFaCfgpovQr4CaOkJC28XqlSRABM+sHdSQXbC8NZ0DShBMHqg=="], - - "@smithy/querystring-parser": ["@smithy/querystring-parser@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-YmWxl32SQRw/kIRccSOxzS/Ib8/b5/f9ex0r5PR40jRJg8X1wgM3KrR2In+8zvOGVhRSXgvyQpw9yOSlmfmSnA=="], - - "@smithy/service-error-classification": ["@smithy/service-error-classification@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0" } }, "sha512-Q73XBrzJlGTut2nf5RglSntHKgAG0+KiTJdO5QQblLfr4TdliGwIAha1iZIjwisc3rA5ulzqwwsYC6xrclxVQg=="], - - "@smithy/shared-ini-file-loader": ["@smithy/shared-ini-file-loader@4.4.1", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-tph+oQYPbpN6NamF030hx1gb5YN2Plog+GLaRHpoEDwp8+ZPG26rIJvStG9hkWzN2HBn3HcWg0sHeB0tmkYzqA=="], - - "@smithy/signature-v4": ["@smithy/signature-v4@5.3.6", "", { "dependencies": { "@smithy/is-array-buffer": "^4.2.0", "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "@smithy/util-hex-encoding": "^4.2.0", "@smithy/util-middleware": "^4.2.6", "@smithy/util-uri-escape": "^4.2.0", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-P1TXDHuQMadTMTOBv4oElZMURU4uyEhxhHfn+qOc2iofW9Rd4sZtBGx58Lzk112rIGVEYZT8eUMK4NftpewpRA=="], - - "@smithy/smithy-client": ["@smithy/smithy-client@4.10.1", "", { "dependencies": { "@smithy/core": "^3.19.0", "@smithy/middleware-endpoint": "^4.4.0", "@smithy/middleware-stack": "^4.2.6", "@smithy/protocol-http": "^5.3.6", "@smithy/types": "^4.10.0", "@smithy/util-stream": "^4.5.7", "tslib": "^2.6.2" } }, "sha512-1ovWdxzYprhq+mWqiGZlt3kF69LJthuQcfY9BIyHx9MywTFKzFapluku1QXoaBB43GCsLDxNqS+1v30ure69AA=="], - - "@smithy/types": ["@smithy/types@4.10.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-K9mY7V/f3Ul+/Gz4LJANZ3vJ/yiBIwCyxe0sPT4vNJK63Srvd+Yk1IzP0t+nE7XFSpIGtzR71yljtnqpUTYFlQ=="], - - "@smithy/url-parser": ["@smithy/url-parser@4.2.6", "", { "dependencies": { "@smithy/querystring-parser": "^4.2.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-tVoyzJ2vXp4R3/aeV4EQjBDmCuWxRa8eo3KybL7Xv4wEM16nObYh7H1sNfcuLWHAAAzb0RVyxUz1S3sGj4X+Tg=="], - - "@smithy/util-base64": ["@smithy/util-base64@4.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^4.2.0", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-GkXZ59JfyxsIwNTWFnjmFEI8kZpRNIBfxKjv09+nkAWPt/4aGaEWMM04m4sxgNVWkbt2MdSvE3KF/PfX4nFedQ=="], - - "@smithy/util-body-length-browser": ["@smithy/util-body-length-browser@4.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-Fkoh/I76szMKJnBXWPdFkQJl2r9SjPt3cMzLdOB6eJ4Pnpas8hVoWPYemX/peO0yrrvldgCUVJqOAjUrOLjbxg=="], - - "@smithy/util-body-length-node": ["@smithy/util-body-length-node@4.2.1", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-h53dz/pISVrVrfxV1iqXlx5pRg3V2YWFcSQyPyXZRrZoZj4R4DeWRDo1a7dd3CPTcFi3kE+98tuNyD2axyZReA=="], - - "@smithy/util-buffer-from": ["@smithy/util-buffer-from@4.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-kAY9hTKulTNevM2nlRtxAG2FQ3B2OR6QIrPY3zE5LqJy1oxzmgBGsHLWTcNhWXKchgA0WHW+mZkQrng/pgcCew=="], - - "@smithy/util-config-provider": ["@smithy/util-config-provider@4.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-YEjpl6XJ36FTKmD+kRJJWYvrHeUvm5ykaUS5xK+6oXffQPHeEM4/nXlZPe+Wu0lsgRUcNZiliYNh/y7q9c2y6Q=="], - - "@smithy/util-defaults-mode-browser": ["@smithy/util-defaults-mode-browser@4.3.15", "", { "dependencies": { "@smithy/property-provider": "^4.2.6", "@smithy/smithy-client": "^4.10.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-LiZQVAg/oO8kueX4c+oMls5njaD2cRLXRfcjlTYjhIqmwHnCwkQO5B3dMQH0c5PACILxGAQf6Mxsq7CjlDc76A=="], - - "@smithy/util-defaults-mode-node": ["@smithy/util-defaults-mode-node@4.2.18", "", { "dependencies": { "@smithy/config-resolver": "^4.4.4", "@smithy/credential-provider-imds": "^4.2.6", "@smithy/node-config-provider": "^4.3.6", "@smithy/property-provider": "^4.2.6", "@smithy/smithy-client": "^4.10.1", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-Kw2J+KzYm9C9Z9nY6+W0tEnoZOofstVCMTshli9jhQbQCy64rueGfKzPfuFBnVUqZD9JobxTh2DzHmPkp/Va/Q=="], - - "@smithy/util-endpoints": ["@smithy/util-endpoints@3.2.6", "", { "dependencies": { "@smithy/node-config-provider": "^4.3.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-v60VNM2+mPvgHCBXEfMCYrQ0RepP6u6xvbAkMenfe4Mi872CqNkJzgcnQL837e8NdeDxBgrWQRTluKq5Lqdhfg=="], - - "@smithy/util-hex-encoding": ["@smithy/util-hex-encoding@4.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-CCQBwJIvXMLKxVbO88IukazJD9a4kQ9ZN7/UMGBjBcJYvatpWk+9g870El4cB8/EJxfe+k+y0GmR9CAzkF+Nbw=="], - - "@smithy/util-middleware": ["@smithy/util-middleware@4.2.6", "", { "dependencies": { "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-qrvXUkxBSAFomM3/OEMuDVwjh4wtqK8D2uDZPShzIqOylPst6gor2Cdp6+XrH4dyksAWq/bE2aSDYBTTnj0Rxg=="], - - "@smithy/util-retry": ["@smithy/util-retry@4.2.6", "", { "dependencies": { "@smithy/service-error-classification": "^4.2.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-x7CeDQLPQ9cb6xN7fRJEjlP9NyGW/YeXWc4j/RUhg4I+H60F0PEeRc2c/z3rm9zmsdiMFzpV/rT+4UHW6KM1SA=="], - - "@smithy/util-stream": ["@smithy/util-stream@4.5.7", "", { "dependencies": { "@smithy/fetch-http-handler": "^5.3.7", "@smithy/node-http-handler": "^4.4.6", "@smithy/types": "^4.10.0", "@smithy/util-base64": "^4.3.0", "@smithy/util-buffer-from": "^4.2.0", "@smithy/util-hex-encoding": "^4.2.0", "@smithy/util-utf8": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-Uuy4S5Aj4oF6k1z+i2OtIBJUns4mlg29Ph4S+CqjR+f4XXpSFVgTCYLzMszHJTicYDBxKFtwq2/QSEDSS5l02A=="], - - "@smithy/util-uri-escape": ["@smithy/util-uri-escape@4.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-igZpCKV9+E/Mzrpq6YacdTQ0qTiLm85gD6N/IrmyDvQFA4UnU3d5g3m8tMT/6zG/vVkWSU+VxeUyGonL62DuxA=="], - - "@smithy/util-utf8": ["@smithy/util-utf8@4.2.0", "", { "dependencies": { "@smithy/util-buffer-from": "^4.2.0", "tslib": "^2.6.2" } }, "sha512-zBPfuzoI8xyBtR2P6WQj63Rz8i3AmfAaJLuNG8dWsfvPe8lO4aCPYLn879mEgHndZH1zQ2oXmG8O1GGzzaoZiw=="], - - "@smithy/util-waiter": ["@smithy/util-waiter@4.2.6", "", { "dependencies": { "@smithy/abort-controller": "^4.2.6", "@smithy/types": "^4.10.0", "tslib": "^2.6.2" } }, "sha512-xU9HwUSik9UUCJmm530yvBy0AwlQFICveKmqvaaTukKkXEAhyiBdHtSrhPrH3rH+uz0ykyaE3LdgsX86C6mDCQ=="], - - "@smithy/uuid": ["@smithy/uuid@1.1.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-4aUIteuyxtBUhVdiQqcDhKFitwfd9hqoSDYY2KRXiWtgoWJ9Bmise+KfEPDiVHWeJepvF8xJO9/9+WDIciMFFw=="], - - "@testcontainers/redis": ["@testcontainers/redis@11.10.0", "", { "dependencies": { "testcontainers": "^11.10.0" } }, "sha512-w/Hnv1IH8jJ4wjIgpSzoll1KABz2L28+i6JAZVSZuSzQPqeTeFa3mZHnRcdKJggjEIMDwpFlqjGXYRYKNAk0Fw=="], - - "@tootallnate/once": ["@tootallnate/once@2.0.0", "", {}, "sha512-XCuKFP5PS55gnMVu3dty8KPatLqUoy/ZYzDzAGCQ8JNFCkLXzmI7vNHCR+XpbZaMWQK/vQubr7PkYq8g470J/A=="], - - "@types/caseless": ["@types/caseless@0.12.5", "", {}, "sha512-hWtVTC2q7hc7xZ/RLbxapMvDMgUnDvKvMOpKal4DrMyfGBUfB1oKaZlIRr6mJL+If3bAP6sV/QneGzF6tJjZDg=="], - - "@types/chai": ["@types/chai@5.2.3", "", { "dependencies": { "@types/deep-eql": "*", "assertion-error": "^2.0.1" } }, "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA=="], - - "@types/deep-eql": ["@types/deep-eql@4.0.2", "", {}, "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw=="], - - "@types/docker-modem": ["@types/docker-modem@3.0.6", "", { "dependencies": { "@types/node": "*", "@types/ssh2": "*" } }, "sha512-yKpAGEuKRSS8wwx0joknWxsmLha78wNMe9R2S3UNsVOkZded8UqOrV8KoeDXoXsjndxwyF3eIhyClGbO1SEhEg=="], - - "@types/dockerode": ["@types/dockerode@3.3.47", "", { "dependencies": { "@types/docker-modem": "*", "@types/node": "*", "@types/ssh2": "*" } }, "sha512-ShM1mz7rCjdssXt7Xz0u1/R2BJC7piWa3SJpUBiVjCf2A3XNn4cP6pUVaD8bLanpPVVn4IKzJuw3dOvkJ8IbYw=="], - - "@types/estree": ["@types/estree@1.0.8", "", {}, "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w=="], - - "@types/js-yaml": ["@types/js-yaml@4.0.9", "", {}, "sha512-k4MGaQl5TGo/iipqb2UDG2UwjXziSWkh0uysQelTlJpX1qGlpUZYm8PnO4DxG1qBomtJUdYJ6qR6xdIah10JLg=="], - - "@types/json-schema": ["@types/json-schema@7.0.15", "", {}, "sha512-5+fP8P8MFNC+AyZCDxrB2pkZFPGzqQWUzpSeuuVLvm8VMcorNYavBqoFcxK8bQz4Qsbn4oUEEem4wDLfcysGHA=="], - - "@types/mime": ["@types/mime@3.0.4", "", {}, "sha512-iJt33IQnVRkqeqC7PzBHPTC6fDlRNRW8vjrgqtScAhrmMwe8c4Eo7+fUGTa+XdWrpEgpyKWMYmi2dIwMAYRzPw=="], - - "@types/minimist": ["@types/minimist@1.2.5", "", {}, "sha512-hov8bUuiLiyFPGyFPE1lwWhmzYbirOXQNNo40+y3zow8aFVTeyn3VWL0VFFfdNddA8S4Vf0Tc062rzyNr7Paag=="], - - "@types/needle": ["@types/needle@3.3.0", "", { "dependencies": { "@types/node": "*" } }, "sha512-UFIuc1gdyzAqeVUYpSL+cliw2MmU/ZUhVZKE7Zo4wPbgc8hbljeKSnn6ls6iG8r5jpegPXLUIhJ+Wb2kLVs8cg=="], - - "@types/node": ["@types/node@20.19.27", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-N2clP5pJhB2YnZJ3PIHFk5RkygRX5WO/5f0WC08tp0wd+sv0rsJk3MqWn3CbNmT2J505a5336jaQj4ph1AdMug=="], - - "@types/normalize-package-data": ["@types/normalize-package-data@2.4.4", "", {}, "sha512-37i+OaWTh9qeK4LSHPsyRC7NahnGotNuZvjLSgcPzblpHB3rrCJxAOgI5gCdKm7coonsaX1Of0ILiTcnZjbfxA=="], - - "@types/pako": ["@types/pako@2.0.4", "", {}, "sha512-VWDCbrLeVXJM9fihYodcLiIv0ku+AlOa/TQ1SvYOaBuyrSKgEcro95LJyIsJ4vSo6BXIxOKxiJAat04CmST9Fw=="], - - "@types/probe-image-size": ["@types/probe-image-size@7.2.5", "", { "dependencies": { "@types/needle": "*", "@types/node": "*" } }, "sha512-9Bg6d/GNnjmhMMxadDstwrSlquuuLf0jQuPszbU6n3QUfybif3V/ryD3J2i9iaiC5JB/FU/8E41n88SM/UB+Tg=="], - - "@types/raf": ["@types/raf@3.4.3", "", {}, "sha512-c4YAvMedbPZ5tEyxzQdMoOhhJ4RD3rngZIdwC2/qDN3d7JpEhB6fiBRKVY1lg5B7Wk+uPBjn5f39j1/2MY1oOw=="], - - "@types/request": ["@types/request@2.48.13", "", { "dependencies": { "@types/caseless": "*", "@types/node": "*", "@types/tough-cookie": "*", "form-data": "^2.5.5" } }, "sha512-FGJ6udDNUCjd19pp0Q3iTiDkwhYup7J8hpMW9c4k53NrccQFFWKRho6hvtPPEhnXWKvukfwAlB6DbDz4yhH5Gg=="], - - "@types/resolve": ["@types/resolve@1.20.2", "", {}, "sha512-60BCwRFOZCQhDncwQdxxeOEEkbc5dIMccYLwbxsS4TUNeVECQ/pBJ0j09mrHOl/JJvpRPGwO9SvE4nR2Nb/a4Q=="], - - "@types/ssh2": ["@types/ssh2@1.15.5", "", { "dependencies": { "@types/node": "^18.11.18" } }, "sha512-N1ASjp/nXH3ovBHddRJpli4ozpk6UdDYIX4RJWFa9L1YKnzdhTlVmiGHm4DZnj/jLbqZpes4aeR30EFGQtvhQQ=="], - - "@types/ssh2-streams": ["@types/ssh2-streams@0.1.13", "", { "dependencies": { "@types/node": "*" } }, "sha512-faHyY3brO9oLEA0QlcO8N2wT7R0+1sHWZvQ+y3rMLwdY1ZyS1z0W3t65j9PqT4HmQ6ALzNe7RZlNuCNE0wBSWA=="], - - "@types/tough-cookie": ["@types/tough-cookie@4.0.5", "", {}, "sha512-/Ad8+nIOV7Rl++6f1BdKxFSMgmoqEoYbHRpPcx3JEfv8VRsQe9Z4mCXeJBzxs7mbHY/XOZZuXlRNfhpVPbs6ZA=="], - - "@types/trusted-types": ["@types/trusted-types@2.0.7", "", {}, "sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw=="], - - "@types/uuid": ["@types/uuid@10.0.0", "", {}, "sha512-7gqG38EyHgyP1S+7+xomFtL+ZNHcKv6DwNaCZmJmo1vgMugyF3TCnXVg4t1uk89mLNwnLtnY3TpOpCOyp1/xHQ=="], - - "@types/ws": ["@types/ws@8.18.1", "", { "dependencies": { "@types/node": "*" } }, "sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg=="], - - "@typescript-eslint/eslint-plugin": ["@typescript-eslint/eslint-plugin@8.50.0", "", { "dependencies": { "@eslint-community/regexpp": "^4.10.0", "@typescript-eslint/scope-manager": "8.50.0", "@typescript-eslint/type-utils": "8.50.0", "@typescript-eslint/utils": "8.50.0", "@typescript-eslint/visitor-keys": "8.50.0", "ignore": "^7.0.0", "natural-compare": "^1.4.0", "ts-api-utils": "^2.1.0" }, "peerDependencies": { "@typescript-eslint/parser": "^8.50.0", "eslint": "^8.57.0 || ^9.0.0", "typescript": ">=4.8.4 <6.0.0" } }, "sha512-O7QnmOXYKVtPrfYzMolrCTfkezCJS9+ljLdKW/+DCvRsc3UAz+sbH6Xcsv7p30+0OwUbeWfUDAQE0vpabZ3QLg=="], - - "@typescript-eslint/parser": ["@typescript-eslint/parser@8.50.0", "", { "dependencies": { "@typescript-eslint/scope-manager": "8.50.0", "@typescript-eslint/types": "8.50.0", "@typescript-eslint/typescript-estree": "8.50.0", "@typescript-eslint/visitor-keys": "8.50.0", "debug": "^4.3.4" }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0", "typescript": ">=4.8.4 <6.0.0" } }, "sha512-6/cmF2piao+f6wSxUsJLZjck7OQsYyRtcOZS02k7XINSNlz93v6emM8WutDQSXnroG2xwYlEVHJI+cPA7CPM3Q=="], - - "@typescript-eslint/project-service": ["@typescript-eslint/project-service@8.50.0", "", { "dependencies": { "@typescript-eslint/tsconfig-utils": "^8.50.0", "@typescript-eslint/types": "^8.50.0", "debug": "^4.3.4" }, "peerDependencies": { "typescript": ">=4.8.4 <6.0.0" } }, "sha512-Cg/nQcL1BcoTijEWyx4mkVC56r8dj44bFDvBdygifuS20f3OZCHmFbjF34DPSi07kwlFvqfv/xOLnJ5DquxSGQ=="], - - "@typescript-eslint/scope-manager": ["@typescript-eslint/scope-manager@8.50.0", "", { "dependencies": { "@typescript-eslint/types": "8.50.0", "@typescript-eslint/visitor-keys": "8.50.0" } }, "sha512-xCwfuCZjhIqy7+HKxBLrDVT5q/iq7XBVBXLn57RTIIpelLtEIZHXAF/Upa3+gaCpeV1NNS5Z9A+ID6jn50VD4A=="], - - "@typescript-eslint/tsconfig-utils": ["@typescript-eslint/tsconfig-utils@8.50.0", "", { "peerDependencies": { "typescript": ">=4.8.4 <6.0.0" } }, "sha512-vxd3G/ybKTSlm31MOA96gqvrRGv9RJ7LGtZCn2Vrc5htA0zCDvcMqUkifcjrWNNKXHUU3WCkYOzzVSFBd0wa2w=="], - - "@typescript-eslint/type-utils": ["@typescript-eslint/type-utils@8.50.0", "", { "dependencies": { "@typescript-eslint/types": "8.50.0", "@typescript-eslint/typescript-estree": "8.50.0", "@typescript-eslint/utils": "8.50.0", "debug": "^4.3.4", "ts-api-utils": "^2.1.0" }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0", "typescript": ">=4.8.4 <6.0.0" } }, "sha512-7OciHT2lKCewR0mFoBrvZJ4AXTMe/sYOe87289WAViOocEmDjjv8MvIOT2XESuKj9jp8u3SZYUSh89QA4S1kQw=="], - - "@typescript-eslint/types": ["@typescript-eslint/types@8.50.0", "", {}, "sha512-iX1mgmGrXdANhhITbpp2QQM2fGehBse9LbTf0sidWK6yg/NE+uhV5dfU1g6EYPlcReYmkE9QLPq/2irKAmtS9w=="], - - "@typescript-eslint/typescript-estree": ["@typescript-eslint/typescript-estree@8.50.0", "", { "dependencies": { "@typescript-eslint/project-service": "8.50.0", "@typescript-eslint/tsconfig-utils": "8.50.0", "@typescript-eslint/types": "8.50.0", "@typescript-eslint/visitor-keys": "8.50.0", "debug": "^4.3.4", "minimatch": "^9.0.4", "semver": "^7.6.0", "tinyglobby": "^0.2.15", "ts-api-utils": "^2.1.0" }, "peerDependencies": { "typescript": ">=4.8.4 <6.0.0" } }, "sha512-W7SVAGBR/IX7zm1t70Yujpbk+zdPq/u4soeFSknWFdXIFuWsBGBOUu/Tn/I6KHSKvSh91OiMuaSnYp3mtPt5IQ=="], - - "@typescript-eslint/utils": ["@typescript-eslint/utils@8.50.0", "", { "dependencies": { "@eslint-community/eslint-utils": "^4.7.0", "@typescript-eslint/scope-manager": "8.50.0", "@typescript-eslint/types": "8.50.0", "@typescript-eslint/typescript-estree": "8.50.0" }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0", "typescript": ">=4.8.4 <6.0.0" } }, "sha512-87KgUXET09CRjGCi2Ejxy3PULXna63/bMYv72tCAlDJC3Yqwln0HiFJ3VJMst2+mEtNtZu5oFvX4qJGjKsnAgg=="], - - "@typescript-eslint/visitor-keys": ["@typescript-eslint/visitor-keys@8.50.0", "", { "dependencies": { "@typescript-eslint/types": "8.50.0", "eslint-visitor-keys": "^4.2.1" } }, "sha512-Xzmnb58+Db78gT/CCj/PVCvK+zxbnsw6F+O1oheYszJbBSdEjVhQi3C/Xttzxgi/GLmpvOggRs1RFpiJ8+c34Q=="], - - "@typespec/ts-http-runtime": ["@typespec/ts-http-runtime@0.3.2", "", { "dependencies": { "http-proxy-agent": "^7.0.0", "https-proxy-agent": "^7.0.0", "tslib": "^2.6.2" } }, "sha512-IlqQ/Gv22xUC1r/WQm4StLkYQmaaTsXAhUVsNE0+xiyf0yRFiH5++q78U3bw6bLKDCTmh0uqKB9eG9+Bt75Dkg=="], - - "@vitest/coverage-v8": ["@vitest/coverage-v8@3.2.4", "", { "dependencies": { "@ampproject/remapping": "^2.3.0", "@bcoe/v8-coverage": "^1.0.2", "ast-v8-to-istanbul": "^0.3.3", "debug": "^4.4.1", "istanbul-lib-coverage": "^3.2.2", "istanbul-lib-report": "^3.0.1", "istanbul-lib-source-maps": "^5.0.6", "istanbul-reports": "^3.1.7", "magic-string": "^0.30.17", "magicast": "^0.3.5", "std-env": "^3.9.0", "test-exclude": "^7.0.1", "tinyrainbow": "^2.0.0" }, "peerDependencies": { "@vitest/browser": "3.2.4", "vitest": "3.2.4" }, "optionalPeers": ["@vitest/browser"] }, "sha512-EyF9SXU6kS5Ku/U82E259WSnvg6c8KTjppUncuNdm5QHpe17mwREHnjDzozC8x9MZ0xfBUFSaLkRv4TMA75ALQ=="], - - "@vitest/expect": ["@vitest/expect@3.2.4", "", { "dependencies": { "@types/chai": "^5.2.2", "@vitest/spy": "3.2.4", "@vitest/utils": "3.2.4", "chai": "^5.2.0", "tinyrainbow": "^2.0.0" } }, "sha512-Io0yyORnB6sikFlt8QW5K7slY4OjqNX9jmJQ02QDda8lyM6B5oNgVWoSoKPac8/kgnCUzuHQKrSLtu/uOqqrig=="], - - "@vitest/mocker": ["@vitest/mocker@3.2.4", "", { "dependencies": { "@vitest/spy": "3.2.4", "estree-walker": "^3.0.3", "magic-string": "^0.30.17" }, "peerDependencies": { "msw": "^2.4.9", "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0" }, "optionalPeers": ["msw"] }, "sha512-46ryTE9RZO/rfDd7pEqFl7etuyzekzEhUbTW3BvmeO/BcCMEgq59BKhek3dXDWgAj4oMK6OZi+vRr1wPW6qjEQ=="], - - "@vitest/pretty-format": ["@vitest/pretty-format@3.2.4", "", { "dependencies": { "tinyrainbow": "^2.0.0" } }, "sha512-IVNZik8IVRJRTr9fxlitMKeJeXFFFN0JaB9PHPGQ8NKQbGpfjlTx9zO4RefN8gp7eqjNy8nyK3NZmBzOPeIxtA=="], - - "@vitest/runner": ["@vitest/runner@3.2.4", "", { "dependencies": { "@vitest/utils": "3.2.4", "pathe": "^2.0.3", "strip-literal": "^3.0.0" } }, "sha512-oukfKT9Mk41LreEW09vt45f8wx7DordoWUZMYdY/cyAk7w5TWkTRCNZYF7sX7n2wB7jyGAl74OxgwhPgKaqDMQ=="], - - "@vitest/snapshot": ["@vitest/snapshot@3.2.4", "", { "dependencies": { "@vitest/pretty-format": "3.2.4", "magic-string": "^0.30.17", "pathe": "^2.0.3" } }, "sha512-dEYtS7qQP2CjU27QBC5oUOxLE/v5eLkGqPE0ZKEIDGMs4vKWe7IjgLOeauHsR0D5YuuycGRO5oSRXnwnmA78fQ=="], - - "@vitest/spy": ["@vitest/spy@3.2.4", "", { "dependencies": { "tinyspy": "^4.0.3" } }, "sha512-vAfasCOe6AIK70iP5UD11Ac4siNUNJ9i/9PZ3NKx07sG6sUxeag1LWdNrMWeKKYBLlzuK+Gn65Yd5nyL6ds+nw=="], - - "@vitest/utils": ["@vitest/utils@3.2.4", "", { "dependencies": { "@vitest/pretty-format": "3.2.4", "loupe": "^3.1.4", "tinyrainbow": "^2.0.0" } }, "sha512-fB2V0JFrQSMsCo9HiSq3Ezpdv4iYaXRG1Sx8edX3MwxfyNn83mKiGzOcH+Fkxt4MHxr3y42fQi1oeAInqgX2QA=="], - - "@xmldom/xmldom": ["@xmldom/xmldom@0.8.11", "", {}, "sha512-cQzWCtO6C8TQiYl1ruKNn2U6Ao4o4WBBcbL61yJl84x+j5sOWWFU9X7DpND8XZG3daDppSsigMdfAIl2upQBRw=="], - - "@zxing/text-encoding": ["@zxing/text-encoding@0.9.0", "", {}, "sha512-U/4aVJ2mxI0aDNI8Uq0wEhMgY+u4CNtEb0om3+y3+niDAsoTCOB33UF0sxpzqzdqXLqmvc+vZyAt4O8pPdfkwA=="], - - "JSONStream": ["JSONStream@1.3.5", "", { "dependencies": { "jsonparse": "^1.2.0", "through": ">=2.2.7 <3" }, "bin": "bin.js" }, "sha512-E+iruNOY8VV9s4JEbe1aNEm6MiszPRr/UfcHMz0TQh1BXSxHK+ASV1R6W4HpjBhSeS+54PIsAMCBmwD06LLsqQ=="], - - "abort-controller": ["abort-controller@3.0.0", "", { "dependencies": { "event-target-shim": "^5.0.0" } }, "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg=="], - - "acorn": ["acorn@8.15.0", "", { "bin": "bin/acorn" }, "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg=="], - - "acorn-jsx": ["acorn-jsx@5.3.2", "", { "peerDependencies": { "acorn": "^6.0.0 || ^7.0.0 || ^8.0.0" } }, "sha512-rq9s+JNhf0IChjtDXxllJ7g41oZk5SlXtp0LHwyA5cejwn7vKmKp4pPri6YEePv2PU65sAsegbXtIinmDFDXgQ=="], - - "add-stream": ["add-stream@1.0.0", "", {}, "sha512-qQLMr+8o0WC4FZGQTcJiKBVC59JylcPSrTtk6usvmIDFUOCKegapy1VHQwRbFMOFyb/inzUVqHs+eMYKDM1YeQ=="], - - "adler-32": ["adler-32@1.3.1", "", {}, "sha512-ynZ4w/nUUv5rrsR8UUGoe1VC9hZj6V5hU9Qw1HlMDJGEJw5S7TfTErWTjMys6M7vr0YWcPqs3qAr4ss0nDfP+A=="], - - "agent-base": ["agent-base@7.1.4", "", {}, "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ=="], - - "ajv": ["ajv@6.12.6", "", { "dependencies": { "fast-deep-equal": "^3.1.1", "fast-json-stable-stringify": "^2.0.0", "json-schema-traverse": "^0.4.1", "uri-js": "^4.2.2" } }, "sha512-j3fVLgvTo527anyYyJOGTYJbG+vnnQYvE0m5mmkc1TK+nxAppkCLMIL0aZ4dblVCNoGShhm+kzE4ZUykBoMg4g=="], - - "ansi-align": ["ansi-align@3.0.1", "", { "dependencies": { "string-width": "^4.1.0" } }, "sha512-IOfwwBF5iczOjp/WeY4YxyjqAFMQoZufdQWDd19SEExbVLNXqvpzSJ/M7Za4/sCPmQ0+GRquoA7bGcINcxew6w=="], - - "ansi-regex": ["ansi-regex@6.2.2", "", {}, "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg=="], - - "ansi-styles": ["ansi-styles@3.2.1", "", { "dependencies": { "color-convert": "^1.9.0" } }, "sha512-VT0ZI6kZRdTh8YyJw3SMbYm/u+NqfsAxEpWO0Pf9sq8/e94WxxOpPKx9FR1FlyCtOVDNOQ+8ntlqFxiRc+r5qA=="], - - "archiver": ["archiver@7.0.1", "", { "dependencies": { "archiver-utils": "^5.0.2", "async": "^3.2.4", "buffer-crc32": "^1.0.0", "readable-stream": "^4.0.0", "readdir-glob": "^1.1.2", "tar-stream": "^3.0.0", "zip-stream": "^6.0.1" } }, "sha512-ZcbTaIqJOfCc03QwD468Unz/5Ir8ATtvAHsK+FdXbDIbGfihqh9mrvdcYunQzqn4HrvWWaFyaxJhGZagaJJpPQ=="], - - "archiver-utils": ["archiver-utils@5.0.2", "", { "dependencies": { "glob": "^10.0.0", "graceful-fs": "^4.2.0", "is-stream": "^2.0.1", "lazystream": "^1.0.0", "lodash": "^4.17.15", "normalize-path": "^3.0.0", "readable-stream": "^4.0.0" } }, "sha512-wuLJMmIBQYCsGZgYLTy5FIB2pF6Lfb6cXMSF8Qywwk3t20zWnAi7zLcQFdKQmIB8wyZpY5ER38x08GbwtR2cLA=="], - - "argparse": ["argparse@2.0.1", "", {}, "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q=="], - - "array-ify": ["array-ify@1.0.0", "", {}, "sha512-c5AMf34bKdvPhQ7tBGhqkgKNUzMr4WUs+WDtC2ZUGOUncbxKMTvqxYctiseW3+L4bA8ec+GcZ6/A/FW4m8ukng=="], - - "arrify": ["arrify@2.0.1", "", {}, "sha512-3duEwti880xqi4eAMN8AyR4a0ByT90zoYdLlevfrvU43vb0YZwZVfxOgxWrLXXXpyugL0hNZc9G6BiB5B3nUug=="], - - "asn1": ["asn1@0.2.6", "", { "dependencies": { "safer-buffer": "~2.1.0" } }, "sha512-ix/FxPn0MDjeyJ7i/yoHGFt/EX6LyNbxSEhPPXODPL+KB0VPk86UYfL0lMdy+KCnv+fmvIzySwaK5COwqVbWTQ=="], - - "assertion-error": ["assertion-error@2.0.1", "", {}, "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA=="], - - "ast-v8-to-istanbul": ["ast-v8-to-istanbul@0.3.9", "", { "dependencies": { "@jridgewell/trace-mapping": "^0.3.31", "estree-walker": "^3.0.3", "js-tokens": "^9.0.1" } }, "sha512-dSC6tJeOJxbZrPzPbv5mMd6CMiQ1ugaVXXPRad2fXUSsy1kstFn9XQWemV9VW7Y7kpxgQ/4WMoZfwdH8XSU48w=="], - - "async": ["async@3.2.6", "", {}, "sha512-htCUDlxyyCLMgaM3xXg0C0LW2xqfuQ6p05pCEIsXuyQ+a1koYKTuBMzRNwmybfLgvJDMd0r1LTn4+E0Ti6C2AA=="], - - "async-lock": ["async-lock@1.4.1", "", {}, "sha512-Az2ZTpuytrtqENulXwO3GGv1Bztugx6TT37NIo7imr/Qo0gsYiGtSdBa2B6fsXhTpVZDNfu1Qn3pk531e3q+nQ=="], - - "async-retry": ["async-retry@1.3.3", "", { "dependencies": { "retry": "0.13.1" } }, "sha512-wfr/jstw9xNi/0teMHrRW7dsz3Lt5ARhYNZ2ewpadnhaIp5mbALhOAP+EAdsC7t4Z6wqsDVv9+W6gm1Dk9mEyw=="], - - "asynckit": ["asynckit@0.4.0", "", {}, "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="], - - "available-typed-arrays": ["available-typed-arrays@1.0.7", "", { "dependencies": { "possible-typed-array-names": "^1.0.0" } }, "sha512-wvUjBtSGN7+7SjNpq/9M2Tg350UZD3q62IFZLbRAR1bSMlCo1ZaeW+BJ+D090e4hIIZLBcTDWe4Mh4jvUDajzQ=="], - - "b4a": ["b4a@1.7.3", "", { "peerDependencies": { "react-native-b4a": "*" }, "optionalPeers": ["react-native-b4a"] }, "sha512-5Q2mfq2WfGuFp3uS//0s6baOJLMoVduPYVeNmDYxu5OUA1/cBfvr2RIS7vi62LdNj/urk1hfmj867I3qt6uZ7Q=="], - - "balanced-match": ["balanced-match@1.0.2", "", {}, "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw=="], - - "bare-events": ["bare-events@2.8.2", "", { "peerDependencies": { "bare-abort-controller": "*" }, "optionalPeers": ["bare-abort-controller"] }, "sha512-riJjyv1/mHLIPX4RwiK+oW9/4c3TEUeORHKefKAKnZ5kyslbN+HXowtbaVEqt4IMUB7OXlfixcs6gsFeo/jhiQ=="], - - "bare-fs": ["bare-fs@4.5.2", "", { "dependencies": { "bare-events": "^2.5.4", "bare-path": "^3.0.0", "bare-stream": "^2.6.4", "bare-url": "^2.2.2", "fast-fifo": "^1.3.2" }, "peerDependencies": { "bare-buffer": "*" }, "optionalPeers": ["bare-buffer"] }, "sha512-veTnRzkb6aPHOvSKIOy60KzURfBdUflr5VReI+NSaPL6xf+XLdONQgZgpYvUuZLVQ8dCqxpBAudaOM1+KpAUxw=="], - - "bare-os": ["bare-os@3.6.2", "", {}, "sha512-T+V1+1srU2qYNBmJCXZkUY5vQ0B4FSlL3QDROnKQYOqeiQR8UbjNHlPa+TIbM4cuidiN9GaTaOZgSEgsvPbh5A=="], - - "bare-path": ["bare-path@3.0.0", "", { "dependencies": { "bare-os": "^3.0.1" } }, "sha512-tyfW2cQcB5NN8Saijrhqn0Zh7AnFNsnczRcuWODH0eYAXBsJ5gVxAUuNr7tsHSC6IZ77cA0SitzT+s47kot8Mw=="], - - "bare-stream": ["bare-stream@2.7.0", "", { "dependencies": { "streamx": "^2.21.0" }, "peerDependencies": { "bare-buffer": "*", "bare-events": "*" }, "optionalPeers": ["bare-buffer"] }, "sha512-oyXQNicV1y8nc2aKffH+BUHFRXmx6VrPzlnaEvMhram0nPBrKcEdcyBg5r08D0i8VxngHFAiVyn1QKXpSG0B8A=="], - - "bare-url": ["bare-url@2.3.2", "", { "dependencies": { "bare-path": "^3.0.0" } }, "sha512-ZMq4gd9ngV5aTMa5p9+UfY0b3skwhHELaDkhEHetMdX0LRkW9kzaym4oo/Eh+Ghm0CCDuMTsRIGM/ytUc1ZYmw=="], - - "base64-arraybuffer": ["base64-arraybuffer@1.0.2", "", {}, "sha512-I3yl4r9QB5ZRY3XuJVEPfc2XhZO6YweFPI+UovAzn+8/hb3oJ6lnysaFcjVpkCPfVWFUDvoZ8kmVDP7WyRtYtQ=="], - - "base64-js": ["base64-js@1.5.1", "", {}, "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA=="], - - "bcrypt-pbkdf": ["bcrypt-pbkdf@1.0.2", "", { "dependencies": { "tweetnacl": "^0.14.3" } }, "sha512-qeFIXtP4MSoi6NLqO12WfqARWWuCKi2Rn/9hJLEmtB5yTNr9DqFWkJRCf2qShWzPeAMRnOgCrq0sg/KLv5ES9w=="], - - "bignumber.js": ["bignumber.js@9.3.1", "", {}, "sha512-Ko0uX15oIUS7wJ3Rb30Fs6SkVbLmPBAKdlm7q9+ak9bbIeFf0MwuBsQV6z7+X768/cHsfg+WlysDWJcmthjsjQ=="], - - "bl": ["bl@4.1.0", "", { "dependencies": { "buffer": "^5.5.0", "inherits": "^2.0.4", "readable-stream": "^3.4.0" } }, "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w=="], - - "block-stream2": ["block-stream2@2.1.0", "", { "dependencies": { "readable-stream": "^3.4.0" } }, "sha512-suhjmLI57Ewpmq00qaygS8UgEq2ly2PCItenIyhMqVjo4t4pGzqMvfgJuX8iWTeSDdfSSqS6j38fL4ToNL7Pfg=="], - - "bluebird": ["bluebird@3.4.7", "", {}, "sha512-iD3898SR7sWVRHbiQv+sHUtHnMvC1o3nW5rAcqnq3uOn07DSAppZYUkIGslDz6gXC7HfunPe7YVBgoEJASPcHA=="], - - "bowser": ["bowser@2.13.1", "", {}, "sha512-OHawaAbjwx6rqICCKgSG0SAnT05bzd7ppyKLVUITZpANBaaMFBAsaNkto3LoQ31tyFP5kNujE8Cdx85G9VzOkw=="], - - "boxen": ["boxen@8.0.1", "", { "dependencies": { "ansi-align": "^3.0.1", "camelcase": "^8.0.0", "chalk": "^5.3.0", "cli-boxes": "^3.0.0", "string-width": "^7.2.0", "type-fest": "^4.21.0", "widest-line": "^5.0.0", "wrap-ansi": "^9.0.0" } }, "sha512-F3PH5k5juxom4xktynS7MoFY+NUWH5LC4CnH11YB8NPew+HLpmBLCybSAEyb2F+4pRXhuhWqFesoQd6DAyc2hw=="], - - "brace-expansion": ["brace-expansion@1.1.12", "", { "dependencies": { "balanced-match": "^1.0.0", "concat-map": "0.0.1" } }, "sha512-9T9UjW3r0UW5c1Q7GTwllptXwhvYmEzFhzMfZ9H7FQWt+uZePjZPjBP/W1ZEyZ1twGWom5/56TF4lPcqjnDHcg=="], - - "browser-or-node": ["browser-or-node@2.1.1", "", {}, "sha512-8CVjaLJGuSKMVTxJ2DpBl5XnlNDiT4cQFeuCJJrvJmts9YrTZDizTX7PjC2s6W4x+MBGZeEY6dGMrF04/6Hgqg=="], - - "buffer": ["buffer@6.0.3", "", { "dependencies": { "base64-js": "^1.3.1", "ieee754": "^1.2.1" } }, "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA=="], - - "buffer-crc32": ["buffer-crc32@1.0.0", "", {}, "sha512-Db1SbgBS/fg/392AblrMJk97KggmvYhr4pB5ZIMTWtaivCPMWLkmb7m21cJvpvgK+J3nsU2CmmixNBZx4vFj/w=="], - - "buffer-equal-constant-time": ["buffer-equal-constant-time@1.0.1", "", {}, "sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA=="], - - "buffer-from": ["buffer-from@1.1.2", "", {}, "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ=="], - - "buildcheck": ["buildcheck@0.0.7", "", {}, "sha512-lHblz4ahamxpTmnsk+MNTRWsjYKv965MwOrSJyeD588rR3Jcu7swE+0wN5F+PbL5cjgu/9ObkhfzEPuofEMwLA=="], - - "bundle-name": ["bundle-name@4.1.0", "", { "dependencies": { "run-applescript": "^7.0.0" } }, "sha512-tjwM5exMg6BGRI+kNmTntNsvdZS1X8BFYS6tnJ2hdH0kVxM6/eVZ2xy+FqStSWvYmtfFMDLIxurorHwDKfDz5Q=="], - - "byline": ["byline@5.0.0", "", {}, "sha512-s6webAy+R4SR8XVuJWt2V2rGvhnrhxN+9S15GNuTK3wKPOXFF6RNc+8ug2XhH+2s4f+uudG4kUVYmYOQWL2g0Q=="], - - "cac": ["cac@6.7.14", "", {}, "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ=="], - - "call-bind": ["call-bind@1.0.8", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.0", "es-define-property": "^1.0.0", "get-intrinsic": "^1.2.4", "set-function-length": "^1.2.2" } }, "sha512-oKlSFMcMwpUg2ednkhQ454wfWiU/ul3CkJe/PEHcTKuiX6RpbehUiFMXu13HalGZxfUwCQzZG747YXBn1im9ww=="], - - "call-bind-apply-helpers": ["call-bind-apply-helpers@1.0.2", "", { "dependencies": { "es-errors": "^1.3.0", "function-bind": "^1.1.2" } }, "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ=="], - - "call-bound": ["call-bound@1.0.4", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "get-intrinsic": "^1.3.0" } }, "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg=="], - - "callsites": ["callsites@3.1.0", "", {}, "sha512-P8BjAsXvZS+VIDUI11hHCQEv74YT67YUi5JJFNWIqL235sBmjX4+qx9Muvls5ivyNENctx46xQLQ3aTuE7ssaQ=="], - - "camelcase": ["camelcase@8.0.0", "", {}, "sha512-8WB3Jcas3swSvjIeA2yvCJ+Miyz5l1ZmB6HFb9R1317dt9LCQoswg/BGrmAmkWVEszSrrg4RwmO46qIm2OEnSA=="], - - "camelcase-keys": ["camelcase-keys@6.2.2", "", { "dependencies": { "camelcase": "^5.3.1", "map-obj": "^4.0.0", "quick-lru": "^4.0.1" } }, "sha512-YrwaA0vEKazPBkn0ipTiMpSajYDSe+KjQfrjhcBMxJt/znbvlHd8Pw/Vamaz5EB4Wfhs3SUR3Z9mwRu/P3s3Yg=="], - - "canvg": ["canvg@3.0.11", "", { "dependencies": { "@babel/runtime": "^7.12.5", "@types/raf": "^3.4.0", "core-js": "^3.8.3", "raf": "^3.4.1", "regenerator-runtime": "^0.13.7", "rgbcolor": "^1.0.1", "stackblur-canvas": "^2.0.0", "svg-pathdata": "^6.0.3" } }, "sha512-5ON+q7jCTgMp9cjpu4Jo6XbvfYwSB2Ow3kzHKfIyJfaCAOHLbdKPQqGKgfED/R5B+3TFFfe8pegYA+b423SRyA=="], - - "cfb": ["cfb@1.2.2", "", { "dependencies": { "adler-32": "~1.3.0", "crc-32": "~1.2.0" } }, "sha512-KfdUZsSOw19/ObEWasvBP/Ac4reZvAGauZhs6S/gqNhXhI7cKwvlH7ulj+dOEYnca4bm4SGo8C1bTAQvnTjgQA=="], - - "chai": ["chai@5.3.3", "", { "dependencies": { "assertion-error": "^2.0.1", "check-error": "^2.1.1", "deep-eql": "^5.0.1", "loupe": "^3.1.0", "pathval": "^2.0.0" } }, "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw=="], - - "chalk": ["chalk@5.6.2", "", {}, "sha512-7NzBL0rN6fMUW+f7A6Io4h40qQlG+xGmtMxfbnH/K7TAtt8JQWVQK+6g0UXKMeVJoyV5EkkNsErQ8pVD3bLHbA=="], - - "chardet": ["chardet@2.1.1", "", {}, "sha512-PsezH1rqdV9VvyNhxxOW32/d75r01NY7TQCmOqomRo15ZSOKbpTFVsfjghxo6JloQUCGnH4k1LGu0R4yCLlWQQ=="], - - "check-error": ["check-error@2.1.1", "", {}, "sha512-OAlb+T7V4Op9OwdkjmguYRqncdlx5JiofwOAUkmTF+jNdHwzTaTs4sRAGpzLF3oOz5xAyDGrPgeIDFQmDOTiJw=="], - - "chownr": ["chownr@1.1.4", "", {}, "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg=="], - - "cli-boxes": ["cli-boxes@3.0.0", "", {}, "sha512-/lzGpEWL/8PfI0BmBOPRwp0c/wFNX1RdUML3jK/RcSBA9T8mZDdQpqYBKtCFTOfQbwPqWEOpjqW+Fnayc0969g=="], - - "cli-cursor": ["cli-cursor@5.0.0", "", { "dependencies": { "restore-cursor": "^5.0.0" } }, "sha512-aCj4O5wKyszjMmDT4tZj93kxyydN/K5zPWSCe6/0AV/AA1pqe5ZBIw0a2ZfPQV7lL5/yb5HsUreJ6UFAF1tEQw=="], - - "cli-spinners": ["cli-spinners@2.9.2", "", {}, "sha512-ywqV+5MmyL4E7ybXgKys4DugZbX0FC6LnwrhjuykIjnK9k8OQacQ7axGKnjDXWNhns0xot3bZI5h55H8yo9cJg=="], - - "cli-table3": ["cli-table3@0.6.5", "", { "dependencies": { "string-width": "^4.2.0" }, "optionalDependencies": { "@colors/colors": "1.5.0" } }, "sha512-+W/5efTR7y5HRD7gACw9yQjqMVvEMLBHmboM/kPWam+H+Hmyrgjh6YncVKK122YZkXrLudzTuAukUw9FnMf7IQ=="], - - "cli-width": ["cli-width@4.1.0", "", {}, "sha512-ouuZd4/dm2Sw5Gmqy6bGyNNNe1qt9RpmxveLSO7KcgsTnU7RXfsw+/bukWGo1abgBiMAic068rclZsO4IWmmxQ=="], - - "cliui": ["cliui@7.0.4", "", { "dependencies": { "string-width": "^4.2.0", "strip-ansi": "^6.0.0", "wrap-ansi": "^7.0.0" } }, "sha512-OcRE68cOsVMXp1Yvonl/fzkQOyjLSu/8bhPDfQt0e0/Eb283TKP20Fs2MqoPsr9SwA595rRCA+QMzYc9nBP+JQ=="], - - "codepage": ["codepage@1.15.0", "", {}, "sha512-3g6NUTPd/YtuuGrhMnOMRjFc+LJw/bnMp3+0r/Wcz3IXUuCosKRJvMphm5+Q+bvTVGcJJuRvVLuYba+WojaFaA=="], - - "color-convert": ["color-convert@1.9.3", "", { "dependencies": { "color-name": "1.1.3" } }, "sha512-QfAUtd+vFdAtFQcC8CCyYt1fYWxSqAiK2cSD6zDB8N3cpsEBAvRxp9zOGg6G/SHHJYAT88/az/IuDGALsNVbGg=="], - - "color-name": ["color-name@1.1.3", "", {}, "sha512-72fSenhMw2HZMTVHeCA9KCmpEIbzWiQsjN+BHcBbS9vr1mtt+vJjPdksIBNUmKAW8TFUDPJK5SUU3QhE9NEXDw=="], - - "combined-stream": ["combined-stream@1.0.8", "", { "dependencies": { "delayed-stream": "~1.0.0" } }, "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg=="], - - "commander": ["commander@11.1.0", "", {}, "sha512-yPVavfyCcRhmorC7rWlkHn15b4wDVgVmBA7kV4QVBsF7kv/9TKJAbAXVTxvTnwP8HHKjRCJDClKbciiYS7p0DQ=="], - - "commondir": ["commondir@1.0.1", "", {}, "sha512-W9pAhw0ja1Edb5GVdIF1mjZw/ASI0AlShXM83UUGe2DVr5TdAPEA1OA8m/g8zWp9x6On7gqufY+FatDbC3MDQg=="], - - "compare-func": ["compare-func@2.0.0", "", { "dependencies": { "array-ify": "^1.0.0", "dot-prop": "^5.1.0" } }, "sha512-zHig5N+tPWARooBnb0Zx1MFcdfpyJrfTJ3Y5L+IFvUm8rM74hHz66z0gw0x4tijh5CorKkKUCnW82R2vmpeCRA=="], - - "compress-commons": ["compress-commons@6.0.2", "", { "dependencies": { "crc-32": "^1.2.0", "crc32-stream": "^6.0.0", "is-stream": "^2.0.1", "normalize-path": "^3.0.0", "readable-stream": "^4.0.0" } }, "sha512-6FqVXeETqWPoGcfzrXb37E50NP0LXT8kAMu5ooZayhWWdgEY4lBEEcbQNXtkuKQsGduxiIcI4gOTsxTmuq/bSg=="], - - "concat-map": ["concat-map@0.0.1", "", {}, "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg=="], - - "concat-stream": ["concat-stream@2.0.0", "", { "dependencies": { "buffer-from": "^1.0.0", "inherits": "^2.0.3", "readable-stream": "^3.0.2", "typedarray": "^0.0.6" } }, "sha512-MWufYdFw53ccGjCA+Ol7XJYpAlW6/prSMzuPOTRnJGcGzuhLn4Scrz7qf6o8bROZ514ltazcIFJZevcfbo0x7A=="], - - "conventional-changelog": ["conventional-changelog@3.1.25", "", { "dependencies": { "conventional-changelog-angular": "^5.0.12", "conventional-changelog-atom": "^2.0.8", "conventional-changelog-codemirror": "^2.0.8", "conventional-changelog-conventionalcommits": "^4.5.0", "conventional-changelog-core": "^4.2.1", "conventional-changelog-ember": "^2.0.9", "conventional-changelog-eslint": "^3.0.9", "conventional-changelog-express": "^2.0.6", "conventional-changelog-jquery": "^3.0.11", "conventional-changelog-jshint": "^2.0.9", "conventional-changelog-preset-loader": "^2.3.4" } }, "sha512-ryhi3fd1mKf3fSjbLXOfK2D06YwKNic1nC9mWqybBHdObPd8KJ2vjaXZfYj1U23t+V8T8n0d7gwnc9XbIdFbyQ=="], - - "conventional-changelog-angular": ["conventional-changelog-angular@5.0.13", "", { "dependencies": { "compare-func": "^2.0.0", "q": "^1.5.1" } }, "sha512-i/gipMxs7s8L/QeuavPF2hLnJgH6pEZAttySB6aiQLWcX3puWDL3ACVmvBhJGxnAy52Qc15ua26BufY6KpmrVA=="], - - "conventional-changelog-atom": ["conventional-changelog-atom@2.0.8", "", { "dependencies": { "q": "^1.5.1" } }, "sha512-xo6v46icsFTK3bb7dY/8m2qvc8sZemRgdqLb/bjpBsH2UyOS8rKNTgcb5025Hri6IpANPApbXMg15QLb1LJpBw=="], - - "conventional-changelog-codemirror": ["conventional-changelog-codemirror@2.0.8", "", { "dependencies": { "q": "^1.5.1" } }, "sha512-z5DAsn3uj1Vfp7po3gpt2Boc+Bdwmw2++ZHa5Ak9k0UKsYAO5mH1UBTN0qSCuJZREIhX6WU4E1p3IW2oRCNzQw=="], - - "conventional-changelog-config-spec": ["conventional-changelog-config-spec@2.1.0", "", {}, "sha512-IpVePh16EbbB02V+UA+HQnnPIohgXvJRxHcS5+Uwk4AT5LjzCZJm5sp/yqs5C6KZJ1jMsV4paEV13BN1pvDuxQ=="], - - "conventional-changelog-conventionalcommits": ["conventional-changelog-conventionalcommits@4.6.3", "", { "dependencies": { "compare-func": "^2.0.0", "lodash": "^4.17.15", "q": "^1.5.1" } }, "sha512-LTTQV4fwOM4oLPad317V/QNQ1FY4Hju5qeBIM1uTHbrnCE+Eg4CdRZ3gO2pUeR+tzWdp80M2j3qFFEDWVqOV4g=="], - - "conventional-changelog-core": ["conventional-changelog-core@4.2.4", "", { "dependencies": { "add-stream": "^1.0.0", "conventional-changelog-writer": "^5.0.0", "conventional-commits-parser": "^3.2.0", "dateformat": "^3.0.0", "get-pkg-repo": "^4.0.0", "git-raw-commits": "^2.0.8", "git-remote-origin-url": "^2.0.0", "git-semver-tags": "^4.1.1", "lodash": "^4.17.15", "normalize-package-data": "^3.0.0", "q": "^1.5.1", "read-pkg": "^3.0.0", "read-pkg-up": "^3.0.0", "through2": "^4.0.0" } }, "sha512-gDVS+zVJHE2v4SLc6B0sLsPiloR0ygU7HaDW14aNJE1v4SlqJPILPl/aJC7YdtRE4CybBf8gDwObBvKha8Xlyg=="], - - "conventional-changelog-ember": ["conventional-changelog-ember@2.0.9", "", { "dependencies": { "q": "^1.5.1" } }, "sha512-ulzIReoZEvZCBDhcNYfDIsLTHzYHc7awh+eI44ZtV5cx6LVxLlVtEmcO+2/kGIHGtw+qVabJYjdI5cJOQgXh1A=="], - - "conventional-changelog-eslint": ["conventional-changelog-eslint@3.0.9", "", { "dependencies": { "q": "^1.5.1" } }, "sha512-6NpUCMgU8qmWmyAMSZO5NrRd7rTgErjrm4VASam2u5jrZS0n38V7Y9CzTtLT2qwz5xEChDR4BduoWIr8TfwvXA=="], - - "conventional-changelog-express": ["conventional-changelog-express@2.0.6", "", { "dependencies": { "q": "^1.5.1" } }, "sha512-SDez2f3iVJw6V563O3pRtNwXtQaSmEfTCaTBPCqn0oG0mfkq0rX4hHBq5P7De2MncoRixrALj3u3oQsNK+Q0pQ=="], - - "conventional-changelog-jquery": ["conventional-changelog-jquery@3.0.11", "", { "dependencies": { "q": "^1.5.1" } }, "sha512-x8AWz5/Td55F7+o/9LQ6cQIPwrCjfJQ5Zmfqi8thwUEKHstEn4kTIofXub7plf1xvFA2TqhZlq7fy5OmV6BOMw=="], - - "conventional-changelog-jshint": ["conventional-changelog-jshint@2.0.9", "", { "dependencies": { "compare-func": "^2.0.0", "q": "^1.5.1" } }, "sha512-wMLdaIzq6TNnMHMy31hql02OEQ8nCQfExw1SE0hYL5KvU+JCTuPaDO+7JiogGT2gJAxiUGATdtYYfh+nT+6riA=="], - - "conventional-changelog-preset-loader": ["conventional-changelog-preset-loader@2.3.4", "", {}, "sha512-GEKRWkrSAZeTq5+YjUZOYxdHq+ci4dNwHvpaBC3+ENalzFWuCWa9EZXSuZBpkr72sMdKB+1fyDV4takK1Lf58g=="], - - "conventional-changelog-writer": ["conventional-changelog-writer@5.0.1", "", { "dependencies": { "conventional-commits-filter": "^2.0.7", "dateformat": "^3.0.0", "handlebars": "^4.7.7", "json-stringify-safe": "^5.0.1", "lodash": "^4.17.15", "meow": "^8.0.0", "semver": "^6.0.0", "split": "^1.0.0", "through2": "^4.0.0" }, "bin": "cli.js" }, "sha512-5WsuKUfxW7suLblAbFnxAcrvf6r+0b7GvNaWUwUIk0bXMnENP/PEieGKVUQrjPqwPT4o3EPAASBXiY6iHooLOQ=="], - - "conventional-commits-filter": ["conventional-commits-filter@2.0.7", "", { "dependencies": { "lodash.ismatch": "^4.4.0", "modify-values": "^1.0.0" } }, "sha512-ASS9SamOP4TbCClsRHxIHXRfcGCnIoQqkvAzCSbZzTFLfcTqJVugB0agRgsEELsqaeWgsXv513eS116wnlSSPA=="], - - "conventional-commits-parser": ["conventional-commits-parser@3.2.4", "", { "dependencies": { "JSONStream": "^1.0.4", "is-text-path": "^1.0.1", "lodash": "^4.17.15", "meow": "^8.0.0", "split2": "^3.0.0", "through2": "^4.0.0" }, "bin": "cli.js" }, "sha512-nK7sAtfi+QXbxHCYfhpZsfRtaitZLIA6889kFIouLvz6repszQDgxBu7wf2WbU+Dco7sAnNCJYERCwt54WPC2Q=="], - - "conventional-recommended-bump": ["conventional-recommended-bump@6.1.0", "", { "dependencies": { "concat-stream": "^2.0.0", "conventional-changelog-preset-loader": "^2.3.4", "conventional-commits-filter": "^2.0.7", "conventional-commits-parser": "^3.2.0", "git-raw-commits": "^2.0.8", "git-semver-tags": "^4.1.1", "meow": "^8.0.0", "q": "^1.5.1" }, "bin": "cli.js" }, "sha512-uiApbSiNGM/kkdL9GTOLAqC4hbptObFo4wW2QRyHsKciGAfQuLU1ShZ1BIVI/+K2BE/W1AWYQMCXAsv4dyKPaw=="], - - "core-js": ["core-js@3.47.0", "", {}, "sha512-c3Q2VVkGAUyupsjRnaNX6u8Dq2vAdzm9iuPj5FW0fRxzlxgq9Q39MDq10IvmQSpLgHQNyQzQmOo6bgGHmH3NNg=="], - - "core-util-is": ["core-util-is@1.0.3", "", {}, "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ=="], - - "cpu-features": ["cpu-features@0.0.10", "", { "dependencies": { "buildcheck": "~0.0.6", "nan": "^2.19.0" } }, "sha512-9IkYqtX3YHPCzoVg1Py+o9057a3i0fp7S530UWokCSaFVTc7CwXPRiOjRjBQQ18ZCNafx78YfnG+HALxtVmOGA=="], - - "crc-32": ["crc-32@1.2.2", "", { "bin": { "crc32": "bin/crc32.njs" } }, "sha512-ROmzCKrTnOwybPcJApAA6WBWij23HVfGVNKqqrZpuyZOHqK2CwHSvpGuyt/UNNvaIjEd8X5IFGp4Mh+Ie1IHJQ=="], - - "crc32-stream": ["crc32-stream@6.0.0", "", { "dependencies": { "crc-32": "^1.2.0", "readable-stream": "^4.0.0" } }, "sha512-piICUB6ei4IlTv1+653yq5+KoqfBYmj9bw6LqXoOneTMDXk5nM1qt12mFW1caG3LlJXEKW1Bp0WggEmIfQB34g=="], - - "cross-spawn": ["cross-spawn@7.0.6", "", { "dependencies": { "path-key": "^3.1.0", "shebang-command": "^2.0.0", "which": "^2.0.1" } }, "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA=="], - - "css-line-break": ["css-line-break@2.1.0", "", { "dependencies": { "utrie": "^1.0.2" } }, "sha512-FHcKFCZcAha3LwfVBhCQbW2nCNbkZXn7KVUJcsT5/P8YmfsVja0FMPJr0B903j/E69HUphKiV9iQArX8SDYA4w=="], - - "csv-parse": ["csv-parse@6.1.0", "", {}, "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw=="], - - "dargs": ["dargs@7.0.0", "", {}, "sha512-2iy1EkLdlBzQGvbweYRFxmFath8+K7+AKB0TlhHWkNuH+TmovaMH/Wp7V7R4u7f4SnX3OgLsU9t1NI9ioDnUpg=="], - - "dateformat": ["dateformat@3.0.3", "", {}, "sha512-jyCETtSl3VMZMWeRo7iY1FL19ges1t55hMo5yaam4Jrsm5EPL89UQkoQRyiI+Yf4k8r2ZpdngkV8hr1lIdjb3Q=="], - - "debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="], - - "decamelize": ["decamelize@1.2.0", "", {}, "sha512-z2S+W9X73hAUUki+N+9Za2lBlun89zigOyGrsax+KUQ6wKW4ZoWpEYBkGhQjwAjjDCkWxhY0VKEhk8wzY7F5cA=="], - - "decamelize-keys": ["decamelize-keys@1.1.1", "", { "dependencies": { "decamelize": "^1.1.0", "map-obj": "^1.0.0" } }, "sha512-WiPxgEirIV0/eIOMcnFBA3/IJZAZqKnwAwWyvvdi4lsr1WCN22nhdf/3db3DoZcUjTV2SqfzIwNyp6y2xs3nmg=="], - - "decode-uri-component": ["decode-uri-component@0.2.2", "", {}, "sha512-FqUYQ+8o158GyGTrMFJms9qh3CqTKvAqgqsTnkLI8sKu0028orqBhxNMFkFen0zGyg6epACD32pjVk58ngIErQ=="], - - "deep-eql": ["deep-eql@5.0.2", "", {}, "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q=="], - - "deep-is": ["deep-is@0.1.4", "", {}, "sha512-oIPzksmTg4/MriiaYGO+okXDT7ztn/w3Eptv/+gSIdMdKsJo0u4CfYNFJPy+4SKMuCqGw2wxnA+URMg3t8a/bQ=="], - - "deepmerge": ["deepmerge@4.3.1", "", {}, "sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A=="], - - "default-browser": ["default-browser@5.4.0", "", { "dependencies": { "bundle-name": "^4.1.0", "default-browser-id": "^5.0.0" } }, "sha512-XDuvSq38Hr1MdN47EDvYtx3U0MTqpCEn+F6ft8z2vYDzMrvQhVp0ui9oQdqW3MvK3vqUETglt1tVGgjLuJ5izg=="], - - "default-browser-id": ["default-browser-id@5.0.1", "", {}, "sha512-x1VCxdX4t+8wVfd1so/9w+vQ4vx7lKd2Qp5tDRutErwmR85OgmfX7RlLRMWafRMY7hbEiXIbudNrjOAPa/hL8Q=="], - - "define-data-property": ["define-data-property@1.1.4", "", { "dependencies": { "es-define-property": "^1.0.0", "es-errors": "^1.3.0", "gopd": "^1.0.1" } }, "sha512-rBMvIzlpA8v6E+SJZoo++HAYqsLrkg7MSfIinMPFhmkorw7X+dOXVJQs+QT69zGkzMyfDnIMN2Wid1+NbL3T+A=="], - - "define-lazy-prop": ["define-lazy-prop@3.0.0", "", {}, "sha512-N+MeXYoqr3pOgn8xfyRPREN7gHakLYjhsHhWGT3fWAiL4IkAt0iDw14QiiEm2bE30c5XX5q0FtAA3CK5f9/BUg=="], - - "delayed-stream": ["delayed-stream@1.0.0", "", {}, "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ=="], - - "detect-indent": ["detect-indent@6.1.0", "", {}, "sha512-reYkTUJAZb9gUuZ2RvVCNhVHdg62RHnJ7WJl8ftMi4diZ6NWlciOzQN88pUhSELEwflJht4oQDv0F0BMlwaYtA=="], - - "detect-newline": ["detect-newline@3.1.0", "", {}, "sha512-TLz+x/vEXm/Y7P7wn1EJFNLxYpUD4TgMosxY6fAVJUnJMbupHBOncxyWUG9OpTaH9EBD7uFI5LfEgmMOc54DsA=="], - - "dingbat-to-unicode": ["dingbat-to-unicode@1.0.1", "", {}, "sha512-98l0sW87ZT58pU4i61wa2OHwxbiYSbuxsCBozaVnYX2iCnr3bLM3fIes1/ej7h1YdOKuKt/MLs706TVnALA65w=="], - - "docker-compose": ["docker-compose@1.3.0", "", { "dependencies": { "yaml": "^2.2.2" } }, "sha512-7Gevk/5eGD50+eMD+XDnFnOrruFkL0kSd7jEG4cjmqweDSUhB7i0g8is/nBdVpl+Bx338SqIB2GLKm32M+Vs6g=="], - - "docker-modem": ["docker-modem@5.0.6", "", { "dependencies": { "debug": "^4.1.1", "readable-stream": "^3.5.0", "split-ca": "^1.0.1", "ssh2": "^1.15.0" } }, "sha512-ens7BiayssQz/uAxGzH8zGXCtiV24rRWXdjNha5V4zSOcxmAZsfGVm/PPFbwQdqEkDnhG+SyR9E3zSHUbOKXBQ=="], - - "dockerode": ["dockerode@4.0.9", "", { "dependencies": { "@balena/dockerignore": "^1.0.2", "@grpc/grpc-js": "^1.11.1", "@grpc/proto-loader": "^0.7.13", "docker-modem": "^5.0.6", "protobufjs": "^7.3.2", "tar-fs": "^2.1.4", "uuid": "^10.0.0" } }, "sha512-iND4mcOWhPaCNh54WmK/KoSb35AFqPAUWFMffTQcp52uQt36b5uNwEJTSXntJZBbeGad72Crbi/hvDIv6us/6Q=="], - - "dompurify": ["dompurify@3.3.1", "", { "optionalDependencies": { "@types/trusted-types": "^2.0.7" } }, "sha512-qkdCKzLNtrgPFP1Vo+98FRzJnBRGe4ffyCea9IwHB1fyxPOeNTHpLKYGd4Uk9xvNoH0ZoOjwZxNptyMwqrId1Q=="], - - "dot-prop": ["dot-prop@5.3.0", "", { "dependencies": { "is-obj": "^2.0.0" } }, "sha512-QM8q3zDe58hqUqjraQOmzZ1LIH9SWQJTlEKCH4kJ2oQvLZk7RbQXvtDM2XEq3fwkV9CCvvH4LA0AV+ogFsBM2Q=="], - - "dotgitignore": ["dotgitignore@2.1.0", "", { "dependencies": { "find-up": "^3.0.0", "minimatch": "^3.0.4" } }, "sha512-sCm11ak2oY6DglEPpCB8TixLjWAxd3kJTs6UIcSasNYxXdFPV+YKlye92c8H4kKFqV5qYMIh7d+cYecEg0dIkA=="], - - "duck": ["duck@0.1.12", "", { "dependencies": { "underscore": "^1.13.1" } }, "sha512-wkctla1O6VfP89gQ+J/yDesM0S7B7XLXjKGzXxMDVFg7uEn706niAtyYovKbyq1oT9YwDcly721/iUWoc8MVRg=="], - - "dunder-proto": ["dunder-proto@1.0.1", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.1", "es-errors": "^1.3.0", "gopd": "^1.2.0" } }, "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A=="], - - "duplexify": ["duplexify@4.1.3", "", { "dependencies": { "end-of-stream": "^1.4.1", "inherits": "^2.0.3", "readable-stream": "^3.1.1", "stream-shift": "^1.0.2" } }, "sha512-M3BmBhwJRZsSx38lZyhE53Csddgzl5R7xGJNk7CVddZD6CcmwMCH8J+7AprIrQKH7TonKxaCjcv27Qmf+sQ+oA=="], - - "eastasianwidth": ["eastasianwidth@0.2.0", "", {}, "sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA=="], - - "ecdsa-sig-formatter": ["ecdsa-sig-formatter@1.0.11", "", { "dependencies": { "safe-buffer": "^5.0.1" } }, "sha512-nagl3RYrbNv6kQkeJIpt6NJZy8twLB/2vtz6yN9Z4vRKHN4/QZJIEbqohALSgwKdnksuY3k5Addp5lg8sVoVcQ=="], - - "emoji-regex": ["emoji-regex@10.6.0", "", {}, "sha512-toUI84YS5YmxW219erniWD0CIVOo46xGKColeNQRgOzDorgBi1v4D71/OFzgD9GO2UGKIv1C3Sp8DAn0+j5w7A=="], - - "end-of-stream": ["end-of-stream@1.4.5", "", { "dependencies": { "once": "^1.4.0" } }, "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg=="], - - "error-ex": ["error-ex@1.3.4", "", { "dependencies": { "is-arrayish": "^0.2.1" } }, "sha512-sqQamAnR14VgCr1A618A3sGrygcpK+HEbenA/HiEAkkUwcZIIB/tgWqHFxWgOyDh4nB4JCRimh79dR5Ywc9MDQ=="], - - "es-define-property": ["es-define-property@1.0.1", "", {}, "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g=="], - - "es-errors": ["es-errors@1.3.0", "", {}, "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw=="], - - "es-module-lexer": ["es-module-lexer@1.7.0", "", {}, "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA=="], - - "es-object-atoms": ["es-object-atoms@1.1.1", "", { "dependencies": { "es-errors": "^1.3.0" } }, "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA=="], - - "es-set-tostringtag": ["es-set-tostringtag@2.1.0", "", { "dependencies": { "es-errors": "^1.3.0", "get-intrinsic": "^1.2.6", "has-tostringtag": "^1.0.2", "hasown": "^2.0.2" } }, "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA=="], - - "esbuild": ["esbuild@0.27.2", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.27.2", "@esbuild/android-arm": "0.27.2", "@esbuild/android-arm64": "0.27.2", "@esbuild/android-x64": "0.27.2", "@esbuild/darwin-arm64": "0.27.2", "@esbuild/darwin-x64": "0.27.2", "@esbuild/freebsd-arm64": "0.27.2", "@esbuild/freebsd-x64": "0.27.2", "@esbuild/linux-arm": "0.27.2", "@esbuild/linux-arm64": "0.27.2", "@esbuild/linux-ia32": "0.27.2", "@esbuild/linux-loong64": "0.27.2", "@esbuild/linux-mips64el": "0.27.2", "@esbuild/linux-ppc64": "0.27.2", "@esbuild/linux-riscv64": "0.27.2", "@esbuild/linux-s390x": "0.27.2", "@esbuild/linux-x64": "0.27.2", "@esbuild/netbsd-arm64": "0.27.2", "@esbuild/netbsd-x64": "0.27.2", "@esbuild/openbsd-arm64": "0.27.2", "@esbuild/openbsd-x64": "0.27.2", "@esbuild/openharmony-arm64": "0.27.2", "@esbuild/sunos-x64": "0.27.2", "@esbuild/win32-arm64": "0.27.2", "@esbuild/win32-ia32": "0.27.2", "@esbuild/win32-x64": "0.27.2" }, "bin": "bin/esbuild" }, "sha512-HyNQImnsOC7X9PMNaCIeAm4ISCQXs5a5YasTXVliKv4uuBo1dKrG0A+uQS8M5eXjVMnLg3WgXaKvprHlFJQffw=="], - - "escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="], - - "escape-string-regexp": ["escape-string-regexp@4.0.0", "", {}, "sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA=="], - - "eslint": ["eslint@9.39.2", "", { "dependencies": { "@eslint-community/eslint-utils": "^4.8.0", "@eslint-community/regexpp": "^4.12.1", "@eslint/config-array": "^0.21.1", "@eslint/config-helpers": "^0.4.2", "@eslint/core": "^0.17.0", "@eslint/eslintrc": "^3.3.1", "@eslint/js": "9.39.2", "@eslint/plugin-kit": "^0.4.1", "@humanfs/node": "^0.16.6", "@humanwhocodes/module-importer": "^1.0.1", "@humanwhocodes/retry": "^0.4.2", "@types/estree": "^1.0.6", "ajv": "^6.12.4", "chalk": "^4.0.0", "cross-spawn": "^7.0.6", "debug": "^4.3.2", "escape-string-regexp": "^4.0.0", "eslint-scope": "^8.4.0", "eslint-visitor-keys": "^4.2.1", "espree": "^10.4.0", "esquery": "^1.5.0", "esutils": "^2.0.2", "fast-deep-equal": "^3.1.3", "file-entry-cache": "^8.0.0", "find-up": "^5.0.0", "glob-parent": "^6.0.2", "ignore": "^5.2.0", "imurmurhash": "^0.1.4", "is-glob": "^4.0.0", "json-stable-stringify-without-jsonify": "^1.0.1", "lodash.merge": "^4.6.2", "minimatch": "^3.1.2", "natural-compare": "^1.4.0", "optionator": "^0.9.3" }, "peerDependencies": { "jiti": "*" }, "optionalPeers": ["jiti"], "bin": "bin/eslint.js" }, "sha512-LEyamqS7W5HB3ujJyvi0HQK/dtVINZvd5mAAp9eT5S/ujByGjiZLCzPcHVzuXbpJDJF/cxwHlfceVUDZ2lnSTw=="], - - "eslint-scope": ["eslint-scope@8.4.0", "", { "dependencies": { "esrecurse": "^4.3.0", "estraverse": "^5.2.0" } }, "sha512-sNXOfKCn74rt8RICKMvJS7XKV/Xk9kA7DyJr8mJik3S7Cwgy3qlkkmyS2uQB3jiJg6VNdZd/pDBJu0nvG2NlTg=="], - - "eslint-visitor-keys": ["eslint-visitor-keys@4.2.1", "", {}, "sha512-Uhdk5sfqcee/9H/rCOJikYz67o0a2Tw2hGRPOG2Y1R2dg7brRe1uG0yaNQDHu+TO/uQPF/5eCapvYSmHUjt7JQ=="], - - "espree": ["espree@10.4.0", "", { "dependencies": { "acorn": "^8.15.0", "acorn-jsx": "^5.3.2", "eslint-visitor-keys": "^4.2.1" } }, "sha512-j6PAQ2uUr79PZhBjP5C5fhl8e39FmRnOjsD5lGnWrFU8i2G776tBK7+nP8KuQUTTyAZUwfQqXAgrVH5MbH9CYQ=="], - - "esquery": ["esquery@1.6.0", "", { "dependencies": { "estraverse": "^5.1.0" } }, "sha512-ca9pw9fomFcKPvFLXhBKUK90ZvGibiGOvRJNbjljY7s7uq/5YO4BOzcYtJqExdx99rF6aAcnRxHmcUHcz6sQsg=="], - - "esrecurse": ["esrecurse@4.3.0", "", { "dependencies": { "estraverse": "^5.2.0" } }, "sha512-KmfKL3b6G+RXvP8N1vr3Tq1kL/oCFgn2NYXEtqP8/L3pKapUA4G8cFVaoF3SU323CD4XypR/ffioHmkti6/Tag=="], - - "estraverse": ["estraverse@5.3.0", "", {}, "sha512-MMdARuVEQziNTeJD8DgMqmhwR11BRQ/cBP+pLtYdSTnf3MIO8fFeiINEbX36ZdNlfU/7A9f3gUw49B3oQsvwBA=="], - - "estree-walker": ["estree-walker@2.0.2", "", {}, "sha512-Rfkk/Mp/DL7JVje3u18FxFujQlTNR2q6QfMSMB7AvCBx91NGj/ba3kCfza0f6dVDbw7YlRf/nDrn7pQrCCyQ/w=="], - - "esutils": ["esutils@2.0.3", "", {}, "sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g=="], - - "event-target-shim": ["event-target-shim@5.0.1", "", {}, "sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ=="], - - "eventemitter3": ["eventemitter3@5.0.1", "", {}, "sha512-GWkBvjiSZK87ELrYOSESUYeVIc9mvLLf/nXalMOS5dYrgZq9o5OVkbZAVM06CVxYsCwH9BDZFPlQTlPA1j4ahA=="], - - "events": ["events@3.3.0", "", {}, "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q=="], - - "events-universal": ["events-universal@1.0.1", "", { "dependencies": { "bare-events": "^2.7.0" } }, "sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw=="], - - "exifr": ["exifr@7.1.3", "", {}, "sha512-g/aje2noHivrRSLbAUtBPWFbxKdKhgj/xr1vATDdUXPOFYJlQ62Ft0oy+72V6XLIpDJfHs6gXLbBLAolqOXYRw=="], - - "expect-type": ["expect-type@1.3.0", "", {}, "sha512-knvyeauYhqjOYvQ66MznSMs83wmHrCycNEN6Ao+2AeYEfxUIkuiVxdEa1qlGEPK+We3n0THiDciYSsCcgW/DoA=="], - - "extend": ["extend@3.0.2", "", {}, "sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g=="], - - "fast-deep-equal": ["fast-deep-equal@3.1.3", "", {}, "sha512-f3qQ9oQy9j2AhBe/H9VC91wLmKBCCU/gDOnKNAYG5hswO7BLKj09Hc5HYNz9cGI++xlpDCIgDaitVs03ATR84Q=="], - - "fast-fifo": ["fast-fifo@1.3.2", "", {}, "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ=="], - - "fast-json-stable-stringify": ["fast-json-stable-stringify@2.1.0", "", {}, "sha512-lhd/wF+Lk98HZoTCtlVraHtfh5XYijIjalXck7saUtuanSDyLMxnHhSXEDJqHxD7msR8D0uCmqlkwjCV8xvwHw=="], - - "fast-levenshtein": ["fast-levenshtein@2.0.6", "", {}, "sha512-DCXu6Ifhqcks7TZKY3Hxp3y6qphY5SJZmrWMDrKcERSOXWQdMhU9Ig/PYrzyw/ul9jOIyh0N4M0tbC5hodg8dw=="], - - "fast-png": ["fast-png@6.4.0", "", { "dependencies": { "@types/pako": "^2.0.3", "iobuffer": "^5.3.2", "pako": "^2.1.0" } }, "sha512-kAqZq1TlgBjZcLr5mcN6NP5Rv4V2f22z00c3g8vRrwkcqjerx7BEhPbOnWCPqaHUl2XWQBJQvOT/FQhdMT7X/Q=="], - - "fast-xml-parser": ["fast-xml-parser@4.5.3", "", { "dependencies": { "strnum": "^1.1.1" }, "bin": { "fxparser": "src/cli/cli.js" } }, "sha512-RKihhV+SHsIUGXObeVy9AXiBbFwkVk7Syp8XgwN5U3JV416+Gwp/GO9i0JYKmikykgz/UHRrrV4ROuZEo/T0ig=="], - - "fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" } }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="], - - "fflate": ["fflate@0.8.2", "", {}, "sha512-cPJU47OaAoCbg0pBvzsgpTPhmhqI5eJjh/JIu8tPj5q+T7iLvW/JAYUqmE7KOB4R1ZyEhzBaIQpQpardBF5z8A=="], - - "figures": ["figures@3.2.0", "", { "dependencies": { "escape-string-regexp": "^1.0.5" } }, "sha512-yaduQFRKLXYOGgEn6AZau90j3ggSOyiqXU0F9JZfeXYhNa+Jk4X+s45A2zg5jns87GAFa34BBm2kXw4XpNcbdg=="], - - "file-entry-cache": ["file-entry-cache@8.0.0", "", { "dependencies": { "flat-cache": "^4.0.0" } }, "sha512-XXTUwCvisa5oacNGRP9SfNtYBNAMi+RPwBFmblZEF7N7swHYQS6/Zfk7SRwx4D5j3CH211YNRco1DEMNVfZCnQ=="], - - "filter-obj": ["filter-obj@1.1.0", "", {}, "sha512-8rXg1ZnX7xzy2NGDVkBVaAy+lSlPNwad13BtgSlLuxfIslyt5Vg64U7tFcCt4WS1R0hvtnQybT/IyCkGZ3DpXQ=="], - - "find-up": ["find-up@5.0.0", "", { "dependencies": { "locate-path": "^6.0.0", "path-exists": "^4.0.0" } }, "sha512-78/PXT1wlLLDgTzDs7sjq9hzz0vXD+zn+7wypEe4fXQxCmdmqfGsEPQxmiCSQI3ajFV91bVSsvNtrJRiW6nGng=="], - - "flat-cache": ["flat-cache@4.0.1", "", { "dependencies": { "flatted": "^3.2.9", "keyv": "^4.5.4" } }, "sha512-f7ccFPK3SXFHpx15UIGyRJ/FJQctuKZ0zVuN3frBo4HnK3cay9VEW0R6yPYFHC0AgqhukPzKjq22t5DmAyqGyw=="], - - "flatbuffers": ["flatbuffers@25.9.23", "", {}, "sha512-MI1qs7Lo4Syw0EOzUl0xjs2lsoeqFku44KpngfIduHBYvzm8h2+7K8YMQh1JtVVVrUvhLpNwqVi4DERegUJhPQ=="], - - "flatted": ["flatted@3.3.3", "", {}, "sha512-GX+ysw4PBCz0PzosHDepZGANEuFCMLrnRTiEy9McGjmkCQYwRq4A/X786G/fjM/+OjsWSU1ZrY5qyARZmO/uwg=="], - - "for-each": ["for-each@0.3.5", "", { "dependencies": { "is-callable": "^1.2.7" } }, "sha512-dKx12eRCVIzqCxFGplyFKJMPvLEWgmNtUrpTiJIR5u97zEhRG8ySrtboPHZXx7daLxQVrl643cTzbab2tkQjxg=="], - - "foreground-child": ["foreground-child@3.3.1", "", { "dependencies": { "cross-spawn": "^7.0.6", "signal-exit": "^4.0.1" } }, "sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw=="], - - "form-data": ["form-data@2.5.5", "", { "dependencies": { "asynckit": "^0.4.0", "combined-stream": "^1.0.8", "es-set-tostringtag": "^2.1.0", "hasown": "^2.0.2", "mime-types": "^2.1.35", "safe-buffer": "^5.2.1" } }, "sha512-jqdObeR2rxZZbPSGL+3VckHMYtu+f9//KXBsVny6JSX/pa38Fy+bGjuG8eW/H6USNQWhLi8Num++cU2yOCNz4A=="], - - "frac": ["frac@1.1.2", "", {}, "sha512-w/XBfkibaTl3YDqASwfDUqkna4Z2p9cFSr1aHDt0WoMTECnRfBOv2WArlZILlqgWlmdIlALXGpM2AOhEk5W3IA=="], - - "fs-constants": ["fs-constants@1.0.0", "", {}, "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow=="], - - "fsevents": ["fsevents@2.3.3", "", { "os": "darwin" }, "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw=="], - - "function-bind": ["function-bind@1.1.2", "", {}, "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA=="], - - "gaxios": ["gaxios@6.7.1", "", { "dependencies": { "extend": "^3.0.2", "https-proxy-agent": "^7.0.1", "is-stream": "^2.0.0", "node-fetch": "^2.6.9", "uuid": "^9.0.1" } }, "sha512-LDODD4TMYx7XXdpwxAVRAIAuB0bzv0s+ywFonY46k126qzQHT9ygyoa9tncmOiQmmDrik65UYsEkv3lbfqQ3yQ=="], - - "gcp-metadata": ["gcp-metadata@6.1.1", "", { "dependencies": { "gaxios": "^6.1.1", "google-logging-utils": "^0.0.2", "json-bigint": "^1.0.0" } }, "sha512-a4tiq7E0/5fTjxPAaH4jpjkSv/uCaU2p5KC6HVGrvl0cDjA8iBZv4vv1gyzlmK0ZUKqwpOyQMKzZQe3lTit77A=="], - - "generator-function": ["generator-function@2.0.1", "", {}, "sha512-SFdFmIJi+ybC0vjlHN0ZGVGHc3lgE0DxPAT0djjVg+kjOnSqclqmj0KQ7ykTOLP6YxoqOvuAODGdcHJn+43q3g=="], - - "get-caller-file": ["get-caller-file@2.0.5", "", {}, "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg=="], - - "get-east-asian-width": ["get-east-asian-width@1.4.0", "", {}, "sha512-QZjmEOC+IT1uk6Rx0sX22V6uHWVwbdbxf1faPqJ1QhLdGgsRGCZoyaQBm/piRdJy/D2um6hM1UP7ZEeQ4EkP+Q=="], - - "get-intrinsic": ["get-intrinsic@1.3.0", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "es-define-property": "^1.0.1", "es-errors": "^1.3.0", "es-object-atoms": "^1.1.1", "function-bind": "^1.1.2", "get-proto": "^1.0.1", "gopd": "^1.2.0", "has-symbols": "^1.1.0", "hasown": "^2.0.2", "math-intrinsics": "^1.1.0" } }, "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ=="], - - "get-pkg-repo": ["get-pkg-repo@4.2.1", "", { "dependencies": { "@hutson/parse-repository-url": "^3.0.0", "hosted-git-info": "^4.0.0", "through2": "^2.0.0", "yargs": "^16.2.0" }, "bin": "src/cli.js" }, "sha512-2+QbHjFRfGB74v/pYWjd5OhU3TDIC2Gv/YKUTk/tCvAz0pkn/Mz6P3uByuBimLOcPvN2jYdScl3xGFSrx0jEcA=="], - - "get-port": ["get-port@7.1.0", "", {}, "sha512-QB9NKEeDg3xxVwCCwJQ9+xycaz6pBB6iQ76wiWMl1927n0Kir6alPiP+yuiICLLU4jpMe08dXfpebuQppFA2zw=="], - - "get-proto": ["get-proto@1.0.1", "", { "dependencies": { "dunder-proto": "^1.0.1", "es-object-atoms": "^1.0.0" } }, "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g=="], - - "get-tsconfig": ["get-tsconfig@4.13.0", "", { "dependencies": { "resolve-pkg-maps": "^1.0.0" } }, "sha512-1VKTZJCwBrvbd+Wn3AOgQP/2Av+TfTCOlE4AcRJE72W1ksZXbAx8PPBR9RzgTeSPzlPMHrbANMH3LbltH73wxQ=="], - - "git-raw-commits": ["git-raw-commits@2.0.11", "", { "dependencies": { "dargs": "^7.0.0", "lodash": "^4.17.15", "meow": "^8.0.0", "split2": "^3.0.0", "through2": "^4.0.0" }, "bin": "cli.js" }, "sha512-VnctFhw+xfj8Va1xtfEqCUD2XDrbAPSJx+hSrE5K7fGdjZruW7XV+QOrN7LF/RJyvspRiD2I0asWsxFp0ya26A=="], - - "git-remote-origin-url": ["git-remote-origin-url@2.0.0", "", { "dependencies": { "gitconfiglocal": "^1.0.0", "pify": "^2.3.0" } }, "sha512-eU+GGrZgccNJcsDH5LkXR3PB9M958hxc7sbA8DFJjrv9j4L2P/eZfKhM+QD6wyzpiv+b1BpK0XrYCxkovtjSLw=="], - - "git-semver-tags": ["git-semver-tags@4.1.1", "", { "dependencies": { "meow": "^8.0.0", "semver": "^6.0.0" }, "bin": "cli.js" }, "sha512-OWyMt5zBe7xFs8vglMmhM9lRQzCWL3WjHtxNNfJTMngGym7pC1kh8sP6jevfydJ6LP3ZvGxfb6ABYgPUM0mtsA=="], - - "gitconfiglocal": ["gitconfiglocal@1.0.0", "", { "dependencies": { "ini": "^1.3.2" } }, "sha512-spLUXeTAVHxDtKsJc8FkFVgFtMdEN9qPGpL23VfSHx4fP4+Ds097IXLvymbnDH8FnmxX5Nr9bPw3A+AQ6mWEaQ=="], - - "glob": ["glob@10.5.0", "", { "dependencies": { "foreground-child": "^3.1.0", "jackspeak": "^3.1.2", "minimatch": "^9.0.4", "minipass": "^7.1.2", "package-json-from-dist": "^1.0.0", "path-scurry": "^1.11.1" }, "bin": "dist/esm/bin.mjs" }, "sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg=="], - - "glob-parent": ["glob-parent@6.0.2", "", { "dependencies": { "is-glob": "^4.0.3" } }, "sha512-XxwI8EOhVQgWp6iDL+3b0r86f4d6AX6zSU55HfB4ydCEuXLXc5FcYeOu+nnGftS4TEju/11rt4KJPTMgbfmv4A=="], - - "globals": ["globals@14.0.0", "", {}, "sha512-oahGvuMGQlPw/ivIYBjVSrWAfWLBeku5tpPE2fOPLi+WHffIWbuh2tCjhyQhTBPMf5E9jDEH4FOmTYgYwbKwtQ=="], - - "google-auth-library": ["google-auth-library@9.15.1", "", { "dependencies": { "base64-js": "^1.3.0", "ecdsa-sig-formatter": "^1.0.11", "gaxios": "^6.1.1", "gcp-metadata": "^6.1.0", "gtoken": "^7.0.0", "jws": "^4.0.0" } }, "sha512-Jb6Z0+nvECVz+2lzSMt9u98UsoakXxA2HGHMCxh+so3n90XgYWkq5dur19JAJV7ONiJY22yBTyJB1TSkvPq9Ng=="], - - "google-logging-utils": ["google-logging-utils@0.0.2", "", {}, "sha512-NEgUnEcBiP5HrPzufUkBzJOD/Sxsco3rLNo1F1TNf7ieU8ryUzBhqba8r756CjLX7rn3fHl6iLEwPYuqpoKgQQ=="], - - "gopd": ["gopd@1.2.0", "", {}, "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg=="], - - "graceful-fs": ["graceful-fs@4.2.11", "", {}, "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ=="], - - "gtoken": ["gtoken@7.1.0", "", { "dependencies": { "gaxios": "^6.0.0", "jws": "^4.0.0" } }, "sha512-pCcEwRi+TKpMlxAQObHDQ56KawURgyAf6jtIY046fJ5tIv3zDe/LEIubckAO8fj6JnAxLdmWkUfNyulQ2iKdEw=="], - - "guid-typescript": ["guid-typescript@1.0.9", "", {}, "sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ=="], - - "handlebars": ["handlebars@4.7.8", "", { "dependencies": { "minimist": "^1.2.5", "neo-async": "^2.6.2", "source-map": "^0.6.1", "wordwrap": "^1.0.0" }, "optionalDependencies": { "uglify-js": "^3.1.4" }, "bin": "bin/handlebars" }, "sha512-vafaFqs8MZkRrSX7sFVUdo3ap/eNiLnb4IakshzvP56X5Nr1iGKAIqdX6tMlm6HcNRIkr6AxO5jFEoJzzpT8aQ=="], - - "hard-rejection": ["hard-rejection@2.1.0", "", {}, "sha512-VIZB+ibDhx7ObhAe7OVtoEbuP4h/MuOTHJ+J8h/eBXotJYl0fBgR72xDFCKgIh22OJZIOVNxBMWuhAr10r8HdA=="], - - "has-flag": ["has-flag@4.0.0", "", {}, "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ=="], - - "has-property-descriptors": ["has-property-descriptors@1.0.2", "", { "dependencies": { "es-define-property": "^1.0.0" } }, "sha512-55JNKuIW+vq4Ke1BjOTjM2YctQIvCT7GFzHwmfZPGo5wnrgkid0YQtnAleFSqumZm4az3n2BS+erby5ipJdgrg=="], - - "has-symbols": ["has-symbols@1.1.0", "", {}, "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ=="], - - "has-tostringtag": ["has-tostringtag@1.0.2", "", { "dependencies": { "has-symbols": "^1.0.3" } }, "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw=="], - - "hasown": ["hasown@2.0.2", "", { "dependencies": { "function-bind": "^1.1.2" } }, "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ=="], - - "hosted-git-info": ["hosted-git-info@4.1.0", "", { "dependencies": { "lru-cache": "^6.0.0" } }, "sha512-kyCuEOWjJqZuDbRHzL8V93NzQhwIB71oFWSyzVo+KPZI+pnQPPxucdkrOZvkLRnrf5URsQM+IJ09Dw29cRALIA=="], - - "html-entities": ["html-entities@2.6.0", "", {}, "sha512-kig+rMn/QOVRvr7c86gQ8lWXq+Hkv6CbAH1hLu+RG338StTpE8Z0b44SDVaqVu7HGKf27frdmUYEs9hTUX/cLQ=="], - - "html-escaper": ["html-escaper@2.0.2", "", {}, "sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg=="], - - "html2canvas": ["html2canvas@1.4.1", "", { "dependencies": { "css-line-break": "^2.1.0", "text-segmentation": "^1.0.3" } }, "sha512-fPU6BHNpsyIhr8yyMpTLLxAbkaK8ArIBcmZIRiBLiDhjeqvXolaEmDGmELFuX9I4xDcaKKcJl+TKZLqruBbmWA=="], - - "http-proxy-agent": ["http-proxy-agent@5.0.0", "", { "dependencies": { "@tootallnate/once": "2", "agent-base": "6", "debug": "4" } }, "sha512-n2hY8YdoRE1i7r6M0w9DIw5GgZN0G25P8zLCRQ8rjXtTU3vsNFBI/vWK/UIeE6g5MUUz6avwAPXmL6Fy9D/90w=="], - - "https-proxy-agent": ["https-proxy-agent@7.0.6", "", { "dependencies": { "agent-base": "^7.1.2", "debug": "4" } }, "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw=="], - - "iconv-lite": ["iconv-lite@0.4.24", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3" } }, "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA=="], - - "ieee754": ["ieee754@1.2.1", "", {}, "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA=="], - - "ignore": ["ignore@7.0.5", "", {}, "sha512-Hs59xBNfUIunMFgWAbGX5cq6893IbWg4KnrjbYwX3tx0ztorVgTDA6B2sxf8ejHJ4wz8BqGUMYlnzNBer5NvGg=="], - - "immediate": ["immediate@3.0.6", "", {}, "sha512-XXOFtyqDjNDAQxVfYxuF7g9Il/IbWmmlQg2MYKOH8ExIT1qg6xc4zyS3HaEEATgs1btfzxq15ciUiY7gjSXRGQ=="], - - "import-fresh": ["import-fresh@3.3.1", "", { "dependencies": { "parent-module": "^1.0.0", "resolve-from": "^4.0.0" } }, "sha512-TR3KfrTZTYLPB6jUjfx6MF9WcWrHL9su5TObK4ZkYgBdWKPOFoSoQIdEuTuR82pmtxH2spWG9h6etwfr1pLBqQ=="], - - "imurmurhash": ["imurmurhash@0.1.4", "", {}, "sha512-JmXMZ6wuvDmLiHEml9ykzqO6lwFbof0GG4IkcGaENdCRDDmMVnny7s5HsIgHCbaq0w2MyPhDqkhTUgS2LU2PHA=="], - - "indent-string": ["indent-string@4.0.0", "", {}, "sha512-EdDDZu4A2OyIK7Lr/2zG+w5jmbuk1DVBnEwREQvBzspBJkCEbRa8GxU1lghYcaGJCnRWibjDXlq779X1/y5xwg=="], - - "inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="], - - "ini": ["ini@1.3.8", "", {}, "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew=="], - - "inquirer": ["inquirer@12.11.1", "", { "dependencies": { "@inquirer/ansi": "^1.0.2", "@inquirer/core": "^10.3.2", "@inquirer/prompts": "^7.10.1", "@inquirer/type": "^3.0.10", "mute-stream": "^2.0.0", "run-async": "^4.0.6", "rxjs": "^7.8.2" }, "peerDependencies": { "@types/node": ">=18" } }, "sha512-9VF7mrY+3OmsAfjH3yKz/pLbJ5z22E23hENKw3/LNSaA/sAt3v49bDRY+Ygct1xwuKT+U+cBfTzjCPySna69Qw=="], - - "iobuffer": ["iobuffer@5.4.0", "", {}, "sha512-DRebOWuqDvxunfkNJAlc3IzWIPD5xVxwUNbHr7xKB8E6aLJxIPfNX3CoMJghcFjpv6RWQsrcJbghtEwSPoJqMA=="], - - "ipaddr.js": ["ipaddr.js@2.3.0", "", {}, "sha512-Zv/pA+ciVFbCSBBjGfaKUya/CcGmUHzTydLMaTwrUUEM2DIEO3iZvueGxmacvmN50fGpGVKeTXpb2LcYQxeVdg=="], - - "is-arguments": ["is-arguments@1.2.0", "", { "dependencies": { "call-bound": "^1.0.2", "has-tostringtag": "^1.0.2" } }, "sha512-7bVbi0huj/wrIAOzb8U1aszg9kdi3KN/CyU19CTI7tAoZYEZoL9yCDXpbXN+uPsuWnP02cyug1gleqq+TU+YCA=="], - - "is-arrayish": ["is-arrayish@0.2.1", "", {}, "sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg=="], - - "is-callable": ["is-callable@1.2.7", "", {}, "sha512-1BC0BVFhS/p0qtw6enp8e+8OD0UrK0oFLztSjNzhcKA3WDuJxxAPXzPuPtKkjEY9UUoEWlX/8fgKeu2S8i9JTA=="], - - "is-core-module": ["is-core-module@2.16.1", "", { "dependencies": { "hasown": "^2.0.2" } }, "sha512-UfoeMA6fIJ8wTYFEUjelnaGI67v6+N7qXJEvQuIGa99l4xsCruSYOVSQ0uPANn4dAzm8lkYPaKLrrijLq7x23w=="], - - "is-docker": ["is-docker@3.0.0", "", { "bin": "cli.js" }, "sha512-eljcgEDlEns/7AXFosB5K/2nCM4P7FQPkGc/DWLy5rmFEWvZayGrik1d9/QIY5nJ4f9YsVvBkA6kJpHn9rISdQ=="], - - "is-extglob": ["is-extglob@2.1.1", "", {}, "sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ=="], - - "is-fullwidth-code-point": ["is-fullwidth-code-point@3.0.0", "", {}, "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg=="], - - "is-generator-function": ["is-generator-function@1.1.2", "", { "dependencies": { "call-bound": "^1.0.4", "generator-function": "^2.0.0", "get-proto": "^1.0.1", "has-tostringtag": "^1.0.2", "safe-regex-test": "^1.1.0" } }, "sha512-upqt1SkGkODW9tsGNG5mtXTXtECizwtS2kA161M+gJPc1xdb/Ax629af6YrTwcOeQHbewrPNlE5Dx7kzvXTizA=="], - - "is-glob": ["is-glob@4.0.3", "", { "dependencies": { "is-extglob": "^2.1.1" } }, "sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg=="], - - "is-inside-container": ["is-inside-container@1.0.0", "", { "dependencies": { "is-docker": "^3.0.0" }, "bin": "cli.js" }, "sha512-KIYLCCJghfHZxqjYBE7rEy0OBuTd5xCHS7tHVgvCLkx7StIoaxwNW3hCALgEUjFfeRk+MG/Qxmp/vtETEF3tRA=="], - - "is-interactive": ["is-interactive@2.0.0", "", {}, "sha512-qP1vozQRI+BMOPcjFzrjXuQvdak2pHNUMZoeG2eRbiSqyvbEf/wQtEOTOX1guk6E3t36RkaqiSt8A/6YElNxLQ=="], - - "is-module": ["is-module@1.0.0", "", {}, "sha512-51ypPSPCoTEIN9dy5Oy+h4pShgJmPCygKfyRCISBI+JoWT/2oJvK8QPxmwv7b/p239jXrm9M1mlQbyKJ5A152g=="], - - "is-obj": ["is-obj@2.0.0", "", {}, "sha512-drqDG3cbczxxEJRoOXcOjtdp1J/lyp1mNn0xaznRs8+muBhgQcrnbspox5X5fOw0HnMnbfDzvnEMEtqDEJEo8w=="], - - "is-plain-obj": ["is-plain-obj@1.1.0", "", {}, "sha512-yvkRyxmFKEOQ4pNXCmJG5AEQNlXJS5LaONXo5/cLdTZdWvsZ1ioJEonLGAosKlMWE8lwUy/bJzMjcw8az73+Fg=="], - - "is-reference": ["is-reference@1.2.1", "", { "dependencies": { "@types/estree": "*" } }, "sha512-U82MsXXiFIrjCK4otLT+o2NA2Cd2g5MLoOVXUZjIOhLurrRxpEXzI8O0KZHr3IjLvlAH1kTPYSuqer5T9ZVBKQ=="], - - "is-regex": ["is-regex@1.2.1", "", { "dependencies": { "call-bound": "^1.0.2", "gopd": "^1.2.0", "has-tostringtag": "^1.0.2", "hasown": "^2.0.2" } }, "sha512-MjYsKHO5O7mCsmRGxWcLWheFqN9DJ/2TmngvjKXihe6efViPqc274+Fx/4fYj/r03+ESvBdTXK0V6tA3rgez1g=="], - - "is-stream": ["is-stream@2.0.1", "", {}, "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg=="], - - "is-text-path": ["is-text-path@1.0.1", "", { "dependencies": { "text-extensions": "^1.0.0" } }, "sha512-xFuJpne9oFz5qDaodwmmG08e3CawH/2ZV8Qqza1Ko7Sk8POWbkRdwIoAWVhqvq0XeUzANEhKo2n0IXUGBm7A/w=="], - - "is-typed-array": ["is-typed-array@1.1.15", "", { "dependencies": { "which-typed-array": "^1.1.16" } }, "sha512-p3EcsicXjit7SaskXHs1hA91QxgTw46Fv6EFKKGS5DRFLD8yKnohjF3hxoju94b/OcMZoQukzpPpBE9uLVKzgQ=="], - - "is-unicode-supported": ["is-unicode-supported@2.1.0", "", {}, "sha512-mE00Gnza5EEB3Ds0HfMyllZzbBrmLOX3vfWoj9A9PEnTfratQ/BcaJOuMhnkhjXvb2+FkY3VuHqtAGpTPmglFQ=="], - - "is-wsl": ["is-wsl@3.1.0", "", { "dependencies": { "is-inside-container": "^1.0.0" } }, "sha512-UcVfVfaK4Sc4m7X3dUSoHoozQGBEFeDC+zVo06t98xe8CzHSZZBekNXH+tu0NalHolcJ/QAGqS46Hef7QXBIMw=="], - - "isarray": ["isarray@1.0.0", "", {}, "sha512-VLghIWNM6ELQzo7zwmcg0NmTVyWKYjvIeM83yjp0wRDTmUnrM678fQbcKBo6n2CJEF0szoG//ytg+TKla89ALQ=="], - - "isexe": ["isexe@2.0.0", "", {}, "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw=="], - - "istanbul-lib-coverage": ["istanbul-lib-coverage@3.2.2", "", {}, "sha512-O8dpsF+r0WV/8MNRKfnmrtCWhuKjxrq2w+jpzBL5UZKTi2LeVWnWOmWRxFlesJONmc+wLAGvKQZEOanko0LFTg=="], - - "istanbul-lib-report": ["istanbul-lib-report@3.0.1", "", { "dependencies": { "istanbul-lib-coverage": "^3.0.0", "make-dir": "^4.0.0", "supports-color": "^7.1.0" } }, "sha512-GCfE1mtsHGOELCU8e/Z7YWzpmybrx/+dSTfLrvY8qRmaY6zXTKWn6WQIjaAFw069icm6GVMNkgu0NzI4iPZUNw=="], - - "istanbul-lib-source-maps": ["istanbul-lib-source-maps@5.0.6", "", { "dependencies": { "@jridgewell/trace-mapping": "^0.3.23", "debug": "^4.1.1", "istanbul-lib-coverage": "^3.0.0" } }, "sha512-yg2d+Em4KizZC5niWhQaIomgf5WlL4vOOjZ5xGCmF8SnPE/mDWWXgvRExdcpCgh9lLRRa1/fSYp2ymmbJ1pI+A=="], - - "istanbul-reports": ["istanbul-reports@3.2.0", "", { "dependencies": { "html-escaper": "^2.0.0", "istanbul-lib-report": "^3.0.0" } }, "sha512-HGYWWS/ehqTV3xN10i23tkPkpH46MLCIMFNCaaKNavAXTF1RkqxawEPtnjnGZ6XKSInBKkiOA5BKS+aZiY3AvA=="], - - "jackspeak": ["jackspeak@3.4.3", "", { "dependencies": { "@isaacs/cliui": "^8.0.2" }, "optionalDependencies": { "@pkgjs/parseargs": "^0.11.0" } }, "sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw=="], - - "js-tokens": ["js-tokens@9.0.1", "", {}, "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ=="], - - "js-yaml": ["js-yaml@4.1.1", "", { "dependencies": { "argparse": "^2.0.1" }, "bin": "bin/js-yaml.js" }, "sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA=="], - - "json-bigint": ["json-bigint@1.0.0", "", { "dependencies": { "bignumber.js": "^9.0.0" } }, "sha512-SiPv/8VpZuWbvLSMtTDU8hEfrZWg/mH/nV/b4o0CYbSxu1UIQPLdwKOCIyLQX+VIPO5vrLX3i8qtqFyhdPSUSQ=="], - - "json-buffer": ["json-buffer@3.0.1", "", {}, "sha512-4bV5BfR2mqfQTJm+V5tPPdf+ZpuhiIvTuAB5g8kcrXOZpTT/QwwVRWBywX1ozr6lEuPdbHxwaJlm9G6mI2sfSQ=="], - - "json-parse-better-errors": ["json-parse-better-errors@1.0.2", "", {}, "sha512-mrqyZKfX5EhL7hvqcV6WG1yYjnjeuYDzDhhcAAUrq8Po85NBQBJP+ZDUT75qZQ98IkUoBqdkExkukOU7Ts2wrw=="], - - "json-parse-even-better-errors": ["json-parse-even-better-errors@2.3.1", "", {}, "sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w=="], - - "json-schema-traverse": ["json-schema-traverse@0.4.1", "", {}, "sha512-xbbCH5dCYU5T8LcEhhuh7HJ88HXuW3qsI3Y0zOZFKfZEHcpWiHU/Jxzk629Brsab/mMiHQti9wMP+845RPe3Vg=="], - - "json-stable-stringify-without-jsonify": ["json-stable-stringify-without-jsonify@1.0.1", "", {}, "sha512-Bdboy+l7tA3OGW6FjyFHWkP5LuByj1Tk33Ljyq0axyzdk9//JSi2u3fP1QSmd1KNwq6VOKYGlAu87CisVir6Pw=="], - - "json-stringify-safe": ["json-stringify-safe@5.0.1", "", {}, "sha512-ZClg6AaYvamvYEE82d3Iyd3vSSIjQ+odgjaTzRuO3s7toCdFKczob2i0zCh7JE8kWn17yvAWhUVxvqGwUalsRA=="], - - "jsonparse": ["jsonparse@1.3.1", "", {}, "sha512-POQXvpdL69+CluYsillJ7SUhKvytYjW9vG/GKpnf+xP8UWgYEM/RaMzHHofbALDiKbbP1W8UEYmgGl39WkPZsg=="], - - "jsonwebtoken": ["jsonwebtoken@9.0.3", "", { "dependencies": { "jws": "^4.0.1", "lodash.includes": "^4.3.0", "lodash.isboolean": "^3.0.3", "lodash.isinteger": "^4.0.4", "lodash.isnumber": "^3.0.3", "lodash.isplainobject": "^4.0.6", "lodash.isstring": "^4.0.1", "lodash.once": "^4.0.0", "ms": "^2.1.1", "semver": "^7.5.4" } }, "sha512-MT/xP0CrubFRNLNKvxJ2BYfy53Zkm++5bX9dtuPbqAeQpTVe0MQTFhao8+Cp//EmJp244xt6Drw/GVEGCUj40g=="], - - "jspdf": ["jspdf@3.0.4", "", { "dependencies": { "@babel/runtime": "^7.28.4", "fast-png": "^6.2.0", "fflate": "^0.8.1" }, "optionalDependencies": { "canvg": "^3.0.11", "core-js": "^3.6.0", "dompurify": "^3.2.4", "html2canvas": "^1.0.0-rc.5" } }, "sha512-dc6oQ8y37rRcHn316s4ngz/nOjayLF/FFxBF4V9zamQKRqXxyiH1zagkCdktdWhtoQId5K20xt1lB90XzkB+hQ=="], - - "jszip": ["jszip@3.10.1", "", { "dependencies": { "lie": "~3.3.0", "pako": "~1.0.2", "readable-stream": "~2.3.6", "setimmediate": "^1.0.5" } }, "sha512-xXDvecyTpGLrqFrvkrUSoxxfJI5AH7U8zxxtVclpsUtMCq4JQ290LY8AW5c7Ggnr/Y/oK+bQMbqK2qmtk3pN4g=="], - - "jwa": ["jwa@2.0.1", "", { "dependencies": { "buffer-equal-constant-time": "^1.0.1", "ecdsa-sig-formatter": "1.0.11", "safe-buffer": "^5.0.1" } }, "sha512-hRF04fqJIP8Abbkq5NKGN0Bbr3JxlQ+qhZufXVr0DvujKy93ZCbXZMHDL4EOtodSbCWxOqR8MS1tXA5hwqCXDg=="], - - "jws": ["jws@4.0.1", "", { "dependencies": { "jwa": "^2.0.1", "safe-buffer": "^5.0.1" } }, "sha512-EKI/M/yqPncGUUh44xz0PxSidXFr/+r0pA70+gIYhjv+et7yxM+s29Y+VGDkovRofQem0fs7Uvf4+YmAdyRduA=="], - - "keyv": ["keyv@4.5.4", "", { "dependencies": { "json-buffer": "3.0.1" } }, "sha512-oxVHkHR/EJf2CNXnWxRLW6mg7JyCCUcG0DtEGmL2ctUo1PNTin1PUil+r/+4r5MpVgC/fn1kjsx7mjSujKqIpw=="], - - "kind-of": ["kind-of@6.0.3", "", {}, "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw=="], - - "kleur": ["kleur@3.0.3", "", {}, "sha512-eTIzlVOSUR+JxdDFepEYcBMtZ9Qqdef+rnzWdRZuMbOywu5tO2w2N7rqjoANZ5k9vywhL6Br1VRjUIgTQx4E8w=="], - - "lazystream": ["lazystream@1.0.1", "", { "dependencies": { "readable-stream": "^2.0.5" } }, "sha512-b94GiNHQNy6JNTrt5w6zNyffMrNkXZb3KTkCZJb2V1xaEGCk093vkZ2jk3tpaeP33/OiXC+WvK9AxUebnf5nbw=="], - - "levn": ["levn@0.4.1", "", { "dependencies": { "prelude-ls": "^1.2.1", "type-check": "~0.4.0" } }, "sha512-+bT2uH4E5LGE7h/n3evcS/sQlJXCpIp6ym8OWJ5eV6+67Dsql/LaaT7qJBAt2rzfoa/5QBGBhxDix1dMt2kQKQ=="], - - "lie": ["lie@3.3.0", "", { "dependencies": { "immediate": "~3.0.5" } }, "sha512-UaiMJzeWRlEujzAuw5LokY1L5ecNQYZKfmyZ9L7wDHb/p5etKaxXhohBcrw0EYby+G/NA52vRSN4N39dxHAIwQ=="], - - "lines-and-columns": ["lines-and-columns@1.2.4", "", {}, "sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg=="], - - "load-json-file": ["load-json-file@4.0.0", "", { "dependencies": { "graceful-fs": "^4.1.2", "parse-json": "^4.0.0", "pify": "^3.0.0", "strip-bom": "^3.0.0" } }, "sha512-Kx8hMakjX03tiGTLAIdJ+lL0htKnXjEZN6hk/tozf/WOuYGdZBJrZ+rCJRbVCugsjB3jMLn9746NsQIf5VjBMw=="], - - "locate-path": ["locate-path@6.0.0", "", { "dependencies": { "p-locate": "^5.0.0" } }, "sha512-iPZK6eYjbxRu3uB4/WZ3EsEIMJFMqAoopl3R+zuq0UjcAm/MO6KCweDgPfP3elTztoKP3KtnVHxTn2NHBSDVUw=="], - - "lodash": ["lodash@4.17.21", "", {}, "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg=="], - - "lodash.camelcase": ["lodash.camelcase@4.3.0", "", {}, "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA=="], - - "lodash.includes": ["lodash.includes@4.3.0", "", {}, "sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w=="], - - "lodash.isboolean": ["lodash.isboolean@3.0.3", "", {}, "sha512-Bz5mupy2SVbPHURB98VAcw+aHh4vRV5IPNhILUCsOzRmsTmSQ17jIuqopAentWoehktxGd9e/hbIXq980/1QJg=="], - - "lodash.isinteger": ["lodash.isinteger@4.0.4", "", {}, "sha512-DBwtEWN2caHQ9/imiNeEA5ys1JoRtRfY3d7V9wkqtbycnAmTvRRmbHKDV4a0EYc678/dia0jrte4tjYwVBaZUA=="], - - "lodash.ismatch": ["lodash.ismatch@4.4.0", "", {}, "sha512-fPMfXjGQEV9Xsq/8MTSgUf255gawYRbjwMyDbcvDhXgV7enSZA0hynz6vMPnpAb5iONEzBHBPsT+0zes5Z301g=="], - - "lodash.isnumber": ["lodash.isnumber@3.0.3", "", {}, "sha512-QYqzpfwO3/CWf3XP+Z+tkQsfaLL/EnUlXWVkIk5FUPc4sBdTehEqZONuyRt2P67PXAk+NXmTBcc97zw9t1FQrw=="], - - "lodash.isplainobject": ["lodash.isplainobject@4.0.6", "", {}, "sha512-oSXzaWypCMHkPC3NvBEaPHf0KsA5mvPrOPgQWDsbg8n7orZ290M0BmC/jgRZ4vcJ6DTAhjrsSYgdsW/F+MFOBA=="], - - "lodash.isstring": ["lodash.isstring@4.0.1", "", {}, "sha512-0wJxfxH1wgO3GrbuP+dTTk7op+6L41QCXbGINEmD+ny/G/eCqGzxyCsh7159S+mgDDcoarnBw6PC1PS5+wUGgw=="], - - "lodash.merge": ["lodash.merge@4.6.2", "", {}, "sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ=="], - - "lodash.once": ["lodash.once@4.1.1", "", {}, "sha512-Sb487aTOCr9drQVL8pIxOzVhafOjZN9UU54hiN8PU3uAiSV7lx1yYNpbNmex2PK6dSJoNTSJUUswT651yww3Mg=="], - - "log-symbols": ["log-symbols@6.0.0", "", { "dependencies": { "chalk": "^5.3.0", "is-unicode-supported": "^1.3.0" } }, "sha512-i24m8rpwhmPIS4zscNzK6MSEhk0DUWa/8iYQWxhffV8jkI4Phvs3F+quL5xvS0gdQR0FyTCMMH33Y78dDTzzIw=="], - - "long": ["long@5.3.2", "", {}, "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA=="], - - "lop": ["lop@0.4.2", "", { "dependencies": { "duck": "^0.1.12", "option": "~0.2.1", "underscore": "^1.13.1" } }, "sha512-RefILVDQ4DKoRZsJ4Pj22TxE3omDO47yFpkIBoDKzkqPRISs5U1cnAdg/5583YPkWPaLIYHOKRMQSvjFsO26cw=="], - - "loupe": ["loupe@3.2.1", "", {}, "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ=="], - - "lru-cache": ["lru-cache@10.4.3", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="], - - "magic-string": ["magic-string@0.30.21", "", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.5" } }, "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ=="], - - "magicast": ["magicast@0.3.5", "", { "dependencies": { "@babel/parser": "^7.25.4", "@babel/types": "^7.25.4", "source-map-js": "^1.2.0" } }, "sha512-L0WhttDl+2BOsybvEOLK7fW3UA0OQ0IQ2d6Zl2x/a6vVRs3bAY0ECOSHHeL5jD+SbOpOCUEi0y1DgHEn9Qn1AQ=="], - - "make-dir": ["make-dir@4.0.0", "", { "dependencies": { "semver": "^7.5.3" } }, "sha512-hXdUTZYIVOt1Ex//jAQi+wTZZpUpwBj/0QsOzqegb3rGMMeJiSEu5xLHnYfBrRV4RH2+OCSOO95Is/7x1WJ4bw=="], - - "mammoth": ["mammoth@1.11.0", "", { "dependencies": { "@xmldom/xmldom": "^0.8.6", "argparse": "~1.0.3", "base64-js": "^1.5.1", "bluebird": "~3.4.0", "dingbat-to-unicode": "^1.0.1", "jszip": "^3.7.1", "lop": "^0.4.2", "path-is-absolute": "^1.0.0", "underscore": "^1.13.1", "xmlbuilder": "^10.0.0" }, "bin": "bin/mammoth" }, "sha512-BcEqqY/BOwIcI1iR5tqyVlqc3KIaMRa4egSoK83YAVrBf6+yqdAAbtUcFDCWX8Zef8/fgNZ6rl4VUv+vVX8ddQ=="], - - "map-obj": ["map-obj@4.3.0", "", {}, "sha512-hdN1wVrZbb29eBGiGjJbeP8JbKjq1urkHJ/LIP/NY48MZ1QVXUsQBV1G1zvYFHn1XE06cwjBsOI2K3Ulnj1YXQ=="], - - "math-intrinsics": ["math-intrinsics@1.1.0", "", {}, "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g=="], - - "meow": ["meow@8.1.2", "", { "dependencies": { "@types/minimist": "^1.2.0", "camelcase-keys": "^6.2.2", "decamelize-keys": "^1.1.0", "hard-rejection": "^2.1.0", "minimist-options": "4.1.0", "normalize-package-data": "^3.0.0", "read-pkg-up": "^7.0.1", "redent": "^3.0.0", "trim-newlines": "^3.0.0", "type-fest": "^0.18.0", "yargs-parser": "^20.2.3" } }, "sha512-r85E3NdZ+mpYk1C6RjPFEMSE+s1iZMuHtsHAqY0DT3jZczl0diWUZ8g6oU7h0M9cD2EL+PzaYghhCLzR0ZNn5Q=="], - - "mime": ["mime@4.1.0", "", { "bin": "bin/cli.js" }, "sha512-X5ju04+cAzsojXKes0B/S4tcYtFAJ6tTMuSPBEn9CPGlrWr8Fiw7qYeLT0XyH80HSoAoqWCaz+MWKh22P7G1cw=="], - - "mime-db": ["mime-db@1.52.0", "", {}, "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg=="], - - "mime-types": ["mime-types@2.1.35", "", { "dependencies": { "mime-db": "1.52.0" } }, "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw=="], - - "mimic-function": ["mimic-function@5.0.1", "", {}, "sha512-VP79XUPxV2CigYP3jWwAUFSku2aKqBH7uTAapFWCBqutsbmDo96KY5o8uh6U+/YSIn5OxJnXp73beVkpqMIGhA=="], - - "min-indent": ["min-indent@1.0.1", "", {}, "sha512-I9jwMn07Sy/IwOj3zVkVik2JTvgpaykDZEigL6Rx6N9LbMywwUSMtxET+7lVoDLLd3O3IXwJwvuuns8UB/HeAg=="], - - "minimatch": ["minimatch@3.1.2", "", { "dependencies": { "brace-expansion": "^1.1.7" } }, "sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw=="], - - "minimist": ["minimist@1.2.8", "", {}, "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA=="], - - "minimist-options": ["minimist-options@4.1.0", "", { "dependencies": { "arrify": "^1.0.1", "is-plain-obj": "^1.1.0", "kind-of": "^6.0.3" } }, "sha512-Q4r8ghd80yhO/0j1O3B2BjweX3fiHg9cdOwjJd2J76Q135c+NDxGCqdYKQ1SKBuFfgWbAUzBfvYjPUEeNgqN1A=="], - - "minio": ["minio@8.0.6", "", { "dependencies": { "async": "^3.2.4", "block-stream2": "^2.1.0", "browser-or-node": "^2.1.1", "buffer-crc32": "^1.0.0", "eventemitter3": "^5.0.1", "fast-xml-parser": "^4.4.1", "ipaddr.js": "^2.0.1", "lodash": "^4.17.21", "mime-types": "^2.1.35", "query-string": "^7.1.3", "stream-json": "^1.8.0", "through2": "^4.0.2", "web-encoding": "^1.1.5", "xml2js": "^0.5.0 || ^0.6.2" } }, "sha512-sOeh2/b/XprRmEtYsnNRFtOqNRTPDvYtMWh+spWlfsuCV/+IdxNeKVUMKLqI7b5Dr07ZqCPuaRGU/rB9pZYVdQ=="], - - "minipass": ["minipass@7.1.2", "", {}, "sha512-qOOzS1cBTWYF4BH8fVePDBOO9iptMnGUEZwNc/cMWnTV2nVLZ7VoNWEPHkYczZA0pdoA7dl6e7FL659nX9S2aw=="], - - "mkdirp": ["mkdirp@1.0.4", "", { "bin": "bin/cmd.js" }, "sha512-vVqVZQyf3WLx2Shd0qJ9xuvqgAyKPLAiqITEtqW0oIUjzo3PePDd6fW9iFz30ef7Ysp/oiWqbhszeGWW2T6Gzw=="], - - "mkdirp-classic": ["mkdirp-classic@0.5.3", "", {}, "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A=="], - - "modify-values": ["modify-values@1.0.1", "", {}, "sha512-xV2bxeN6F7oYjZWTe/YPAy6MN2M+sL4u/Rlm2AHCIVGfo2p1yGmBHQ6vHehl4bRTZBdHu3TSkWdYgkwpYzAGSw=="], - - "ms": ["ms@2.1.3", "", {}, "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA=="], - - "mute-stream": ["mute-stream@2.0.0", "", {}, "sha512-WWdIxpyjEn+FhQJQQv9aQAYlHoNVdzIzUySNV1gHUPDSdZJ3yZn7pAAbQcV7B56Mvu881q9FZV+0Vx2xC44VWA=="], - - "nan": ["nan@2.24.0", "", {}, "sha512-Vpf9qnVW1RaDkoNKFUvfxqAbtI8ncb8OJlqZ9wwpXzWPEsvsB1nvdUi6oYrHIkQ1Y/tMDnr1h4nczS0VB9Xykg=="], - - "nanoid": ["nanoid@3.3.11", "", { "bin": "bin/nanoid.cjs" }, "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w=="], - - "natural-compare": ["natural-compare@1.4.0", "", {}, "sha512-OWND8ei3VtNC9h7V60qff3SVobHr996CTwgxubgyQYEpg290h9J0buyECNNJexkFm5sOajh5G116RYA1c8ZMSw=="], - - "needle": ["needle@2.9.1", "", { "dependencies": { "debug": "^3.2.6", "iconv-lite": "^0.4.4", "sax": "^1.2.4" }, "bin": "bin/needle" }, "sha512-6R9fqJ5Zcmf+uYaFgdIHmLwNldn5HbK8L5ybn7Uz+ylX/rnOsSp1AHcvQSrCaFN+qNM1wpymHqD7mVasEOlHGQ=="], - - "neo-async": ["neo-async@2.6.2", "", {}, "sha512-Yd3UES5mWCSqR+qNT93S3UoYUkqAZ9lLg8a7g9rimsWmYGK8cVToA4/sF3RrshdyV3sAGMXVUmpMYOw+dLpOuw=="], - - "node-fetch": ["node-fetch@2.7.0", "", { "dependencies": { "whatwg-url": "^5.0.0" }, "peerDependencies": { "encoding": "^0.1.0" }, "optionalPeers": ["encoding"] }, "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A=="], - - "normalize-package-data": ["normalize-package-data@3.0.3", "", { "dependencies": { "hosted-git-info": "^4.0.1", "is-core-module": "^2.5.0", "semver": "^7.3.4", "validate-npm-package-license": "^3.0.1" } }, "sha512-p2W1sgqij3zMMyRC067Dg16bfzVH+w7hyegmpIvZ4JNjqtGOVAIvLmjBx3yP7YTe9vKJgkoNOPjwQGogDoMXFA=="], - - "normalize-path": ["normalize-path@3.0.0", "", {}, "sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA=="], - - "once": ["once@1.4.0", "", { "dependencies": { "wrappy": "1" } }, "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w=="], - - "onetime": ["onetime@7.0.0", "", { "dependencies": { "mimic-function": "^5.0.0" } }, "sha512-VXJjc87FScF88uafS3JllDgvAm+c/Slfz06lorj2uAY34rlUu0Nt+v8wreiImcrgAjjIHp1rXpTDlLOGw29WwQ=="], - - "onnxruntime-common": ["onnxruntime-common@1.23.2", "", {}, "sha512-5LFsC9Dukzp2WV6kNHYLNzp8sT6V02IubLCbzw2Xd6X5GOlr65gAX6xiJwyi2URJol/s71gaQLC5F2C25AAR2w=="], - - "onnxruntime-web": ["onnxruntime-web@1.23.2", "", { "dependencies": { "flatbuffers": "^25.1.24", "guid-typescript": "^1.0.9", "long": "^5.2.3", "onnxruntime-common": "1.23.2", "platform": "^1.3.6", "protobufjs": "^7.2.4" } }, "sha512-T09JUtMn+CZLk3mFwqiH0lgQf+4S7+oYHHtk6uhaYAAJI95bTcKi5bOOZYwORXfS/RLZCjDDEXGWIuOCAFlEjg=="], - - "open": ["open@10.2.0", "", { "dependencies": { "default-browser": "^5.2.1", "define-lazy-prop": "^3.0.0", "is-inside-container": "^1.0.0", "wsl-utils": "^0.1.0" } }, "sha512-YgBpdJHPyQ2UE5x+hlSXcnejzAvD0b22U2OuAP+8OnlJT+PjWPxtgmGqKKc+RgTM63U9gN0YzrYc71R2WT/hTA=="], - - "option": ["option@0.2.4", "", {}, "sha512-pkEqbDyl8ou5cpq+VsnQbe/WlEy5qS7xPzMS1U55OCG9KPvwFD46zDbxQIj3egJSFc3D+XhYOPUzz49zQAVy7A=="], - - "optionator": ["optionator@0.9.4", "", { "dependencies": { "deep-is": "^0.1.3", "fast-levenshtein": "^2.0.6", "levn": "^0.4.1", "prelude-ls": "^1.2.1", "type-check": "^0.4.0", "word-wrap": "^1.2.5" } }, "sha512-6IpQ7mKUxRcZNLIObR0hz7lxsapSSIYNZJwXPGeF0mTVqGKFIXj1DQcMoT22S3ROcLyY/rz0PWaWZ9ayWmad9g=="], - - "ora": ["ora@8.2.0", "", { "dependencies": { "chalk": "^5.3.0", "cli-cursor": "^5.0.0", "cli-spinners": "^2.9.2", "is-interactive": "^2.0.0", "is-unicode-supported": "^2.0.0", "log-symbols": "^6.0.0", "stdin-discarder": "^0.2.2", "string-width": "^7.2.0", "strip-ansi": "^7.1.0" } }, "sha512-weP+BZ8MVNnlCm8c0Qdc1WSWq4Qn7I+9CJGm7Qali6g44e/PUzbjNqJX5NJ9ljlNMosfJvg1fKEGILklK9cwnw=="], - - "p-limit": ["p-limit@3.1.0", "", { "dependencies": { "yocto-queue": "^0.1.0" } }, "sha512-TYOanM3wGwNGsZN2cVTYPArw454xnXj5qmWF1bEoAc4+cU/ol7GVh7odevjp1FNHduHc3KZMcFduxU5Xc6uJRQ=="], - - "p-locate": ["p-locate@5.0.0", "", { "dependencies": { "p-limit": "^3.0.2" } }, "sha512-LaNjtRWUBY++zB5nE/NwcaoMylSPk+S+ZHNB1TzdbMJMny6dynpAGt7X/tl/QYq3TIeE6nxHppbo2LGymrG5Pw=="], - - "p-try": ["p-try@2.2.0", "", {}, "sha512-R4nPAVTAU0B9D35/Gk3uJf/7XYbQcyohSKdvAxIRSNghFl4e71hVoGnBNQz9cWaXxO2I10KTC+3jMdvvoKw6dQ=="], - - "package-json-from-dist": ["package-json-from-dist@1.0.1", "", {}, "sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw=="], - - "pako": ["pako@2.1.0", "", {}, "sha512-w+eufiZ1WuJYgPXbV/PO3NCMEc3xqylkKHzp8bxp1uW4qaSNQUkwmLLEc3kKsfz8lpV1F8Ht3U1Cm+9Srog2ug=="], - - "parent-module": ["parent-module@1.0.1", "", { "dependencies": { "callsites": "^3.0.0" } }, "sha512-GQ2EWRpQV8/o+Aw8YqtfZZPfNRWZYkbidE9k5rpl/hC3vtHHBfGm2Ifi6qWV+coDGkrUKZAxE3Lot5kcsRlh+g=="], - - "parse-json": ["parse-json@4.0.0", "", { "dependencies": { "error-ex": "^1.3.1", "json-parse-better-errors": "^1.0.1" } }, "sha512-aOIos8bujGN93/8Ox/jPLh7RwVnPEysynVFE+fQZyg6jKELEHwzgKdLRFHUgXJL6kylijVSBC4BvN9OmsB48Rw=="], - - "path-exists": ["path-exists@4.0.0", "", {}, "sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w=="], - - "path-is-absolute": ["path-is-absolute@1.0.1", "", {}, "sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg=="], - - "path-key": ["path-key@3.1.1", "", {}, "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q=="], - - "path-parse": ["path-parse@1.0.7", "", {}, "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw=="], - - "path-scurry": ["path-scurry@1.11.1", "", { "dependencies": { "lru-cache": "^10.2.0", "minipass": "^5.0.0 || ^6.0.2 || ^7.0.0" } }, "sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA=="], - - "path-type": ["path-type@3.0.0", "", { "dependencies": { "pify": "^3.0.0" } }, "sha512-T2ZUsdZFHgA3u4e5PfPbjd7HDDpxPnQb5jN0SrDsjNSuVXHJqtwTnWqG0B1jZrgmJ/7lj1EmVIByWt1gxGkWvg=="], - - "pathe": ["pathe@2.0.3", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="], - - "pathval": ["pathval@2.0.1", "", {}, "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ=="], - - "pdfjs-dist": ["pdfjs-dist@4.10.38", "", { "optionalDependencies": { "@napi-rs/canvas": "^0.1.65" } }, "sha512-/Y3fcFrXEAsMjJXeL9J8+ZG9U01LbuWaYypvDW2ycW1jL269L3js3DVBjDJ0Up9Np1uqDXsDrRihHANhZOlwdQ=="], - - "performance-now": ["performance-now@2.1.0", "", {}, "sha512-7EAHlyLHI56VEIdK57uwHdHKIaAGbnXPiw0yWbarQZOKaKpvUIgW0jWRVLiatnM+XXlSwsanIBH/hzGMJulMow=="], - - "picocolors": ["picocolors@1.1.1", "", {}, "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="], - - "picomatch": ["picomatch@4.0.3", "", {}, "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q=="], - - "pify": ["pify@2.3.0", "", {}, "sha512-udgsAY+fTnvv7kI7aaxbqwWNb0AHiB0qBO89PZKPkoTmGOgdbrHDKD+0B2X4uTfJ/FT1R09r9gTsjUjNJotuog=="], - - "platform": ["platform@1.3.6", "", {}, "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg=="], - - "possible-typed-array-names": ["possible-typed-array-names@1.1.0", "", {}, "sha512-/+5VFTchJDoVj3bhoqi6UeymcD00DAwb1nJwamzPvHEszJ4FpF6SNNbUbOS8yI56qHzdV8eK0qEfOSiodkTdxg=="], - - "postcss": ["postcss@8.5.6", "", { "dependencies": { "nanoid": "^3.3.11", "picocolors": "^1.1.1", "source-map-js": "^1.2.1" } }, "sha512-3Ybi1tAuwAP9s0r1UQ2J4n5Y0G05bJkpUIO0/bI9MhwmD70S5aTWbXGBwxHrelT+XM1k6dM0pk+SwNkpTRN7Pg=="], - - "prelude-ls": ["prelude-ls@1.2.1", "", {}, "sha512-vkcDPrRZo1QZLbn5RLGPpg/WmIQ65qoWWhcGKf/b5eplkkarX0m9z8ppCat4mlOqUsWpyNuYgO3VRyrYHSzX5g=="], - - "probe-image-size": ["probe-image-size@7.2.3", "", { "dependencies": { "lodash.merge": "^4.6.2", "needle": "^2.5.2", "stream-parser": "~0.3.1" } }, "sha512-HubhG4Rb2UH8YtV4ba0Vp5bQ7L78RTONYu/ujmCu5nBI8wGv24s4E9xSKBi0N1MowRpxk76pFCpJtW0KPzOK0w=="], - - "process": ["process@0.11.10", "", {}, "sha512-cdGef/drWFoydD1JsMzuFf8100nZl+GT+yacc2bEced5f9Rjk4z+WtFUTBu9PhOi9j/jfmBPu0mMEY4wIdAF8A=="], - - "process-nextick-args": ["process-nextick-args@2.0.1", "", {}, "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag=="], - - "prompts": ["prompts@2.4.2", "", { "dependencies": { "kleur": "^3.0.3", "sisteransi": "^1.0.5" } }, "sha512-NxNv/kLguCA7p3jE8oL2aEBsrJWgAakBpgmgK6lpPWV+WuOmY6r2/zbAVnP+T8bQlA0nzHXSJSJW0Hq7ylaD2Q=="], - - "proper-lockfile": ["proper-lockfile@4.1.2", "", { "dependencies": { "graceful-fs": "^4.2.4", "retry": "^0.12.0", "signal-exit": "^3.0.2" } }, "sha512-TjNPblN4BwAWMXU8s9AEz4JmQxnD1NNL7bNOY/AKUzyamc379FWASUhc/K1pL2noVb+XmZKLL68cjzLsiOAMaA=="], - - "properties-reader": ["properties-reader@2.3.0", "", { "dependencies": { "mkdirp": "^1.0.4" } }, "sha512-z597WicA7nDZxK12kZqHr2TcvwNU1GCfA5UwfDY/HDp3hXPoPlb5rlEx9bwGTiJnc0OqbBTkU975jDToth8Gxw=="], - - "protobufjs": ["protobufjs@7.5.4", "", { "dependencies": { "@protobufjs/aspromise": "^1.1.2", "@protobufjs/base64": "^1.1.2", "@protobufjs/codegen": "^2.0.4", "@protobufjs/eventemitter": "^1.1.0", "@protobufjs/fetch": "^1.1.0", "@protobufjs/float": "^1.0.2", "@protobufjs/inquire": "^1.1.0", "@protobufjs/path": "^1.1.2", "@protobufjs/pool": "^1.1.0", "@protobufjs/utf8": "^1.1.0", "@types/node": ">=13.7.0", "long": "^5.0.0" } }, "sha512-CvexbZtbov6jW2eXAvLukXjXUW1TzFaivC46BpWc/3BpcCysb5Vffu+B3XHMm8lVEuy2Mm4XGex8hBSg1yapPg=="], - - "pump": ["pump@3.0.3", "", { "dependencies": { "end-of-stream": "^1.1.0", "once": "^1.3.1" } }, "sha512-todwxLMY7/heScKmntwQG8CXVkWUOdYxIvY2s0VWAAMh/nd8SoYiRaKjlr7+iCs984f2P8zvrfWcDDYVb73NfA=="], - - "punycode": ["punycode@2.3.1", "", {}, "sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg=="], - - "q": ["q@1.5.1", "", {}, "sha512-kV/CThkXo6xyFEZUugw/+pIOywXcDbFYgSct5cT3gqlbkBE1SJdwy6UQoZvodiWF/ckQLZyDE/Bu1M6gVu5lVw=="], - - "query-string": ["query-string@7.1.3", "", { "dependencies": { "decode-uri-component": "^0.2.2", "filter-obj": "^1.1.0", "split-on-first": "^1.0.0", "strict-uri-encode": "^2.0.0" } }, "sha512-hh2WYhq4fi8+b+/2Kg9CEge4fDPvHS534aOOvOZeQ3+Vf2mCFsaFBYj0i+iXcAq6I9Vzp5fjMFBlONvayDC1qg=="], - - "quick-lru": ["quick-lru@4.0.1", "", {}, "sha512-ARhCpm70fzdcvNQfPoy49IaanKkTlRWF2JMzqhcJbhSFRZv7nPTvZJdcY7301IPmvW+/p0RgIWnQDLJxifsQ7g=="], - - "raf": ["raf@3.4.1", "", { "dependencies": { "performance-now": "^2.1.0" } }, "sha512-Sq4CW4QhwOHE8ucn6J34MqtZCeWFP2aQSmrlroYgqAV1PjStIhJXxYuTgUIfkEk7zTLjmIjLmU5q+fbD1NnOJA=="], - - "randombytes": ["randombytes@2.1.0", "", { "dependencies": { "safe-buffer": "^5.1.0" } }, "sha512-vYl3iOX+4CKUWuxGi9Ukhie6fsqXqS9FE2Zaic4tNFD2N2QQaXOMFbuKK4QmDHC0JO6B1Zp41J0LpT0oR68amQ=="], - - "read-pkg": ["read-pkg@3.0.0", "", { "dependencies": { "load-json-file": "^4.0.0", "normalize-package-data": "^2.3.2", "path-type": "^3.0.0" } }, "sha512-BLq/cCO9two+lBgiTYNqD6GdtK8s4NpaWrl6/rCO9w0TUS8oJl7cmToOZfRYllKTISY6nt1U7jQ53brmKqY6BA=="], - - "read-pkg-up": ["read-pkg-up@3.0.0", "", { "dependencies": { "find-up": "^2.0.0", "read-pkg": "^3.0.0" } }, "sha512-YFzFrVvpC6frF1sz8psoHDBGF7fLPc+llq/8NB43oagqWkx8ar5zYtsTORtOjw9W2RHLpWP+zTWwBvf1bCmcSw=="], - - "readable-stream": ["readable-stream@3.6.2", "", { "dependencies": { "inherits": "^2.0.3", "string_decoder": "^1.1.1", "util-deprecate": "^1.0.1" } }, "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA=="], - - "readdir-glob": ["readdir-glob@1.1.3", "", { "dependencies": { "minimatch": "^5.1.0" } }, "sha512-v05I2k7xN8zXvPD9N+z/uhXPaj0sUFCe2rcWZIpBsqxfP7xXFQ0tipAd/wjj1YxWyWtUS5IDJpOG82JKt2EAVA=="], - - "redent": ["redent@3.0.0", "", { "dependencies": { "indent-string": "^4.0.0", "strip-indent": "^3.0.0" } }, "sha512-6tDA8g98We0zd0GvVeMT9arEOnTw9qM03L9cJXaCjrip1OO764RDBLBfrB4cwzNGDj5OA5ioymC9GkizgWJDUg=="], - - "regenerator-runtime": ["regenerator-runtime@0.13.11", "", {}, "sha512-kY1AZVr2Ra+t+piVaJ4gxaFaReZVH40AKNo7UCX6W+dEwBo/2oZJzqfuN1qLq1oL45o56cPaTXELwrTh8Fpggg=="], - - "require-directory": ["require-directory@2.1.1", "", {}, "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q=="], - - "resolve": ["resolve@1.22.11", "", { "dependencies": { "is-core-module": "^2.16.1", "path-parse": "^1.0.7", "supports-preserve-symlinks-flag": "^1.0.0" }, "bin": "bin/resolve" }, "sha512-RfqAvLnMl313r7c9oclB1HhUEAezcpLjz95wFH4LVuhk9JF/r22qmVP9AMmOU4vMX7Q8pN8jwNg/CSpdFnMjTQ=="], - - "resolve-from": ["resolve-from@4.0.0", "", {}, "sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g=="], - - "resolve-pkg-maps": ["resolve-pkg-maps@1.0.0", "", {}, "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw=="], - - "restore-cursor": ["restore-cursor@5.1.0", "", { "dependencies": { "onetime": "^7.0.0", "signal-exit": "^4.1.0" } }, "sha512-oMA2dcrw6u0YfxJQXm342bFKX/E4sG9rbTzO9ptUcR/e8A33cHuvStiYOwH7fszkZlZ1z/ta9AAoPk2F4qIOHA=="], - - "retry": ["retry@0.12.0", "", {}, "sha512-9LkiTwjUh6rT555DtE9rTX+BKByPfrMzEAtnlEtdEwr3Nkffwiihqe2bWADg+OQRjt9gl6ICdmB/ZFDCGAtSow=="], - - "retry-request": ["retry-request@7.0.2", "", { "dependencies": { "@types/request": "^2.48.8", "extend": "^3.0.2", "teeny-request": "^9.0.0" } }, "sha512-dUOvLMJ0/JJYEn8NrpOaGNE7X3vpI5XlZS/u0ANjqtcZVKnIxP7IgCFwrKTxENw29emmwug53awKtaMm4i9g5w=="], - - "rgbcolor": ["rgbcolor@1.0.1", "", {}, "sha512-9aZLIrhRaD97sgVhtJOW6ckOEh6/GnvQtdVNfdZ6s67+3/XwLS9lBcQYzEEhYVeUowN7pRzMLsyGhK2i/xvWbw=="], - - "roaring-wasm": ["roaring-wasm@1.1.0", "", {}, "sha512-mhNqA0BOqIW7k4ZYSYe3kCyvn5T3VWT+2661G7fZH0C6XcVkGoTDLAqne7b47xCNQE6LhuYviMKBnzbOiBXkdw=="], - - "rollup": ["rollup@4.53.5", "", { "dependencies": { "@types/estree": "1.0.8" }, "optionalDependencies": { "@rollup/rollup-android-arm-eabi": "4.53.5", "@rollup/rollup-android-arm64": "4.53.5", "@rollup/rollup-darwin-arm64": "4.53.5", "@rollup/rollup-darwin-x64": "4.53.5", "@rollup/rollup-freebsd-arm64": "4.53.5", "@rollup/rollup-freebsd-x64": "4.53.5", "@rollup/rollup-linux-arm-gnueabihf": "4.53.5", "@rollup/rollup-linux-arm-musleabihf": "4.53.5", "@rollup/rollup-linux-arm64-gnu": "4.53.5", "@rollup/rollup-linux-arm64-musl": "4.53.5", "@rollup/rollup-linux-loong64-gnu": "4.53.5", "@rollup/rollup-linux-ppc64-gnu": "4.53.5", "@rollup/rollup-linux-riscv64-gnu": "4.53.5", "@rollup/rollup-linux-riscv64-musl": "4.53.5", "@rollup/rollup-linux-s390x-gnu": "4.53.5", "@rollup/rollup-linux-x64-gnu": "4.53.5", "@rollup/rollup-linux-x64-musl": "4.53.5", "@rollup/rollup-openharmony-arm64": "4.53.5", "@rollup/rollup-win32-arm64-msvc": "4.53.5", "@rollup/rollup-win32-ia32-msvc": "4.53.5", "@rollup/rollup-win32-x64-gnu": "4.53.5", "@rollup/rollup-win32-x64-msvc": "4.53.5", "fsevents": "~2.3.2" }, "bin": "dist/bin/rollup" }, "sha512-iTNAbFSlRpcHeeWu73ywU/8KuU/LZmNCSxp6fjQkJBD3ivUb8tpDrXhIxEzA05HlYMEwmtaUnb3RP+YNv162OQ=="], - - "run-applescript": ["run-applescript@7.1.0", "", {}, "sha512-DPe5pVFaAsinSaV6QjQ6gdiedWDcRCbUuiQfQa2wmWV7+xC9bGulGI8+TdRmoFkAPaBXk8CrAbnlY2ISniJ47Q=="], - - "run-async": ["run-async@4.0.6", "", {}, "sha512-IoDlSLTs3Yq593mb3ZoKWKXMNu3UpObxhgA/Xuid5p4bbfi2jdY1Hj0m1K+0/tEuQTxIGMhQDqGjKb7RuxGpAQ=="], - - "rxjs": ["rxjs@7.8.2", "", { "dependencies": { "tslib": "^2.1.0" } }, "sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA=="], - - "safe-buffer": ["safe-buffer@5.2.1", "", {}, "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ=="], - - "safe-regex-test": ["safe-regex-test@1.1.0", "", { "dependencies": { "call-bound": "^1.0.2", "es-errors": "^1.3.0", "is-regex": "^1.2.1" } }, "sha512-x/+Cz4YrimQxQccJf5mKEbIa1NzeCRNI5Ecl/ekmlYaampdNLPalVyIcCZNNH3MvmqBugV5TMYZXv0ljslUlaw=="], - - "safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="], - - "sax": ["sax@1.4.3", "", {}, "sha512-yqYn1JhPczigF94DMS+shiDMjDowYO6y9+wB/4WgO0Y19jWYk0lQ4tuG5KI7kj4FTp1wxPj5IFfcrz/s1c3jjQ=="], - - "semver": ["semver@7.7.3", "", { "bin": "bin/semver.js" }, "sha512-SdsKMrI9TdgjdweUSR9MweHA4EJ8YxHn8DFaDisvhVlUOe4BF1tLD7GAj0lIqWVl+dPb/rExr0Btby5loQm20Q=="], - - "serialize-javascript": ["serialize-javascript@6.0.2", "", { "dependencies": { "randombytes": "^2.1.0" } }, "sha512-Saa1xPByTTq2gdeFZYLLo+RFE35NHZkAbqZeWNd3BpzppeVisAqpDjcp8dyf6uIvEqJRd46jemmyA4iFIeVk8g=="], - - "set-function-length": ["set-function-length@1.2.2", "", { "dependencies": { "define-data-property": "^1.1.4", "es-errors": "^1.3.0", "function-bind": "^1.1.2", "get-intrinsic": "^1.2.4", "gopd": "^1.0.1", "has-property-descriptors": "^1.0.2" } }, "sha512-pgRc4hJ4/sNjWCSS9AmnS40x3bNMDTknHgL5UaMBTMyJnU90EgWh1Rz+MC9eFu4BuN/UwZjKQuY/1v3rM7HMfg=="], - - "setimmediate": ["setimmediate@1.0.5", "", {}, "sha512-MATJdZp8sLqDl/68LfQmbP8zKPLQNV6BIZoIgrscFDQ+RsvK/BxeDQOgyxKKoh0y/8h3BqVFnCqQ/gd+reiIXA=="], - - "shebang-command": ["shebang-command@2.0.0", "", { "dependencies": { "shebang-regex": "^3.0.0" } }, "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA=="], - - "shebang-regex": ["shebang-regex@3.0.0", "", {}, "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A=="], - - "siginfo": ["siginfo@2.0.0", "", {}, "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g=="], - - "signal-exit": ["signal-exit@3.0.7", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="], - - "sisteransi": ["sisteransi@1.0.5", "", {}, "sha512-bLGGlR1QxBcynn2d5YmDX4MGjlZvy2MRBDRNHLJ8VI6l6+9FUiyTFNJ0IveOSP0bcXgVDPRcfGqA0pjaqUpfVg=="], - - "smob": ["smob@1.5.0", "", {}, "sha512-g6T+p7QO8npa+/hNx9ohv1E5pVCmWrVCUzUXJyLdMmftX6ER0oiWY/w9knEonLpnOp6b6FenKnMfR8gqwWdwig=="], - - "source-map": ["source-map@0.6.1", "", {}, "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g=="], - - "source-map-js": ["source-map-js@1.2.1", "", {}, "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA=="], - - "source-map-support": ["source-map-support@0.5.21", "", { "dependencies": { "buffer-from": "^1.0.0", "source-map": "^0.6.0" } }, "sha512-uBHU3L3czsIyYXKX88fdrGovxdSCoTGDRZ6SYXtSRxLZUzHg5P/66Ht6uoUlHu9EZod+inXhKo3qQgwXUT/y1w=="], - - "spdx-correct": ["spdx-correct@3.2.0", "", { "dependencies": { "spdx-expression-parse": "^3.0.0", "spdx-license-ids": "^3.0.0" } }, "sha512-kN9dJbvnySHULIluDHy32WHRUu3Og7B9sbY7tsFLctQkIqnMh3hErYgdMjTYuqmcXX+lK5T1lnUt3G7zNswmZA=="], - - "spdx-exceptions": ["spdx-exceptions@2.5.0", "", {}, "sha512-PiU42r+xO4UbUS1buo3LPJkjlO7430Xn5SVAhdpzzsPHsjbYVflnnFdATgabnLude+Cqu25p6N+g2lw/PFsa4w=="], - - "spdx-expression-parse": ["spdx-expression-parse@3.0.1", "", { "dependencies": { "spdx-exceptions": "^2.1.0", "spdx-license-ids": "^3.0.0" } }, "sha512-cbqHunsQWnJNE6KhVSMsMeH5H/L9EpymbzqTQ3uLwNCLZ1Q481oWaofqH7nO6V07xlXwY6PhQdQ2IedWx/ZK4Q=="], - - "spdx-license-ids": ["spdx-license-ids@3.0.22", "", {}, "sha512-4PRT4nh1EImPbt2jASOKHX7PB7I+e4IWNLvkKFDxNhJlfjbYlleYQh285Z/3mPTHSAK/AvdMmw5BNNuYH8ShgQ=="], - - "split": ["split@1.0.1", "", { "dependencies": { "through": "2" } }, "sha512-mTyOoPbrivtXnwnIxZRFYRrPNtEFKlpB2fvjSnCQUiAA6qAZzqwna5envK4uk6OIeP17CsdF3rSBGYVBsU0Tkg=="], - - "split-ca": ["split-ca@1.0.1", "", {}, "sha512-Q5thBSxp5t8WPTTJQS59LrGqOZqOsrhDGDVm8azCqIBjSBd7nd9o2PM+mDulQQkh8h//4U6hFZnc/mul8t5pWQ=="], - - "split-on-first": ["split-on-first@1.1.0", "", {}, "sha512-43ZssAJaMusuKWL8sKUBQXHWOpq8d6CfN/u1p4gUzfJkM05C8rxTmYrkIPTXapZpORA6LkkzcUulJ8FqA7Uudw=="], - - "split2": ["split2@3.2.2", "", { "dependencies": { "readable-stream": "^3.0.0" } }, "sha512-9NThjpgZnifTkJpzTZ7Eue85S49QwpNhZTq6GRJwObb6jnLFNGB7Qm73V5HewTROPyxD0C29xqmaI68bQtV+hg=="], - - "sprintf-js": ["sprintf-js@1.0.3", "", {}, "sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g=="], - - "ssf": ["ssf@0.11.2", "", { "dependencies": { "frac": "~1.1.2" } }, "sha512-+idbmIXoYET47hH+d7dfm2epdOMUDjqcB4648sTZ+t2JwoyBFL/insLfB/racrDmsKB3diwsDA696pZMieAC5g=="], - - "ssh-remote-port-forward": ["ssh-remote-port-forward@1.0.4", "", { "dependencies": { "@types/ssh2": "^0.5.48", "ssh2": "^1.4.0" } }, "sha512-x0LV1eVDwjf1gmG7TTnfqIzf+3VPRz7vrNIjX6oYLbeCrf/PeVY6hkT68Mg+q02qXxQhrLjB0jfgvhevoCRmLQ=="], - - "ssh2": ["ssh2@1.17.0", "", { "dependencies": { "asn1": "^0.2.6", "bcrypt-pbkdf": "^1.0.2" }, "optionalDependencies": { "cpu-features": "~0.0.10", "nan": "^2.23.0" } }, "sha512-wPldCk3asibAjQ/kziWQQt1Wh3PgDFpC0XpwclzKcdT1vql6KeYxf5LIt4nlFkUeR8WuphYMKqUA56X4rjbfgQ=="], - - "stackback": ["stackback@0.0.2", "", {}, "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw=="], - - "stackblur-canvas": ["stackblur-canvas@2.7.0", "", {}, "sha512-yf7OENo23AGJhBriGx0QivY5JP6Y1HbrrDI6WLt6C5auYZXlQrheoY8hD4ibekFKz1HOfE48Ww8kMWMnJD/zcQ=="], - - "standard-version": ["standard-version@9.5.0", "", { "dependencies": { "chalk": "^2.4.2", "conventional-changelog": "3.1.25", "conventional-changelog-config-spec": "2.1.0", "conventional-changelog-conventionalcommits": "4.6.3", "conventional-recommended-bump": "6.1.0", "detect-indent": "^6.0.0", "detect-newline": "^3.1.0", "dotgitignore": "^2.1.0", "figures": "^3.1.0", "find-up": "^5.0.0", "git-semver-tags": "^4.0.0", "semver": "^7.1.1", "stringify-package": "^1.0.1", "yargs": "^16.0.0" }, "bin": "bin/cli.js" }, "sha512-3zWJ/mmZQsOaO+fOlsa0+QK90pwhNd042qEcw6hKFNoLFs7peGyvPffpEBbK/DSGPbyOvli0mUIFv5A4qTjh2Q=="], - - "std-env": ["std-env@3.10.0", "", {}, "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg=="], - - "stdin-discarder": ["stdin-discarder@0.2.2", "", {}, "sha512-UhDfHmA92YAlNnCfhmq0VeNL5bDbiZGg7sZ2IvPsXubGkiNa9EC+tUTsjBRsYUAz87btI6/1wf4XoVvQ3uRnmQ=="], - - "stream-chain": ["stream-chain@2.2.5", "", {}, "sha512-1TJmBx6aSWqZ4tx7aTpBDXK0/e2hhcNSTV8+CbFJtDjbb+I1mZ8lHit0Grw9GRT+6JbIrrDd8esncgBi8aBXGA=="], - - "stream-events": ["stream-events@1.0.5", "", { "dependencies": { "stubs": "^3.0.0" } }, "sha512-E1GUzBSgvct8Jsb3v2X15pjzN1tYebtbLaMg+eBOUOAxgbLoSbT2NS91ckc5lJD1KfLjId+jXJRgo0qnV5Nerg=="], - - "stream-json": ["stream-json@1.9.1", "", { "dependencies": { "stream-chain": "^2.2.5" } }, "sha512-uWkjJ+2Nt/LO9Z/JyKZbMusL8Dkh97uUBTv3AJQ74y07lVahLY4eEFsPsE97pxYBwr8nnjMAIch5eqI0gPShyw=="], - - "stream-parser": ["stream-parser@0.3.1", "", { "dependencies": { "debug": "2" } }, "sha512-bJ/HgKq41nlKvlhccD5kaCr/P+Hu0wPNKPJOH7en+YrJu/9EgqUF+88w5Jb6KNcjOFMhfX4B2asfeAtIGuHObQ=="], - - "stream-shift": ["stream-shift@1.0.3", "", {}, "sha512-76ORR0DO1o1hlKwTbi/DM3EXWGf3ZJYO8cXX5RJwnul2DEg2oyoZyjLNoQM8WsvZiFKCRfC1O0J7iCvie3RZmQ=="], - - "streamx": ["streamx@2.23.0", "", { "dependencies": { "events-universal": "^1.0.0", "fast-fifo": "^1.3.2", "text-decoder": "^1.1.0" } }, "sha512-kn+e44esVfn2Fa/O0CPFcex27fjIL6MkVae0Mm6q+E6f0hWv578YCERbv+4m02cjxvDsPKLnmxral/rR6lBMAg=="], - - "strict-uri-encode": ["strict-uri-encode@2.0.0", "", {}, "sha512-QwiXZgpRcKkhTj2Scnn++4PKtWsH0kpzZ62L2R6c/LUVYv7hVnZqcg2+sMuT6R7Jusu1vviK/MFsu6kNJfWlEQ=="], - - "string-width": ["string-width@7.2.0", "", { "dependencies": { "emoji-regex": "^10.3.0", "get-east-asian-width": "^1.0.0", "strip-ansi": "^7.1.0" } }, "sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ=="], - - "string-width-cjs": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "string_decoder": ["string_decoder@1.3.0", "", { "dependencies": { "safe-buffer": "~5.2.0" } }, "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA=="], - - "stringify-package": ["stringify-package@1.0.1", "", {}, "sha512-sa4DUQsYciMP1xhKWGuFM04fB0LG/9DlluZoSVywUMRNvzid6XucHK0/90xGxRoHrAaROrcHK1aPKaijCtSrhg=="], - - "strip-ansi": ["strip-ansi@7.1.2", "", { "dependencies": { "ansi-regex": "^6.0.1" } }, "sha512-gmBGslpoQJtgnMAvOVqGZpEz9dyoKTCzy2nfz/n8aIFhN/jCE/rCmcxabB6jOOHV+0WNnylOxaxBQPSvcWklhA=="], - - "strip-ansi-cjs": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "strip-bom": ["strip-bom@3.0.0", "", {}, "sha512-vavAMRXOgBVNF6nyEEmL3DBK19iRpDcoIwW+swQ+CbGiu7lju6t+JklA1MHweoWtadgt4ISVUsXLyDq34ddcwA=="], - - "strip-indent": ["strip-indent@3.0.0", "", { "dependencies": { "min-indent": "^1.0.0" } }, "sha512-laJTa3Jb+VQpaC6DseHhF7dXVqHTfJPCRDaEbid/drOhgitgYku/letMUqOXFoWV0zIIUbjpdH2t+tYj4bQMRQ=="], - - "strip-json-comments": ["strip-json-comments@3.1.1", "", {}, "sha512-6fPc+R4ihwqP6N/aIv2f1gMH8lOVtWQHoqC4yK6oSDVVocumAsfCqjkXnqiYMhmMwS/mEHLp7Vehlt3ql6lEig=="], - - "strip-literal": ["strip-literal@3.1.0", "", { "dependencies": { "js-tokens": "^9.0.1" } }, "sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg=="], - - "strnum": ["strnum@1.1.2", "", {}, "sha512-vrN+B7DBIoTTZjnPNewwhx6cBA/H+IS7rfW68n7XxC1y7uoiGQBxaKzqucGUgavX15dJgiGztLJ8vxuEzwqBdA=="], - - "stubs": ["stubs@3.0.0", "", {}, "sha512-PdHt7hHUJKxvTCgbKX9C1V/ftOcjJQgz8BZwNfV5c4B6dcGqlpelTbJ999jBGZ2jYiPAwcX5dP6oBwVlBlUbxw=="], - - "supports-color": ["supports-color@7.2.0", "", { "dependencies": { "has-flag": "^4.0.0" } }, "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw=="], - - "supports-preserve-symlinks-flag": ["supports-preserve-symlinks-flag@1.0.0", "", {}, "sha512-ot0WnXS9fgdkgIcePe6RHNk1WA8+muPa6cSjeR3V8K27q9BB1rTE3R1p7Hv0z1ZyAc8s6Vvv8DIyWf681MAt0w=="], - - "svg-pathdata": ["svg-pathdata@6.0.3", "", {}, "sha512-qsjeeq5YjBZ5eMdFuUa4ZosMLxgr5RZ+F+Y1OrDhuOCEInRMA3x74XdBtggJcj9kOeInz0WE+LgCPDkZFlBYJw=="], - - "tar-fs": ["tar-fs@3.1.1", "", { "dependencies": { "pump": "^3.0.0", "tar-stream": "^3.1.5" }, "optionalDependencies": { "bare-fs": "^4.0.1", "bare-path": "^3.0.0" } }, "sha512-LZA0oaPOc2fVo82Txf3gw+AkEd38szODlptMYejQUhndHMLQ9M059uXR+AfS7DNo0NpINvSqDsvyaCrBVkptWg=="], - - "tar-stream": ["tar-stream@3.1.7", "", { "dependencies": { "b4a": "^1.6.4", "fast-fifo": "^1.2.0", "streamx": "^2.15.0" } }, "sha512-qJj60CXt7IU1Ffyc3NJMjh6EkuCFej46zUqJ4J7pqYlThyd9bO0XBTmcOIhSzZJVWfsLks0+nle/j538YAW9RQ=="], - - "teeny-request": ["teeny-request@9.0.0", "", { "dependencies": { "http-proxy-agent": "^5.0.0", "https-proxy-agent": "^5.0.0", "node-fetch": "^2.6.9", "stream-events": "^1.0.5", "uuid": "^9.0.0" } }, "sha512-resvxdc6Mgb7YEThw6G6bExlXKkv6+YbuzGg9xuXxSgxJF7Ozs+o8Y9+2R3sArdWdW8nOokoQb1yrpFB0pQK2g=="], - - "terser": ["terser@5.44.1", "", { "dependencies": { "@jridgewell/source-map": "^0.3.3", "acorn": "^8.15.0", "commander": "^2.20.0", "source-map-support": "~0.5.20" }, "bin": "bin/terser" }, "sha512-t/R3R/n0MSwnnazuPpPNVO60LX0SKL45pyl9YlvxIdkH0Of7D5qM2EVe+yASRIlY5pZ73nclYJfNANGWPwFDZw=="], - - "test-exclude": ["test-exclude@7.0.1", "", { "dependencies": { "@istanbuljs/schema": "^0.1.2", "glob": "^10.4.1", "minimatch": "^9.0.4" } }, "sha512-pFYqmTw68LXVjeWJMST4+borgQP2AyMNbg1BpZh9LbyhUeNkeaPF9gzfPGUAnSMV3qPYdWUwDIjjCLiSDOl7vg=="], - - "testcontainers": ["testcontainers@11.10.0", "", { "dependencies": { "@balena/dockerignore": "^1.0.2", "@types/dockerode": "^3.3.47", "archiver": "^7.0.1", "async-lock": "^1.4.1", "byline": "^5.0.0", "debug": "^4.4.3", "docker-compose": "^1.3.0", "dockerode": "^4.0.9", "get-port": "^7.1.0", "proper-lockfile": "^4.1.2", "properties-reader": "^2.3.0", "ssh-remote-port-forward": "^1.0.4", "tar-fs": "^3.1.1", "tmp": "^0.2.5", "undici": "^7.16.0" } }, "sha512-8hwK2EnrOZfrHPpDC7CPe03q7H8Vv8j3aXdcmFFyNV8dzpBzgZYmqyDtduJ8YQ5kbzj+A+jUXMQ6zI8B5U3z+g=="], - - "text-decoder": ["text-decoder@1.2.3", "", { "dependencies": { "b4a": "^1.6.4" } }, "sha512-3/o9z3X0X0fTupwsYvR03pJ/DjWuqqrfwBgTQzdWDiQSm9KitAyz/9WqsT2JQW7KV2m+bC2ol/zqpW37NHxLaA=="], - - "text-extensions": ["text-extensions@1.9.0", "", {}, "sha512-wiBrwC1EhBelW12Zy26JeOUkQ5mRu+5o8rpsJk5+2t+Y5vE7e842qtZDQ2g1NpX/29HdyFeJ4nSIhI47ENSxlQ=="], - - "text-segmentation": ["text-segmentation@1.0.3", "", { "dependencies": { "utrie": "^1.0.2" } }, "sha512-iOiPUo/BGnZ6+54OsWxZidGCsdU8YbE4PSpdPinp7DeMtUJNJBoJ/ouUSTJjHkh1KntHaltHl/gDs2FC4i5+Nw=="], - - "through": ["through@2.3.8", "", {}, "sha512-w89qg7PI8wAdvX60bMDP+bFoD5Dvhm9oLheFp5O4a2QF0cSBGsBX4qZmadPMvVqlLJBBci+WqGGOAPvcDeNSVg=="], - - "through2": ["through2@4.0.2", "", { "dependencies": { "readable-stream": "3" } }, "sha512-iOqSav00cVxEEICeD7TjLB1sueEL+81Wpzp2bY17uZjZN0pWZPuo4suZ/61VujxmqSGFfgOcNuTZ85QJwNZQpw=="], - - "tinybench": ["tinybench@2.9.0", "", {}, "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg=="], - - "tinyexec": ["tinyexec@0.3.2", "", {}, "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA=="], - - "tinyglobby": ["tinyglobby@0.2.15", "", { "dependencies": { "fdir": "^6.5.0", "picomatch": "^4.0.3" } }, "sha512-j2Zq4NyQYG5XMST4cbs02Ak8iJUdxRM0XI5QyxXuZOzKOINmWurp3smXu3y5wDcJrptwpSjgXHzIQxR0omXljQ=="], - - "tinypool": ["tinypool@1.1.1", "", {}, "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg=="], - - "tinyrainbow": ["tinyrainbow@2.0.0", "", {}, "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw=="], - - "tinyspy": ["tinyspy@4.0.4", "", {}, "sha512-azl+t0z7pw/z958Gy9svOTuzqIk6xq+NSheJzn5MMWtWTFywIacg2wUlzKFGtt3cthx0r2SxMK0yzJOR0IES7Q=="], - - "tmp": ["tmp@0.2.5", "", {}, "sha512-voyz6MApa1rQGUxT3E+BK7/ROe8itEx7vD8/HEvt4xwXucvQ5G5oeEiHkmHZJuBO21RpOf+YYm9MOivj709jow=="], - - "tr46": ["tr46@0.0.3", "", {}, "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw=="], - - "trim-newlines": ["trim-newlines@3.0.1", "", {}, "sha512-c1PTsA3tYrIsLGkJkzHF+w9F2EyxfXGo4UyJc4pFL++FMjnq0HJS69T3M7d//gKrFKwy429bouPescbjecU+Zw=="], - - "ts-api-utils": ["ts-api-utils@2.1.0", "", { "peerDependencies": { "typescript": ">=4.8.4" } }, "sha512-CUgTZL1irw8u29bzrOD/nH85jqyc74D6SshFgujOIA7osm2Rz7dYH77agkx7H4FBNxDq7Cjf+IjaX/8zwFW+ZQ=="], - - "tslib": ["tslib@2.8.1", "", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="], - - "tsx": ["tsx@4.21.0", "", { "dependencies": { "esbuild": "~0.27.0", "get-tsconfig": "^4.7.5" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "bin": "dist/cli.mjs" }, "sha512-5C1sg4USs1lfG0GFb2RLXsdpXqBSEhAaA/0kPL01wxzpMqLILNxIxIOKiILz+cdg/pLnOUxFYOR5yhHU666wbw=="], - - "tweetnacl": ["tweetnacl@0.14.5", "", {}, "sha512-KXXFFdAbFXY4geFIwoyNK+f5Z1b7swfXABfL7HXCmoIWMKU3dmS26672A4EeQtDzLKy7SXmfBu51JolvEKwtGA=="], - - "type-check": ["type-check@0.4.0", "", { "dependencies": { "prelude-ls": "^1.2.1" } }, "sha512-XleUoc9uwGXqjWwXaUTZAmzMcFZ5858QA2vvx1Ur5xIcixXIP+8LnFDgRplU30us6teqdlskFfu+ae4K79Ooew=="], - - "type-fest": ["type-fest@4.41.0", "", {}, "sha512-TeTSQ6H5YHvpqVwBRcnLDCBnDOHWYu7IvGbHT6N8AOymcr9PJGjc1GTtiWZTYg0NCgYwvnYWEkVChQAr9bjfwA=="], - - "typedarray": ["typedarray@0.0.6", "", {}, "sha512-/aCDEGatGvZ2BIk+HmLf4ifCJFwvKFNb9/JeZPMulfgFracn9QFcAf5GO8B/mweUjSoblS5In0cWhqpfs/5PQA=="], - - "typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="], - - "uglify-js": ["uglify-js@3.19.3", "", { "bin": { "uglifyjs": "bin/uglifyjs" } }, "sha512-v3Xu+yuwBXisp6QYTcH4UbH+xYJXqnq2m/LtQVWKWzYc1iehYnLixoQDN9FH6/j9/oybfd6W9Ghwkl8+UMKTKQ=="], - - "underscore": ["underscore@1.13.7", "", {}, "sha512-GMXzWtsc57XAtguZgaQViUOzs0KTkk8ojr3/xAxXLITqf/3EMwxC0inyETfDFjH/Krbhuep0HNbbjI9i/q3F3g=="], - - "undici": ["undici@7.16.0", "", {}, "sha512-QEg3HPMll0o3t2ourKwOeUAZ159Kn9mx5pnzHRQO8+Wixmh88YdZRiIwat0iNzNNXn0yoEtXJqFpyW7eM8BV7g=="], - - "undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="], - - "uri-js": ["uri-js@4.4.1", "", { "dependencies": { "punycode": "^2.1.0" } }, "sha512-7rKUyy33Q1yc98pQ1DAmLtwX109F7TIfWlW1Ydo8Wl1ii1SeHieeh0HHfPeL2fMXK6z0s8ecKs9frCuLJvndBg=="], - - "util": ["util@0.12.5", "", { "dependencies": { "inherits": "^2.0.3", "is-arguments": "^1.0.4", "is-generator-function": "^1.0.7", "is-typed-array": "^1.1.3", "which-typed-array": "^1.1.2" } }, "sha512-kZf/K6hEIrWHI6XqOFUiiMa+79wE/D8Q+NCNAWclkyg3b4d2k7s0QGepNjiABc+aR3N1PAyHL7p6UcLY6LmrnA=="], - - "util-deprecate": ["util-deprecate@1.0.2", "", {}, "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw=="], - - "utrie": ["utrie@1.0.2", "", { "dependencies": { "base64-arraybuffer": "^1.0.2" } }, "sha512-1MLa5ouZiOmQzUbjbu9VmjLzn1QLXBhwpUa7kdLUQK+KQ5KA9I1vk5U4YHe/X2Ch7PYnJfWuWT+VbuxbGwljhw=="], - - "uuid": ["uuid@9.0.1", "", { "bin": "dist/bin/uuid" }, "sha512-b+1eJOlsR9K8HJpow9Ok3fiWOWSIcIzXodvv0rQjVoOVNpWMpxf1wZNpt4y9h10odCNrqnYp1OBzRktckBe3sA=="], - - "validate-npm-package-license": ["validate-npm-package-license@3.0.4", "", { "dependencies": { "spdx-correct": "^3.0.0", "spdx-expression-parse": "^3.0.0" } }, "sha512-DpKm2Ui/xN7/HQKCtpZxoRWBhZ9Z0kqtygG8XCgNQ8ZlDnxuQmWhj566j8fN4Cu3/JmbhsDo7fcAJq4s9h27Ew=="], - - "vite": ["vite@7.3.0", "", { "dependencies": { "esbuild": "^0.27.0", "fdir": "^6.5.0", "picomatch": "^4.0.3", "postcss": "^8.5.6", "rollup": "^4.43.0", "tinyglobby": "^0.2.15" }, "optionalDependencies": { "fsevents": "~2.3.3" }, "peerDependencies": { "@types/node": "^20.19.0 || >=22.12.0", "jiti": ">=1.21.0", "less": "^4.0.0", "lightningcss": "^1.21.0", "sass": "^1.70.0", "sass-embedded": "^1.70.0", "stylus": ">=0.54.8", "sugarss": "^5.0.0", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["jiti", "less", "lightningcss", "sass", "sass-embedded", "stylus", "sugarss"], "bin": "bin/vite.js" }, "sha512-dZwN5L1VlUBewiP6H9s2+B3e3Jg96D0vzN+Ry73sOefebhYr9f94wwkMNN/9ouoU8pV1BqA1d1zGk8928cx0rg=="], - - "vite-node": ["vite-node@3.2.4", "", { "dependencies": { "cac": "^6.7.14", "debug": "^4.4.1", "es-module-lexer": "^1.7.0", "pathe": "^2.0.3", "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0" }, "bin": "vite-node.mjs" }, "sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg=="], - - "vitest": ["vitest@3.2.4", "", { "dependencies": { "@types/chai": "^5.2.2", "@vitest/expect": "3.2.4", "@vitest/mocker": "3.2.4", "@vitest/pretty-format": "^3.2.4", "@vitest/runner": "3.2.4", "@vitest/snapshot": "3.2.4", "@vitest/spy": "3.2.4", "@vitest/utils": "3.2.4", "chai": "^5.2.0", "debug": "^4.4.1", "expect-type": "^1.2.1", "magic-string": "^0.30.17", "pathe": "^2.0.3", "picomatch": "^4.0.2", "std-env": "^3.9.0", "tinybench": "^2.9.0", "tinyexec": "^0.3.2", "tinyglobby": "^0.2.14", "tinypool": "^1.1.1", "tinyrainbow": "^2.0.0", "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0", "vite-node": "3.2.4", "why-is-node-running": "^2.3.0" }, "peerDependencies": { "@edge-runtime/vm": "*", "@types/debug": "^4.1.12", "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", "@vitest/browser": "3.2.4", "@vitest/ui": "3.2.4", "happy-dom": "*", "jsdom": "*" }, "optionalPeers": ["@edge-runtime/vm", "@types/debug", "@vitest/browser", "@vitest/ui", "happy-dom", "jsdom"], "bin": "vitest.mjs" }, "sha512-LUCP5ev3GURDysTWiP47wRRUpLKMOfPh+yKTx3kVIEiu5KOMeqzpnYNsKyOoVrULivR8tLcks4+lga33Whn90A=="], - - "web-encoding": ["web-encoding@1.1.5", "", { "dependencies": { "util": "^0.12.3" }, "optionalDependencies": { "@zxing/text-encoding": "0.9.0" } }, "sha512-HYLeVCdJ0+lBYV2FvNZmv3HJ2Nt0QYXqZojk3d9FJOLkwnuhzM9tmamh8d7HPM8QqjKH8DeHkFTx+CFlWpZZDA=="], - - "webidl-conversions": ["webidl-conversions@3.0.1", "", {}, "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ=="], - - "whatwg-url": ["whatwg-url@5.0.0", "", { "dependencies": { "tr46": "~0.0.3", "webidl-conversions": "^3.0.0" } }, "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw=="], - - "which": ["which@2.0.2", "", { "dependencies": { "isexe": "^2.0.0" }, "bin": { "node-which": "bin/node-which" } }, "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA=="], - - "which-typed-array": ["which-typed-array@1.1.19", "", { "dependencies": { "available-typed-arrays": "^1.0.7", "call-bind": "^1.0.8", "call-bound": "^1.0.4", "for-each": "^0.3.5", "get-proto": "^1.0.1", "gopd": "^1.2.0", "has-tostringtag": "^1.0.2" } }, "sha512-rEvr90Bck4WZt9HHFC4DJMsjvu7x+r6bImz0/BrbWb7A2djJ8hnZMrWnHo9F8ssv0OMErasDhftrfROTyqSDrw=="], - - "why-is-node-running": ["why-is-node-running@2.3.0", "", { "dependencies": { "siginfo": "^2.0.0", "stackback": "0.0.2" }, "bin": "cli.js" }, "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w=="], - - "widest-line": ["widest-line@5.0.0", "", { "dependencies": { "string-width": "^7.0.0" } }, "sha512-c9bZp7b5YtRj2wOe6dlj32MK+Bx/M/d+9VB2SHM1OtsUHR0aV0tdP6DWh/iMt0kWi1t5g1Iudu6hQRNd1A4PVA=="], - - "wmf": ["wmf@1.0.2", "", {}, "sha512-/p9K7bEh0Dj6WbXg4JG0xvLQmIadrner1bi45VMJTfnbVHsc7yIajZyoSoK60/dtVBs12Fm6WkUI5/3WAVsNMw=="], - - "word": ["word@0.3.0", "", {}, "sha512-OELeY0Q61OXpdUfTp+oweA/vtLVg5VDOXh+3he3PNzLGG/y0oylSOC1xRVj0+l4vQ3tj/bB1HVHv1ocXkQceFA=="], - - "word-wrap": ["word-wrap@1.2.5", "", {}, "sha512-BN22B5eaMMI9UMtjrGd5g5eCYPpCPDUy0FJXbYsaT5zYxjFOckS53SQDE3pWkVoWpHXVb3BrYcEN4Twa55B5cA=="], - - "wordwrap": ["wordwrap@1.0.0", "", {}, "sha512-gvVzJFlPycKc5dZN4yPkP8w7Dc37BtP1yczEneOb4uq34pXZcvrtRTmWV8W+Ume+XCxKgbjM+nevkyFPMybd4Q=="], - - "wrap-ansi": ["wrap-ansi@9.0.2", "", { "dependencies": { "ansi-styles": "^6.2.1", "string-width": "^7.0.0", "strip-ansi": "^7.1.0" } }, "sha512-42AtmgqjV+X1VpdOfyTGOYRi0/zsoLqtXQckTmqTeybT+BDIbM/Guxo7x3pE2vtpr1ok6xRqM9OpBe+Jyoqyww=="], - - "wrap-ansi-cjs": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], - - "wrappy": ["wrappy@1.0.2", "", {}, "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ=="], - - "ws": ["ws@8.18.3", "", { "peerDependencies": { "bufferutil": "^4.0.1", "utf-8-validate": ">=5.0.2" }, "optionalPeers": ["bufferutil", "utf-8-validate"] }, "sha512-PEIGCY5tSlUt50cqyMXfCzX+oOPqN0vuGqWzbcJ2xvnkzkq46oOpz7dQaTDBdfICb4N14+GARUDw2XV2N4tvzg=="], - - "wsl-utils": ["wsl-utils@0.1.0", "", { "dependencies": { "is-wsl": "^3.1.0" } }, "sha512-h3Fbisa2nKGPxCpm89Hk33lBLsnaGBvctQopaBSOW/uIs6FTe1ATyAnKFJrzVs9vpGdsTe73WF3V4lIsk4Gacw=="], - - "xlsx": ["xlsx@0.18.5", "", { "dependencies": { "adler-32": "~1.3.0", "cfb": "~1.2.1", "codepage": "~1.15.0", "crc-32": "~1.2.1", "ssf": "~0.11.2", "wmf": "~1.0.1", "word": "~0.3.0" }, "bin": "bin/xlsx.njs" }, "sha512-dmg3LCjBPHZnQp5/F/+nnTa+miPJxUXB6vtk42YjBBKayDNagxGEeIdWApkYPOf3Z3pm3k62Knjzp7lMeTEtFQ=="], - - "xml2js": ["xml2js@0.6.2", "", { "dependencies": { "sax": ">=0.6.0", "xmlbuilder": "~11.0.0" } }, "sha512-T4rieHaC1EXcES0Kxxj4JWgaUQHDk+qwHcYOCFHfiwKz7tOVPLq7Hjq9dM1WCMhylqMEfP7hMcOIChvotiZegA=="], - - "xmlbuilder": ["xmlbuilder@10.1.1", "", {}, "sha512-OyzrcFLL/nb6fMGHbiRDuPup9ljBycsdCypwuyg5AAHvyWzGfChJpCXMG88AGTIMFhGZ9RccFN1e6lhg3hkwKg=="], - - "xtend": ["xtend@4.0.2", "", {}, "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ=="], - - "y18n": ["y18n@5.0.8", "", {}, "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA=="], - - "yallist": ["yallist@4.0.0", "", {}, "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A=="], - - "yaml": ["yaml@2.8.2", "", { "bin": "bin.mjs" }, "sha512-mplynKqc1C2hTVYxd0PU2xQAc22TI1vShAYGksCCfxbn/dFwnHTNi1bvYsBTkhdUNtGIf5xNOg938rrSSYvS9A=="], - - "yargs": ["yargs@16.2.0", "", { "dependencies": { "cliui": "^7.0.2", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "require-directory": "^2.1.1", "string-width": "^4.2.0", "y18n": "^5.0.5", "yargs-parser": "^20.2.2" } }, "sha512-D1mvvtDG0L5ft/jGWkLpG1+m0eQxOfaBvTNELraWj22wSVUMWxZUvYgJYcKh6jGGIkJFhH4IZPQhR4TKpc8mBw=="], - - "yargs-parser": ["yargs-parser@20.2.9", "", {}, "sha512-y11nGElTIV+CT3Zv9t7VKl+Q3hTQoT9a1Qzezhhl6Rp21gJ/IVTW7Z3y9EWXhuUBC2Shnf+DX0antecpAwSP8w=="], - - "yocto-queue": ["yocto-queue@0.1.0", "", {}, "sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q=="], - - "yoctocolors-cjs": ["yoctocolors-cjs@2.1.3", "", {}, "sha512-U/PBtDf35ff0D8X8D0jfdzHYEPFxAI7jJlxZXwCSez5M3190m+QobIfh+sWDWSHMCWWJN2AWamkegn6vr6YBTw=="], - - "zip-stream": ["zip-stream@6.0.1", "", { "dependencies": { "archiver-utils": "^5.0.0", "compress-commons": "^6.0.2", "readable-stream": "^4.0.0" } }, "sha512-zK7YHHz4ZXpW89AHXUPbQVGKI7uvkd3hzusTdotCg1UxyaVtg0zFJSTfW/Dq5f7OBBVnq6cZIaC8Ti4hb6dtCA=="], - - "@aws-crypto/sha1-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="], - - "@aws-crypto/sha256-browser/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="], - - "@aws-crypto/util/@smithy/util-utf8": ["@smithy/util-utf8@2.3.0", "", { "dependencies": { "@smithy/util-buffer-from": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A=="], - - "@aws-sdk/xml-builder/fast-xml-parser": ["fast-xml-parser@5.2.5", "", { "dependencies": { "strnum": "^2.1.0" }, "bin": { "fxparser": "src/cli/cli.js" } }, "sha512-pfX9uG9Ki0yekDHx2SiuRIyFdyAr1kMIMitPvb0YBo8SUfKvia7w7FIyd/l6av85pFYRhZscS75MwMnbvY+hcQ=="], - - "@azure/core-xml/fast-xml-parser": ["fast-xml-parser@5.2.5", "", { "dependencies": { "strnum": "^2.1.0" }, "bin": { "fxparser": "src/cli/cli.js" } }, "sha512-pfX9uG9Ki0yekDHx2SiuRIyFdyAr1kMIMitPvb0YBo8SUfKvia7w7FIyd/l6av85pFYRhZscS75MwMnbvY+hcQ=="], - - "@azure/msal-node/uuid": ["uuid@8.3.2", "", { "bin": "dist/bin/uuid" }, "sha512-+NYs2QeMWy+GWFOEm9xnn6HCDp0l7QBD7ml8zLUmJ+93Q5NF0NocErnwkTkXVFNiX3/fpC6afS8Dhb/gz7R7eg=="], - - "@babel/code-frame/js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="], - - "@eslint-community/eslint-utils/eslint-visitor-keys": ["eslint-visitor-keys@3.4.3", "", {}, "sha512-wpc+LXeiyiisxPlEkUzU6svyS1frIO3Mgxj1fdy7Pm8Ygzguax2N3Fa/D/ag1WqbOprdI+uY6wMUl8/a2G+iag=="], - - "@eslint/eslintrc/ignore": ["ignore@5.3.2", "", {}, "sha512-hsBTNUqQTDwkWtcdYI2i06Y/nUBEsNEDJKjWdigLvegy8kDuJAS8uRlpkkcQpyEXL0Z/pjDy5HBmMjRCJ2gq+g=="], - - "@google-cloud/storage/mime": ["mime@3.0.0", "", { "bin": "cli.js" }, "sha512-jSCU7/VB1loIWBZe14aEYHU/+1UMEHoaO7qxCOVJOw9GgH72VAWppxNcjU+x9a2k3GSIBXNKxXQFqRvvZ7vr3A=="], - - "@google-cloud/storage/uuid": ["uuid@8.3.2", "", { "bin": "dist/bin/uuid" }, "sha512-+NYs2QeMWy+GWFOEm9xnn6HCDp0l7QBD7ml8zLUmJ+93Q5NF0NocErnwkTkXVFNiX3/fpC6afS8Dhb/gz7R7eg=="], - - "@grpc/grpc-js/@grpc/proto-loader": ["@grpc/proto-loader@0.8.0", "", { "dependencies": { "lodash.camelcase": "^4.3.0", "long": "^5.0.0", "protobufjs": "^7.5.3", "yargs": "^17.7.2" }, "bin": { "proto-loader-gen-types": "build/bin/proto-loader-gen-types.js" } }, "sha512-rc1hOQtjIWGxcxpb9aHAfLpIctjEnsDehj0DAiVfBlmT84uvR0uUtN2hEi/ecvWVjXUGf5qPF4qEgiLOx1YIMQ=="], - - "@grpc/proto-loader/yargs": ["yargs@17.7.2", "", { "dependencies": { "cliui": "^8.0.1", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "require-directory": "^2.1.1", "string-width": "^4.2.3", "y18n": "^5.0.5", "yargs-parser": "^21.1.1" } }, "sha512-7dSzzRQ++CKnNI/krKnYRV7JKKPUXMEh61soaHKg9mrWEhzFWhFnxPxGl+69cD1Ou63C13NUPCnmIcrvqCuM6w=="], - - "@inquirer/core/signal-exit": ["signal-exit@4.1.0", "", {}, "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw=="], - - "@inquirer/core/wrap-ansi": ["wrap-ansi@6.2.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-r6lPcBGxZXlIcymEu7InxDMhdW0KDxpLgoFLcguasxCaJ/SOIZwINatK9KY/tf+ZrlywOKU0UDj3ATXUBfxJXA=="], - - "@inquirer/external-editor/iconv-lite": ["iconv-lite@0.7.1", "", { "dependencies": { "safer-buffer": ">= 2.1.2 < 3.0.0" } }, "sha512-2Tth85cXwGFHfvRgZWszZSvdo+0Xsqmw8k8ZwxScfcBneNUraK+dxRxRm24nszx80Y0TVio8kKLt5sLE7ZCLlw=="], - - "@isaacs/cliui/string-width": ["string-width@5.1.2", "", { "dependencies": { "eastasianwidth": "^0.2.0", "emoji-regex": "^9.2.2", "strip-ansi": "^7.0.1" } }, "sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA=="], - - "@isaacs/cliui/wrap-ansi": ["wrap-ansi@8.1.0", "", { "dependencies": { "ansi-styles": "^6.1.0", "string-width": "^5.0.1", "strip-ansi": "^7.0.1" } }, "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ=="], - - "@types/ssh2/@types/node": ["@types/node@18.19.130", "", { "dependencies": { "undici-types": "~5.26.4" } }, "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg=="], - - "@typescript-eslint/typescript-estree/minimatch": ["minimatch@9.0.5", "", { "dependencies": { "brace-expansion": "^2.0.1" } }, "sha512-G6T0ZX48xgozx7587koeX9Ys2NYy6Gmv//P89sEte9V9whIapMNF4idKxnW2QtCcLiTWlb/wfCabAtAFWhhBow=="], - - "@typespec/ts-http-runtime/http-proxy-agent": ["http-proxy-agent@7.0.2", "", { "dependencies": { "agent-base": "^7.1.0", "debug": "^4.3.4" } }, "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig=="], - - "@vitest/mocker/estree-walker": ["estree-walker@3.0.3", "", { "dependencies": { "@types/estree": "^1.0.0" } }, "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g=="], - - "ansi-align/string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "archiver/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], - - "archiver-utils/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], - - "ast-v8-to-istanbul/estree-walker": ["estree-walker@3.0.3", "", { "dependencies": { "@types/estree": "^1.0.0" } }, "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g=="], - - "async-retry/retry": ["retry@0.13.1", "", {}, "sha512-XQBQ3I8W1Cge0Seh+6gjj03LbmRFWuoszgK9ooCpwYIrhhoO80pfq4cUkU5DkknwfOfFteRwlZ56PYOGYyFWdg=="], - - "bl/buffer": ["buffer@5.7.1", "", { "dependencies": { "base64-js": "^1.3.1", "ieee754": "^1.1.13" } }, "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ=="], - - "camelcase-keys/camelcase": ["camelcase@5.3.1", "", {}, "sha512-L28STB170nwWS63UjtlEOE3dldQApaJXZkOI1uMFfzf3rRuPegHaHesyee+YxQ+W6SvRDQV6UrdOdRiR153wJg=="], - - "cli-table3/string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "cliui/string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "cliui/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "cliui/wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], - - "compress-commons/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], - - "conventional-changelog-writer/semver": ["semver@6.3.1", "", { "bin": "bin/semver.js" }, "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA=="], - - "crc32-stream/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], - - "decamelize-keys/map-obj": ["map-obj@1.0.1", "", {}, "sha512-7N/q3lyZ+LVCp7PzuxrJr4KMbBE2hW7BT7YNia330OFxIf4d3r5zVpicP2650l7CPN6RM9zOJRl3NGpqSiw3Eg=="], - - "dockerode/tar-fs": ["tar-fs@2.1.4", "", { "dependencies": { "chownr": "^1.1.1", "mkdirp-classic": "^0.5.2", "pump": "^3.0.0", "tar-stream": "^2.1.4" } }, "sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ=="], - - "dockerode/uuid": ["uuid@10.0.0", "", { "bin": "dist/bin/uuid" }, "sha512-8XkAphELsDnEGrDxUOHB3RGvXz6TeuYSGEZBOjtTtPm2lwhGBjLgOzLHB63IUWfBpNucQjND6d3AOudO+H3RWQ=="], - - "dotgitignore/find-up": ["find-up@3.0.0", "", { "dependencies": { "locate-path": "^3.0.0" } }, "sha512-1yD6RmLI1XBfxugvORwlck6f75tYL+iR0jqwsOrOxMZyGYqUuDhJ0l4AXdO1iX/FTs9cBAMEk1gWSEx1kSbylg=="], - - "eslint/chalk": ["chalk@4.1.2", "", { "dependencies": { "ansi-styles": "^4.1.0", "supports-color": "^7.1.0" } }, "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA=="], - - "eslint/ignore": ["ignore@5.3.2", "", {}, "sha512-hsBTNUqQTDwkWtcdYI2i06Y/nUBEsNEDJKjWdigLvegy8kDuJAS8uRlpkkcQpyEXL0Z/pjDy5HBmMjRCJ2gq+g=="], - - "figures/escape-string-regexp": ["escape-string-regexp@1.0.5", "", {}, "sha512-vbRorB5FUQWvla16U8R/qgaFIya2qGzwDrNmCZuYKrbdSUMG6I1ZCGQRefkRVhuOkIGVne7BQ35DSfo1qvJqFg=="], - - "foreground-child/signal-exit": ["signal-exit@4.1.0", "", {}, "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw=="], - - "get-pkg-repo/through2": ["through2@2.0.5", "", { "dependencies": { "readable-stream": "~2.3.6", "xtend": "~4.0.1" } }, "sha512-/mrRod8xqpA+IHSLyGCQ2s8SPHiCDEeQJSep1jqLYeEUClOFG2Qsh+4FU6G9VeqpZnGW/Su8LQGc4YKni5rYSQ=="], - - "git-semver-tags/semver": ["semver@6.3.1", "", { "bin": "bin/semver.js" }, "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA=="], - - "glob/minimatch": ["minimatch@9.0.5", "", { "dependencies": { "brace-expansion": "^2.0.1" } }, "sha512-G6T0ZX48xgozx7587koeX9Ys2NYy6Gmv//P89sEte9V9whIapMNF4idKxnW2QtCcLiTWlb/wfCabAtAFWhhBow=="], - - "hosted-git-info/lru-cache": ["lru-cache@6.0.0", "", { "dependencies": { "yallist": "^4.0.0" } }, "sha512-Jo6dJ04CmSjuznwJSS3pUeWmd/H0ffTlkXXgwZi+eq1UCmqQwCh+eLsYOYCwY991i2Fah4h1BEMCx4qThGbsiA=="], - - "http-proxy-agent/agent-base": ["agent-base@6.0.2", "", { "dependencies": { "debug": "4" } }, "sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ=="], - - "jszip/pako": ["pako@1.0.11", "", {}, "sha512-4hLB8Py4zZce5s4yd9XzopqwVv/yGNhV1Bl8NTmCq1763HeK2+EwVTv+leGeL13Dnh2wfbqowVPXCIO0z4taYw=="], - - "jszip/readable-stream": ["readable-stream@2.3.8", "", { "dependencies": { "core-util-is": "~1.0.0", "inherits": "~2.0.3", "isarray": "~1.0.0", "process-nextick-args": "~2.0.0", "safe-buffer": "~5.1.1", "string_decoder": "~1.1.1", "util-deprecate": "~1.0.1" } }, "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA=="], - - "lazystream/readable-stream": ["readable-stream@2.3.8", "", { "dependencies": { "core-util-is": "~1.0.0", "inherits": "~2.0.3", "isarray": "~1.0.0", "process-nextick-args": "~2.0.0", "safe-buffer": "~5.1.1", "string_decoder": "~1.1.1", "util-deprecate": "~1.0.1" } }, "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA=="], - - "load-json-file/pify": ["pify@3.0.0", "", {}, "sha512-C3FsVNH1udSEX48gGX1xfvwTWfsYWj5U+8/uK15BGzIGrKoUpghX8hWZwa/OFnakBiiVNmBvemTJR5mcy7iPcg=="], - - "log-symbols/is-unicode-supported": ["is-unicode-supported@1.3.0", "", {}, "sha512-43r2mRvz+8JRIKnWJ+3j8JtjRKZ6GmjzfaE/qiBJnikNnYv/6bagRJ1kUhNk8R5EX/GkobD+r+sfxCPJsiKBLQ=="], - - "mammoth/argparse": ["argparse@1.0.10", "", { "dependencies": { "sprintf-js": "~1.0.2" } }, "sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg=="], - - "meow/read-pkg-up": ["read-pkg-up@7.0.1", "", { "dependencies": { "find-up": "^4.1.0", "read-pkg": "^5.2.0", "type-fest": "^0.8.1" } }, "sha512-zK0TB7Xd6JpCLmlLmufqykGE+/TlOePD6qKClNW7hHDKFh/J7/7gCWGR7joEQEW1bKq3a3yUZSObOoWLFQ4ohg=="], - - "meow/type-fest": ["type-fest@0.18.1", "", {}, "sha512-OIAYXk8+ISY+qTOwkHtKqzAuxchoMiD9Udx+FSGQDuiRR+PJKJHc2NJAXlbhkGwTt/4/nKZxELY1w3ReWOL8mw=="], - - "minimist-options/arrify": ["arrify@1.0.1", "", {}, "sha512-3CYzex9M9FGQjCGMGyi6/31c8GJbgb0qGyrx5HWxPd0aCwh4cB2YjMb2Xf9UuoogrMrlO9cTqnB5rI5GHZTcUA=="], - - "needle/debug": ["debug@3.2.7", "", { "dependencies": { "ms": "^2.1.1" } }, "sha512-CFjzYYAi4ThfiQvizrFQevTTXHtnCqWfe7x1AhgEscTz6ZbLbfoLRLPugTQyBth6f8ZERVUSyWHFD/7Wu4t1XQ=="], - - "path-type/pify": ["pify@3.0.0", "", {}, "sha512-C3FsVNH1udSEX48gGX1xfvwTWfsYWj5U+8/uK15BGzIGrKoUpghX8hWZwa/OFnakBiiVNmBvemTJR5mcy7iPcg=="], - - "read-pkg/normalize-package-data": ["normalize-package-data@2.5.0", "", { "dependencies": { "hosted-git-info": "^2.1.4", "resolve": "^1.10.0", "semver": "2 || 3 || 4 || 5", "validate-npm-package-license": "^3.0.1" } }, "sha512-/5CMN3T0R4XTj4DcGaexo+roZSdSFW/0AOOTROrjxzCG1wrWXEsGbRKevjlIL+ZDE4sZlJr5ED4YW0yqmkK+eA=="], - - "read-pkg-up/find-up": ["find-up@2.1.0", "", { "dependencies": { "locate-path": "^2.0.0" } }, "sha512-NWzkk0jSJtTt08+FBFMvXoeZnOJD+jTtsRmBYbAIzJdX6l7dLgR7CTubCM5/eDdPUBvLCeVasP1brfVR/9/EZQ=="], - - "readdir-glob/minimatch": ["minimatch@5.1.6", "", { "dependencies": { "brace-expansion": "^2.0.1" } }, "sha512-lKwV/1brpG6mBUFHtb7NUmtABCb2WZZmm2wNiOA5hAb8VdCS4B3dtMWyvcoViccwAW/COERjXLt0zP1zXUN26g=="], - - "restore-cursor/signal-exit": ["signal-exit@4.1.0", "", {}, "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw=="], - - "ssh-remote-port-forward/@types/ssh2": ["@types/ssh2@0.5.52", "", { "dependencies": { "@types/node": "*", "@types/ssh2-streams": "*" } }, "sha512-lbLLlXxdCZOSJMCInKH2+9V/77ET2J6NPQHpFI0kda61Dd1KglJs+fPQBchizmzYSOJBgdTajhPqBO1xxLywvg=="], - - "standard-version/chalk": ["chalk@2.4.2", "", { "dependencies": { "ansi-styles": "^3.2.1", "escape-string-regexp": "^1.0.5", "supports-color": "^5.3.0" } }, "sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ=="], - - "stream-parser/debug": ["debug@2.6.9", "", { "dependencies": { "ms": "2.0.0" } }, "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA=="], - - "string-width-cjs/emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - - "string-width-cjs/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "strip-ansi-cjs/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "teeny-request/https-proxy-agent": ["https-proxy-agent@5.0.1", "", { "dependencies": { "agent-base": "6", "debug": "4" } }, "sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA=="], - - "terser/commander": ["commander@2.20.3", "", {}, "sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ=="], - - "test-exclude/minimatch": ["minimatch@9.0.5", "", { "dependencies": { "brace-expansion": "^2.0.1" } }, "sha512-G6T0ZX48xgozx7587koeX9Ys2NYy6Gmv//P89sEte9V9whIapMNF4idKxnW2QtCcLiTWlb/wfCabAtAFWhhBow=="], - - "wrap-ansi/ansi-styles": ["ansi-styles@6.2.3", "", {}, "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg=="], - - "wrap-ansi-cjs/ansi-styles": ["ansi-styles@4.3.0", "", { "dependencies": { "color-convert": "^2.0.1" } }, "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg=="], - - "wrap-ansi-cjs/string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "wrap-ansi-cjs/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "xml2js/xmlbuilder": ["xmlbuilder@11.0.1", "", {}, "sha512-fDlsI/kFEx7gLvbecc0/ohLG50fugQp8ryHzMTuW9vSa1GJ0XYWKnhsUx7oie3G98+r56aTQIUB4kht42R3JvA=="], - - "yargs/string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "zip-stream/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], - - "@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="], - - "@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="], - - "@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from": ["@smithy/util-buffer-from@2.2.0", "", { "dependencies": { "@smithy/is-array-buffer": "^2.2.0", "tslib": "^2.6.2" } }, "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA=="], - - "@aws-sdk/xml-builder/fast-xml-parser/strnum": ["strnum@2.1.2", "", {}, "sha512-l63NF9y/cLROq/yqKXSLtcMeeyOfnSQlfMSlzFt/K73oIaD8DGaQWd7Z34X9GPiKqP5rbSh84Hl4bOlLcjiSrQ=="], - - "@azure/core-xml/fast-xml-parser/strnum": ["strnum@2.1.2", "", {}, "sha512-l63NF9y/cLROq/yqKXSLtcMeeyOfnSQlfMSlzFt/K73oIaD8DGaQWd7Z34X9GPiKqP5rbSh84Hl4bOlLcjiSrQ=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs": ["yargs@17.7.2", "", { "dependencies": { "cliui": "^8.0.1", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "require-directory": "^2.1.1", "string-width": "^4.2.3", "y18n": "^5.0.5", "yargs-parser": "^21.1.1" } }, "sha512-7dSzzRQ++CKnNI/krKnYRV7JKKPUXMEh61soaHKg9mrWEhzFWhFnxPxGl+69cD1Ou63C13NUPCnmIcrvqCuM6w=="], - - "@grpc/proto-loader/yargs/cliui": ["cliui@8.0.1", "", { "dependencies": { "string-width": "^4.2.0", "strip-ansi": "^6.0.1", "wrap-ansi": "^7.0.0" } }, "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ=="], - - "@grpc/proto-loader/yargs/string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "@grpc/proto-loader/yargs/yargs-parser": ["yargs-parser@21.1.1", "", {}, "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw=="], - - "@inquirer/core/wrap-ansi/ansi-styles": ["ansi-styles@4.3.0", "", { "dependencies": { "color-convert": "^2.0.1" } }, "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg=="], - - "@inquirer/core/wrap-ansi/string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "@inquirer/core/wrap-ansi/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "@isaacs/cliui/string-width/emoji-regex": ["emoji-regex@9.2.2", "", {}, "sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg=="], - - "@isaacs/cliui/wrap-ansi/ansi-styles": ["ansi-styles@6.2.3", "", {}, "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg=="], - - "@types/ssh2/@types/node/undici-types": ["undici-types@5.26.5", "", {}, "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA=="], - - "@typescript-eslint/typescript-estree/minimatch/brace-expansion": ["brace-expansion@2.0.2", "", { "dependencies": { "balanced-match": "^1.0.0" } }, "sha512-Jt0vHyM+jmUBqojB7E1NIYadt0vI0Qxjxd2TErW94wDz+E2LAm5vKMXXwg6ZZBTHPuUlDgQHKXvjGBdfcF1ZDQ=="], - - "ansi-align/string-width/emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - - "ansi-align/string-width/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "cli-table3/string-width/emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - - "cli-table3/string-width/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "cliui/string-width/emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - - "cliui/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "cliui/wrap-ansi/ansi-styles": ["ansi-styles@4.3.0", "", { "dependencies": { "color-convert": "^2.0.1" } }, "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg=="], - - "dockerode/tar-fs/tar-stream": ["tar-stream@2.2.0", "", { "dependencies": { "bl": "^4.0.3", "end-of-stream": "^1.4.1", "fs-constants": "^1.0.0", "inherits": "^2.0.3", "readable-stream": "^3.1.1" } }, "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ=="], - - "dotgitignore/find-up/locate-path": ["locate-path@3.0.0", "", { "dependencies": { "p-locate": "^3.0.0", "path-exists": "^3.0.0" } }, "sha512-7AO748wWnIhNqAuaty2ZWHkQHRSNfPVIsPIfwEOWO22AmaoVrWavlOcMR5nzTLNYvp36X220/maaRsrec1G65A=="], - - "eslint/chalk/ansi-styles": ["ansi-styles@4.3.0", "", { "dependencies": { "color-convert": "^2.0.1" } }, "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg=="], - - "get-pkg-repo/through2/readable-stream": ["readable-stream@2.3.8", "", { "dependencies": { "core-util-is": "~1.0.0", "inherits": "~2.0.3", "isarray": "~1.0.0", "process-nextick-args": "~2.0.0", "safe-buffer": "~5.1.1", "string_decoder": "~1.1.1", "util-deprecate": "~1.0.1" } }, "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA=="], - - "glob/minimatch/brace-expansion": ["brace-expansion@2.0.2", "", { "dependencies": { "balanced-match": "^1.0.0" } }, "sha512-Jt0vHyM+jmUBqojB7E1NIYadt0vI0Qxjxd2TErW94wDz+E2LAm5vKMXXwg6ZZBTHPuUlDgQHKXvjGBdfcF1ZDQ=="], - - "jszip/readable-stream/safe-buffer": ["safe-buffer@5.1.2", "", {}, "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="], - - "jszip/readable-stream/string_decoder": ["string_decoder@1.1.1", "", { "dependencies": { "safe-buffer": "~5.1.0" } }, "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg=="], - - "lazystream/readable-stream/safe-buffer": ["safe-buffer@5.1.2", "", {}, "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="], - - "lazystream/readable-stream/string_decoder": ["string_decoder@1.1.1", "", { "dependencies": { "safe-buffer": "~5.1.0" } }, "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg=="], - - "meow/read-pkg-up/find-up": ["find-up@4.1.0", "", { "dependencies": { "locate-path": "^5.0.0", "path-exists": "^4.0.0" } }, "sha512-PpOwAdQ/YlXQ2vj8a3h8IipDuYRi3wceVQQGYWxNINccq40Anw7BlsEXCMbt1Zt+OLA6Fq9suIpIWD0OsnISlw=="], - - "meow/read-pkg-up/read-pkg": ["read-pkg@5.2.0", "", { "dependencies": { "@types/normalize-package-data": "^2.4.0", "normalize-package-data": "^2.5.0", "parse-json": "^5.0.0", "type-fest": "^0.6.0" } }, "sha512-Ug69mNOpfvKDAc2Q8DRpMjjzdtrnv9HcSMX+4VsZxD1aZ6ZzrIE7rlzXBtWTyhULSMKg076AW6WR5iZpD0JiOg=="], - - "meow/read-pkg-up/type-fest": ["type-fest@0.8.1", "", {}, "sha512-4dbzIzqvjtgiM5rw1k5rEHtBANKmdudhGyBEajN01fEyhaAIhsoKNy6y7+IN93IfpFtwY9iqi7kD+xwKhQsNJA=="], - - "read-pkg-up/find-up/locate-path": ["locate-path@2.0.0", "", { "dependencies": { "p-locate": "^2.0.0", "path-exists": "^3.0.0" } }, "sha512-NCI2kiDkyR7VeEKm27Kda/iQHyKJe1Bu0FlTbYp3CqJu+9IFe9bLyAjMxf5ZDDbEg+iMPzB5zYyUTSm8wVTKmA=="], - - "read-pkg/normalize-package-data/hosted-git-info": ["hosted-git-info@2.8.9", "", {}, "sha512-mxIDAb9Lsm6DoOJ7xH+5+X4y1LU/4Hi50L9C5sIswK3JzULS4bwk1FvjdBgvYR4bzT4tuUQiC15FE2f5HbLvYw=="], - - "read-pkg/normalize-package-data/semver": ["semver@5.7.2", "", { "bin": "bin/semver" }, "sha512-cBznnQ9KjJqU67B52RMC65CMarK2600WFnbkcaiwWq3xy/5haFJlshgnpjovMVJ+Hff49d8GEn0b87C5pDQ10g=="], - - "readdir-glob/minimatch/brace-expansion": ["brace-expansion@2.0.2", "", { "dependencies": { "balanced-match": "^1.0.0" } }, "sha512-Jt0vHyM+jmUBqojB7E1NIYadt0vI0Qxjxd2TErW94wDz+E2LAm5vKMXXwg6ZZBTHPuUlDgQHKXvjGBdfcF1ZDQ=="], - - "standard-version/chalk/escape-string-regexp": ["escape-string-regexp@1.0.5", "", {}, "sha512-vbRorB5FUQWvla16U8R/qgaFIya2qGzwDrNmCZuYKrbdSUMG6I1ZCGQRefkRVhuOkIGVne7BQ35DSfo1qvJqFg=="], - - "standard-version/chalk/supports-color": ["supports-color@5.5.0", "", { "dependencies": { "has-flag": "^3.0.0" } }, "sha512-QjVjwdXIt408MIiAqCX4oUKsgU2EqAGzs2Ppkm4aQYbjm+ZEWEcW4SfFNTr4uMNZma0ey4f5lgLrkB0aX0QMow=="], - - "stream-parser/debug/ms": ["ms@2.0.0", "", {}, "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A=="], - - "string-width-cjs/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "teeny-request/https-proxy-agent/agent-base": ["agent-base@6.0.2", "", { "dependencies": { "debug": "4" } }, "sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ=="], - - "test-exclude/minimatch/brace-expansion": ["brace-expansion@2.0.2", "", { "dependencies": { "balanced-match": "^1.0.0" } }, "sha512-Jt0vHyM+jmUBqojB7E1NIYadt0vI0Qxjxd2TErW94wDz+E2LAm5vKMXXwg6ZZBTHPuUlDgQHKXvjGBdfcF1ZDQ=="], - - "wrap-ansi-cjs/ansi-styles/color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], - - "wrap-ansi-cjs/string-width/emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - - "wrap-ansi-cjs/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "yargs/string-width/emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - - "yargs/string-width/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "@aws-crypto/sha1-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="], - - "@aws-crypto/sha256-browser/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="], - - "@aws-crypto/util/@smithy/util-utf8/@smithy/util-buffer-from/@smithy/is-array-buffer": ["@smithy/is-array-buffer@2.2.0", "", { "dependencies": { "tslib": "^2.6.2" } }, "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/cliui": ["cliui@8.0.1", "", { "dependencies": { "string-width": "^4.2.0", "strip-ansi": "^6.0.1", "wrap-ansi": "^7.0.0" } }, "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/yargs-parser": ["yargs-parser@21.1.1", "", {}, "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw=="], - - "@grpc/proto-loader/yargs/cliui/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "@grpc/proto-loader/yargs/cliui/wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], - - "@grpc/proto-loader/yargs/string-width/emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - - "@grpc/proto-loader/yargs/string-width/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "@inquirer/core/wrap-ansi/ansi-styles/color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], - - "@inquirer/core/wrap-ansi/string-width/emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - - "@inquirer/core/wrap-ansi/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "ansi-align/string-width/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "cli-table3/string-width/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "cliui/wrap-ansi/ansi-styles/color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], - - "dotgitignore/find-up/locate-path/p-locate": ["p-locate@3.0.0", "", { "dependencies": { "p-limit": "^2.0.0" } }, "sha512-x+12w/To+4GFfgJhBEpiDcLozRJGegY+Ei7/z0tSLkMmxGZNybVMSfWj9aJn8Z5Fc7dBUNJOOVgPv2H7IwulSQ=="], - - "dotgitignore/find-up/locate-path/path-exists": ["path-exists@3.0.0", "", {}, "sha512-bpC7GYwiDYQ4wYLe+FA8lhRjhQCMcQGuSgGGqDkg/QerRWw9CmGRT0iSOVRSZJ29NMLZgIzqaljJ63oaL4NIJQ=="], - - "eslint/chalk/ansi-styles/color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], - - "get-pkg-repo/through2/readable-stream/safe-buffer": ["safe-buffer@5.1.2", "", {}, "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="], - - "get-pkg-repo/through2/readable-stream/string_decoder": ["string_decoder@1.1.1", "", { "dependencies": { "safe-buffer": "~5.1.0" } }, "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg=="], - - "meow/read-pkg-up/find-up/locate-path": ["locate-path@5.0.0", "", { "dependencies": { "p-locate": "^4.1.0" } }, "sha512-t7hw9pI+WvuwNJXwk5zVHpyhIqzg2qTlklJOf0mVxGSbe3Fp2VieZcduNYjaLDoy6p9uGpQEGWG87WpMKlNq8g=="], - - "meow/read-pkg-up/read-pkg/normalize-package-data": ["normalize-package-data@2.5.0", "", { "dependencies": { "hosted-git-info": "^2.1.4", "resolve": "^1.10.0", "semver": "2 || 3 || 4 || 5", "validate-npm-package-license": "^3.0.1" } }, "sha512-/5CMN3T0R4XTj4DcGaexo+roZSdSFW/0AOOTROrjxzCG1wrWXEsGbRKevjlIL+ZDE4sZlJr5ED4YW0yqmkK+eA=="], - - "meow/read-pkg-up/read-pkg/parse-json": ["parse-json@5.2.0", "", { "dependencies": { "@babel/code-frame": "^7.0.0", "error-ex": "^1.3.1", "json-parse-even-better-errors": "^2.3.0", "lines-and-columns": "^1.1.6" } }, "sha512-ayCKvm/phCGxOkYRSCM82iDwct8/EonSEgCSxWxD7ve6jHggsFl4fZVQBPRNgQoKiuV/odhFrGzQXZwbifC8Rg=="], - - "meow/read-pkg-up/read-pkg/type-fest": ["type-fest@0.6.0", "", {}, "sha512-q+MB8nYR1KDLrgr4G5yemftpMC7/QLqVndBmEEdqzmNj5dcFOO4Oo8qlwZE3ULT3+Zim1F8Kq4cBnikNhlCMlg=="], - - "read-pkg-up/find-up/locate-path/p-locate": ["p-locate@2.0.0", "", { "dependencies": { "p-limit": "^1.1.0" } }, "sha512-nQja7m7gSKuewoVRen45CtVfODR3crN3goVQ0DDZ9N3yHxgpkuBhZqsaiotSQRrADUrne346peY7kT3TSACykg=="], - - "read-pkg-up/find-up/locate-path/path-exists": ["path-exists@3.0.0", "", {}, "sha512-bpC7GYwiDYQ4wYLe+FA8lhRjhQCMcQGuSgGGqDkg/QerRWw9CmGRT0iSOVRSZJ29NMLZgIzqaljJ63oaL4NIJQ=="], - - "standard-version/chalk/supports-color/has-flag": ["has-flag@3.0.0", "", {}, "sha512-sKJf1+ceQBr4SMkvQnBDNDtf4TXpVhVGateu0t918bl30FnbE2m4vNLX+VWe/dpjlb+HugGYzW7uQXH98HPEYw=="], - - "wrap-ansi-cjs/ansi-styles/color-convert/color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], - - "yargs/string-width/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/cliui/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/cliui/wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/string-width/emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/string-width/strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], - - "@grpc/proto-loader/yargs/cliui/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "@grpc/proto-loader/yargs/cliui/wrap-ansi/ansi-styles": ["ansi-styles@4.3.0", "", { "dependencies": { "color-convert": "^2.0.1" } }, "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg=="], - - "@grpc/proto-loader/yargs/string-width/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "@inquirer/core/wrap-ansi/ansi-styles/color-convert/color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], - - "cliui/wrap-ansi/ansi-styles/color-convert/color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], - - "dotgitignore/find-up/locate-path/p-locate/p-limit": ["p-limit@2.3.0", "", { "dependencies": { "p-try": "^2.0.0" } }, "sha512-//88mFWSJx8lxCzwdAABTJL2MyWB12+eIY7MDL2SqLmAkeKU9qxRvWuSyTjm3FUmpBEMuFfckAIqEaVGUDxb6w=="], - - "eslint/chalk/ansi-styles/color-convert/color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], - - "meow/read-pkg-up/find-up/locate-path/p-locate": ["p-locate@4.1.0", "", { "dependencies": { "p-limit": "^2.2.0" } }, "sha512-R79ZZ/0wAxKGu3oYMlz8jy/kbhsNrS7SKZ7PxEHBgJ5+F2mtFW2fK2cOtBh1cHYkQsbzFV7I+EoRKe6Yt0oK7A=="], - - "meow/read-pkg-up/read-pkg/normalize-package-data/hosted-git-info": ["hosted-git-info@2.8.9", "", {}, "sha512-mxIDAb9Lsm6DoOJ7xH+5+X4y1LU/4Hi50L9C5sIswK3JzULS4bwk1FvjdBgvYR4bzT4tuUQiC15FE2f5HbLvYw=="], - - "meow/read-pkg-up/read-pkg/normalize-package-data/semver": ["semver@5.7.2", "", { "bin": "bin/semver" }, "sha512-cBznnQ9KjJqU67B52RMC65CMarK2600WFnbkcaiwWq3xy/5haFJlshgnpjovMVJ+Hff49d8GEn0b87C5pDQ10g=="], - - "read-pkg-up/find-up/locate-path/p-locate/p-limit": ["p-limit@1.3.0", "", { "dependencies": { "p-try": "^1.0.0" } }, "sha512-vvcXsLAJ9Dr5rQOPk7toZQZJApBl2K4J6dANSsEuh6QI41JYcsS/qhTGa9ErIUUgK3WNQoJYvylxvjqmiqEA9Q=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/cliui/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/cliui/wrap-ansi/ansi-styles": ["ansi-styles@4.3.0", "", { "dependencies": { "color-convert": "^2.0.1" } }, "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/string-width/strip-ansi/ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], - - "@grpc/proto-loader/yargs/cliui/wrap-ansi/ansi-styles/color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], - - "meow/read-pkg-up/find-up/locate-path/p-locate/p-limit": ["p-limit@2.3.0", "", { "dependencies": { "p-try": "^2.0.0" } }, "sha512-//88mFWSJx8lxCzwdAABTJL2MyWB12+eIY7MDL2SqLmAkeKU9qxRvWuSyTjm3FUmpBEMuFfckAIqEaVGUDxb6w=="], - - "read-pkg-up/find-up/locate-path/p-locate/p-limit/p-try": ["p-try@1.0.0", "", {}, "sha512-U1etNYuMJoIz3ZXSrrySFjsXQTWOx2/jdi86L+2pRvph/qMKL6sbcCYdH23fqsbm8TH2Gn0OybpT4eSFlCVHww=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/cliui/wrap-ansi/ansi-styles/color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], - - "@grpc/proto-loader/yargs/cliui/wrap-ansi/ansi-styles/color-convert/color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], - - "@grpc/grpc-js/@grpc/proto-loader/yargs/cliui/wrap-ansi/ansi-styles/color-convert/color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], - } -} diff --git a/docs/ADR-001-generational-mvcc.md b/docs/ADR-001-generational-mvcc.md deleted file mode 100644 index b4c4d8dd..00000000 --- a/docs/ADR-001-generational-mvcc.md +++ /dev/null @@ -1,295 +0,0 @@ -# ADR-001: Generational MVCC storage and the immutable Db API - -**Status:** Accepted (ships in 8.0) -**Date:** 2026-06-10 - -## Context - -Before 8.0, Brainy carried two overlapping version-control subsystems: a -copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a -separate versioning subsystem (`versions.save/list/compare/restore/prune`), -plus a read-only historical adapter for commit-based time travel. Together -they were ~5,100 LOC of mechanism for one product need: *read a consistent -past state while the store keeps moving, and snapshot/restore cheaply.* - -Neither subsystem gave a precise isolation guarantee. Reads raced in-place -JSON overwrites, so a "snapshot" was only as immutable as the bytes it -happened to share with the live store. - -8.0 replaces both with **one mechanism**: generational MVCC over immutable, -generation-stamped records, exposed through a Datomic-style immutable -database value (`Db`). The same model is implemented natively by versioned -index providers (LSM snapshots), so semantics are identical on the pure-JS -path and the native path. - -## Decision - -### The model - -- A **monotonic u64 generation counter** is the store's logical clock. It - advances once per committed `transact()` batch and once per - single-operation write (`add`/`update`/`remove`/`relate`/…), so - `brain.generation()` is always a meaningful watermark. It is persisted in - `_system/generation.json` and never reissued for anything durable. -- `brain.now()` **pins** the current generation in O(1) and returns a `Db` — - an immutable view. Pins are refcounted; `db.release()` (with a - `FinalizationRegistry` backstop for leaked values) ends the pin. -- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative - batch atomically as **exactly one generation**, with whole-store - compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and - reified transaction metadata appended to `_system/tx-log.jsonl`. -- `brain.asOf(generation | Date | snapshotPath)` opens past state; - `db.with(ops)` layers a speculative in-memory overlay (never touching - disk, the counter, or index providers); `db.persist(path)` cuts an - instant snapshot; `brain.restore(path, { confirm: true })` replaces state - from one; `Brainy.load(path)` opens a snapshot read-only with the full - query surface. - -### Persisted layout - -All paths are storage-root-relative: - -``` -_system/generation.json { generation, updatedAt } atomic tmp+rename -_system/manifest.json { version, generation, atomic tmp+rename - committedAt, horizon } (the commit point) -_system/tx-log.jsonl one line per committed append-only - transact: { generation, - timestamp, meta? } -_generations//tx.json the generation-N delta: immutable - touched noun/verb ids + meta -_generations//prev/.json before-image of as of immutable - commit N (raw stored bytes; - null parts = file was absent) -``` - -**Why per-generation deltas instead of a global `id → latest generation` -map in the manifest:** a global map makes every commit O(all ids) — the -whole map must be rewritten to swap it atomically. The delta layout makes a -commit O(ids touched) and keeps the manifest a fixed-size watermark, while -point-in-time resolution stays correct (see "Read resolution" below). The -trade is that resolution at a pinned generation scans the deltas of later -commits — bounded by the number of commits since the pin, which is exactly -the window compaction keeps short. - -Before-images are deliberately the *only* per-id records. They serve both -roles the layer needs — the crash-recovery undo log and the point-in-time -read source. After-images would duplicate state that is already readable -(the canonical entity files hold the latest bytes; earlier states resolve -from later before-images) and would double record I/O per commit. - -### Commit protocol (durability) - -`transact()` commits under a store-wide mutex: - -1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError` - before anything is staged. -2. **Reserve** generation `N` (counter increment). -3. **Stage the undo log:** write the before-image of every touched id plus - `tx.json` under `_generations/N/`, then **fsync** the files and their - directories. From this point, any crash is recoverable to the exact - pre-transaction bytes. -4. **Execute** the planned batch through the TransactionManager (which has - its own operation-level rollback for in-flight failures). -5. **Commit point:** persist the counter, then write `_system/manifest.json` - via atomic tmp+rename and fsync it. The rename *is* the commit: a - generation directory is committed if and only if `N ≤ - manifest.generation`. -6. Append the tx-log line (advisory metadata — a crash between 5 and 6 - keeps the transaction). - -**Crash recovery (on open):** any `_generations/` directory with -`N > manifest.generation` is an uncommitted transaction. Its before-images -are restored to the canonical paths (idempotently — recovery itself can -crash and rerun) and the directory is removed. Because recovery runs before -any index is built, and a recovery that rolled something back forces a full -index rebuild, derived indexes never observe rolled-back state. Reader-mode -instances skip recovery (readers never write; the next writer repairs). - -A failed (non-crash) transaction takes the same staging directory down the -abort path: the TransactionManager rolls back applied operations, the -staging directory is removed, and the generation reservation is returned — -a failed batch leaves the generation counter unchanged. - -### Read resolution at a pinned generation - -The state of id X at pinned generation G is: - -- the before-image stored by the **first committed generation after G that - touched X**, or -- the live canonical bytes, when nothing after G touched X. - -While nothing has committed past G, *every* read on the `Db` delegates to -the live fast paths untouched — `now()` adds no read overhead until history -actually moves. - -**Two read paths, one result set.** `get()`, metadata-level `find()`, and -filter-based `related()` resolve directly through the record layer at any -reachable pinned generation — no extra cost beyond scanning the deltas of -later commits. Index-accelerated dimensions (semantic/vector search, graph -traversal, cursors, aggregation) are served by **at-generation index -materialization**: the first such query on a historical `Db` copies the -exact at-G record set (live bytes for ids untouched since the pin, -before-images for the rest; a final reconciliation pass runs under the -commit mutex so transactions racing the copy cannot skew it) into an -ephemeral in-memory store and opens a read-only engine over it — the same -vector/metadata/graph index classes the live brain uses, sharing the host's -embedder and aggregate definitions. The handle is cached on the `Db` and -freed by `release()`. - -**Cost, stated plainly:** materialization is O(n at G) time and memory, -once per `Db`. That is the open-core price of historical index queries. A -native `VersionedIndexProvider` (`isGenerationVisible()` + pins over -retained LSM segments) serves the same reads with no rebuild at all — the -materializer is the correctness baseline, the provider is the accelerator. - -**The one remaining boundary.** Speculative `with()` overlays throw -`SpeculativeOverlayError` for index-accelerated queries and `persist()`: -overlay entities carry no embeddings (`with()` never invokes the embedder), -so a "full" index query over an overlay would silently exclude the -overlay's own entities. Commit with `transact()` to get the full surface. - -**History granularity (Model-B).** EVERY write is its own immutable generation -— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`. -Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/ -`diff()`/`history()` exactly like transacts; a pin always freezes against later -writes. `transact()` groups several operations into ONE atomic generation. - -Single-op history durability is **async group-commit**: the live write hits -canonical storage immediately (acknowledged), while its before-image is buffered -and persisted to disk in one batched fsync on a size/timer trigger (or forced by -`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in -point-in-time resolution exactly like on-disk generations, so the synchronous -`now()` freezes with no forced flush. A hard crash before the flush loses only -the buffered *history* of the last window — never live data — and a crash -*mid-flush* is recovered by **drop-without-restore** (the partial generation's -before-images are discarded, never replayed, because the live write was already -acknowledged; restoring them would silently revert it). - -### Pinning, retention, compaction - -- Each live `Db` holds one refcounted pin on its generation (plus a - `pin(generation)` on every registered `VersionedIndexProvider`, whose - explicit pin lifetime overrides any time-based snapshot retention the - provider has). -- The constructor **`retention`** knob governs auto-compaction (on `flush()`/ - `close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config; - driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) · - `'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit - CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims - manually on the same caps — the oldest unpinned record-sets are reclaimed - while ANY supplied cap is exceeded. -- A record-set `N` is reclaimed only when `N` is at or below **every** live - pin — deleting `N` can only break readers pinned *below* `N`, because - resolution reads before-images from generations strictly greater than the - pin. Live pins are ALWAYS exempt, in every retention mode. -- The manifest records the **horizon** (highest reclaimed generation). - Generations below the horizon are unreachable; `asOf()` on them throws - `GenerationCompactedError`. The horizon itself stays reachable, resolved - from the record-sets above it. To keep a generation readable forever, - `persist()` it first — snapshots are self-contained. - -### Snapshots and restore - -`db.persist(path)` flushes indexes, then cuts the snapshot under the -store's commit mutex (no commit, compaction, or counter write can -interleave). On filesystem storage it is a **hard-link farm**: every data -file is immutable-by-rename, so linking is safe — later rewrites swap -inodes and the snapshot keeps the old bytes. The two exceptions are handled -explicitly: the append-in-place tx-log is byte-copied, and process-local -lock state is excluded. Cross-device targets (and filesystems that refuse -links) fall back to per-file byte copies. In-memory stores serialize to the -same directory layout, so persisting a memory brain produces a real, -durable, loadable store. - -`persist()` requires the view to still be the store's latest generation -(a snapshot captures current bytes); a view that history has moved past -throws rather than persisting the wrong state. - -`restore(path, { confirm: true })` replaces the store's contents from a -snapshot via byte copy (never links — the snapshot stays independent), -reloads all adapter-internal derived state, rebuilds all indexes, and -floors the generation counter at its pre-restore value so observed -generation numbers are never reissued. Live pins do not survive a restore; -a warning is logged when any exist. - -### Versioned index providers - -Native index providers may implement the optional 4-method -`VersionedIndexProvider` capability (`generation()`, -`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the -boundary). The locked consistency model: providers are **post-commit -appliers**. The storage-record commit is the source of truth; provider -index state is derived. On open, a provider behind the committed watermark -replays the gap from storage (or requests a rebuild) — there are no -provider rollback hooks, because uncommitted transactions are repaired at -the storage layer before any index opens. Speculative `with()` overlays -never reach providers. - -## Guarantees (and their proofs) - -Each stated guarantee has a test that proves it, not merely exercises it -(`tests/integration/db-mvcc.test.ts`, plus -`tests/unit/db/generationStore.test.ts` for the record layer in isolation): - -| Guarantee | Proof | -|---|---| -| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) | -| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) | -| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) | -| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c | -| Compaction never breaks a pinned read; release enables reclaim | proof 5 | -| `with()` overlays touch nothing durable | proof 6 | -| Generation monotonicity across close/reopen | proof 7 | -| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) | -| Balanced provider pin/release lockstep | proof 9 | - -One deliberate softness: single-operation generation bumps persist the -counter coalesced (per write burst), not per write. Durable artifacts — -records, manifests, snapshots — always persist the counter synchronously at -their own commit points, so a crash inside the coalescing window can lose -only counter values that nothing durable ever referenced. - -## Failure modes - -| Failure | Outcome | -|---|---| -| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. | -| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. | -| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. | -| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. | -| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. | -| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. | -| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). | -| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. | -| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. | - -## Lineage - -The design is an assembly of well-understood prior art, chosen for being -boring where it counts: - -- **Datomic** — the database-as-a-value: an immutable `Db` you query, with - `with()` for speculation and reified transaction metadata instead of - commit messages. -- **LMDB** — reader pins: readers never block writers; a reader's view - stays valid because nothing overwrites the pages (here: records) it - references; reclamation waits for the last reader. -- **LSM trees / Cassandra** — immutable segments make snapshots hard links - and make compaction a retention policy instead of a locking problem. - -## Consequences - -- One mechanism replaces the COW and versioning subsystems (their removal - is the companion change to this ADR). -- In-place branch switching (`checkout`) is gone by design; the replacement - is opening a persisted snapshot as a separate instance — a name→path - mapping where a product needs named branches. -- Every commit pays O(ids touched) extra writes (before-images + delta + - manifest). Single-operation writes pay only an in-memory counter bump - with coalesced persistence. -- The full query surface works at every reachable pinned generation. - Record-path reads (`get`, metadata `find`, filter `related`) are - effectively free; index-accelerated historical queries pay a one-time - O(n at G) materialization per `Db` on the open-core path (freed on - `release()`), and run rebuild-free on a native `VersionedIndexProvider`. diff --git a/docs/API_DECISION_TREE.md b/docs/API_DECISION_TREE.md new file mode 100644 index 00000000..775c0d22 --- /dev/null +++ b/docs/API_DECISION_TREE.md @@ -0,0 +1,481 @@ +# 🧠 Brainy API Decision Tree + +*Choose the right API for your use case with confidence* + +This guide helps you navigate Brainy's comprehensive API surface by asking the right questions to find the perfect method for your specific needs. + +## 🎯 Quick Start: What do you want to do? + +### 📝 **Adding Data** +- **Single entity** → [`brainy.add()`](#adding-single-entities) +- **Multiple entities** → [`brainy.addMany()`](#adding-multiple-entities) +- **Streaming/real-time data** → [Streaming Pipeline](#streaming-data) + +### 🔍 **Finding Data** +- **Natural language search** → [`brainy.find("search query")`](#natural-language-search) +- **Structured/filtered search** → [`brainy.find({ query, where, type })`](#structured-search) +- **Similar entities** → [`brainy.similar()`](#similarity-search) +- **Get by ID** → [`brainy.get()`](#retrieval-by-id) + +### 🔗 **Relationships** +- **Create relationships** → [`brainy.relate()`](#creating-relationships) +- **Query relationships** → [`brainy.getRelations()`](#querying-relationships) +- **Graph traversal** → [Graph Navigation](#graph-operations) + +### 📊 **Advanced Features** +- **File management** → [VFS (Virtual File System)](#file-operations) +- **AI-powered analysis** → [Neural API](#neural-analysis) +- **Clustering/insights** → [Intelligence Systems](#intelligence-systems) + +--- + +## 🔀 Decision Tree Flow + +```mermaid +graph TD + A[What are you trying to do?] --> B[Store Data] + A --> C[Find Data] + A --> D[Manage Relationships] + A --> E[Work with Files] + A --> F[AI Analysis] + + B --> B1[Single Item] + B --> B2[Multiple Items] + B --> B3[Real-time Stream] + + C --> C1[I know the ID] + C --> C2[Natural language query] + C --> C3[Complex filters] + C --> C4[Find similar items] + + D --> D1[Create relationship] + D --> D2[Query relationships] + D --> D3[Graph traversal] + + E --> E1[File operations] + E --> E2[Knowledge-enhanced files] + + F --> F1[Clustering] + F --> F2[Similarity analysis] + F --> F3[Insights generation] +``` + +--- + +## 📝 Adding Data + +### Adding Single Entities + +**Use `brainy.add()` when:** +- Adding one entity at a time +- You need the ID immediately for further operations +- Working with user input or real-time data + +```typescript +// ✅ Perfect for single entities +const id = await brainy.add({ + data: "New research paper on quantum computing", + type: NounType.Document, + metadata: { category: "research", priority: "high" } +}) +``` + +**Decision factors:** +- **Single item?** → `add()` +- **Need immediate ID?** → `add()` +- **Interactive application?** → `add()` + +### Adding Multiple Entities + +**Use `brainy.addMany()` when:** +- Bulk importing data +- Processing batches (>10 items) +- Performance is critical + +```typescript +// ✅ Perfect for bulk operations +const result = await brainy.addMany({ + items: documents.map(doc => ({ + data: doc.content, + type: NounType.Document, + metadata: doc.metadata + })), + chunkSize: 100, + parallel: true +}) +``` + +**Decision factors:** +- **Multiple items (>10)?** → `addMany()` +- **Batch processing?** → `addMany()` +- **Can tolerate some failures?** → `addMany()` with `continueOnError: true` + +### Streaming Data + +**Use Streaming Pipeline when:** +- Real-time data ingestion +- Processing large datasets that don't fit in memory +- Need transformation during ingestion + +```typescript +// ✅ Perfect for streaming +const pipeline = brainy.streaming.pipeline() + .transform(data => ({ ...data, processed: true })) + .batch(50) + .into(brainy) +``` + +--- + +## 🔍 Finding Data + +### Natural Language Search + +**Use `brainy.find("query string")` when:** +- User is typing search queries +- You want semantic understanding +- Building search interfaces + +```typescript +// ✅ Perfect for user searches +const results = await brainy.find("documents about machine learning") +``` + +**Decision factors:** +- **User-generated query?** → Natural language `find()` +- **Semantic understanding needed?** → Natural language `find()` +- **Search interface?** → Natural language `find()` + +### Structured Search + +**Use `brainy.find({ query, where, type })` when:** +- Complex filtering requirements +- Combining text search with metadata filters +- Performance-critical searches + +```typescript +// ✅ Perfect for complex queries +const results = await brainy.find({ + query: "neural networks", + type: NounType.Document, + where: { + status: "published", + year: { $gte: 2020 } + }, + limit: 20 +}) +``` + +**Decision factors:** +- **Need metadata filtering?** → Structured `find()` +- **Performance critical?** → Structured `find()` +- **Complex criteria?** → Structured `find()` + +### Similarity Search + +**Use `brainy.similar()` when:** +- Finding "more like this" content +- Recommendation systems +- Duplicate detection + +```typescript +// ✅ Perfect for recommendations +const similar = await brainy.similar({ + to: "document-id-123", + limit: 10, + type: NounType.Document +}) +``` + +**Decision factors:** +- **"More like this" feature?** → `similar()` +- **Recommendations?** → `similar()` +- **Duplicate detection?** → `similar()` + +### Retrieval by ID + +**Use `brainy.get()` when:** +- You know the exact ID +- Loading specific entities +- Following relationships + +```typescript +// ✅ Perfect for direct access +const entity = await brainy.get("known-id-123") +``` + +**Decision factors:** +- **Known ID?** → `get()` +- **Direct access needed?** → `get()` +- **Following relationships?** → `get()` + +--- + +## 🔗 Relationships + +### Creating Relationships + +**Use `brainy.relate()` when:** +- Connecting two entities +- Building knowledge graphs +- Modeling real-world relationships + +```typescript +// ✅ Perfect for connections +await brainy.relate({ + from: "user-123", + to: "project-456", + type: VerbType.WorksOn, + metadata: { role: "lead", since: "2024-01-01" } +}) +``` + +**Decision factors:** +- **Connecting entities?** → `relate()` +- **Need relationship metadata?** → `relate()` +- **Building graphs?** → `relate()` + +### Querying Relationships + +**Use `brainy.getRelations()` when:** +- Finding all connections for an entity +- Exploring relationship patterns +- Building relationship views + +```typescript +// ✅ Perfect for relationship queries +const relations = await brainy.getRelations({ + from: "user-123", + type: VerbType.WorksOn +}) +``` + +--- + +## 📁 File Operations + +### Basic File Operations + +**Use VFS when:** +- Managing files and directories +- Need hierarchical structure +- Building file explorers + +```typescript +// ✅ Perfect for file management +const vfs = brainy.vfs({ storage: 'filesystem' }) +await vfs.writeFile('/docs/readme.md', 'content') +const files = await vfs.getDirectChildren('/docs') +``` + +**Decision factors:** +- **File management?** → VFS +- **Directory structure?** → VFS +- **File explorer interface?** → VFS + +### Intelligent File Management + +**Use VFS (Semantic VFS) when:** +- Need semantic file search +- Want AI-powered concept extraction +- Building smart file systems +- Require multi-dimensional file access + +```typescript +// ✅ Perfect for intelligent file systems +const knowledgeVFS = await vfs.withKnowledge(brainy) +const insights = await knowledgeVFS.getFileInsights('/project') +``` + +--- + +## 🧠 AI Analysis + +### Clustering + +**Use Neural API clustering when:** +- Discovering data patterns +- Organizing large datasets +- Creating automatic categories + +```typescript +// ✅ Perfect for pattern discovery +const neural = brainy.neural() +const clusters = await neural.cluster({ + entities: entityIds, + k: 5, + method: 'hierarchical' +}) +``` + +### Intelligence Systems + +**Use Triple Intelligence when:** +- Complex multi-criteria searches +- Advanced relationship queries +- Performance-critical operations + +```typescript +// ✅ Perfect for complex queries +const intelligence = brainy.getTripleIntelligence() +const results = await intelligence.query({ + vector: queryVector, + metadata: { category: 'research' }, + graph: { connected: 'user-123' } +}) +``` + +--- + +## 🚀 Performance Optimization Guide + +### When Performance Matters + +| Scenario | Best Choice | Why | +|----------|-------------|-----| +| **Bulk Import** | `addMany()` | Batched operations, parallel processing | +| **Metadata-only Search** | `find({ where: {...} })` | Skips vector computation | +| **Known ID Access** | `get()` | Direct index lookup | +| **Large Result Sets** | Pagination with `offset`/`limit` | Memory efficient | +| **Real-time Streams** | Streaming Pipeline | Memory efficient, scalable | + +### Memory Usage Optimization + +```typescript +// ❌ Memory intensive +const allResults = await brainy.find({ limit: 10000 }) + +// ✅ Memory efficient +for (let offset = 0; offset < total; offset += 100) { + const batch = await brainy.find({ + query: "...", + limit: 100, + offset + }) + await processBatch(batch) +} +``` + +--- + +## 🎯 Common Use Case Patterns + +### Building a Search Interface + +```typescript +// User types query → Natural language search +const searchResults = await brainy.find(userQuery) + +// User applies filters → Structured search +const filteredResults = await brainy.find({ + query: userQuery, + where: selectedFilters, + type: selectedTypes +}) + +// User clicks "more like this" → Similarity search +const similar = await brainy.similar({ to: selectedId }) +``` + +### Building a Recommendation System + +```typescript +// 1. Get user's interaction history +const user = await brainy.get(userId) + +// 2. Find similar users +const similarUsers = await brainy.similar({ to: userId, type: NounType.Person }) + +// 3. Get their liked content +const recommendations = [] +for (const similarUser of similarUsers) { + const relations = await brainy.getRelations({ + from: similarUser.id, + type: VerbType.Likes + }) + recommendations.push(...relations) +} +``` + +### Building a Knowledge Graph + +```typescript +// 1. Add entities +const entities = await Promise.all([ + brainy.add({ data: "Person: Alice", type: NounType.Person }), + brainy.add({ data: "Company: TechCorp", type: NounType.Organization }), + brainy.add({ data: "Project: AI Assistant", type: NounType.Thing }) +]) + +// 2. Create relationships +await brainy.relate({ + from: entities[0], // Alice + to: entities[1], // TechCorp + type: VerbType.WorksFor +}) + +await brainy.relate({ + from: entities[0], // Alice + to: entities[2], // AI Assistant + type: VerbType.WorksOn +}) + +// 3. Query the graph +const aliceConnections = await brainy.getRelations({ from: entities[0] }) +``` + +--- + +## 🔧 Migration Guide + +### From v2.x to v3.x APIs + +| v2.x (Deprecated) | v3.x (Current) | When to Use | +|-------------------|----------------|-------------| +| `brain.store()` | `brainy.add()` | Adding entities | +| `brain.search()` | `brainy.find()` | Searching content | +| `brain.query()` | `brainy.find({ ... })` | Complex queries | +| `brain.similar()` | `brainy.similar()` | ✅ Same API | +| `brain.connect()` | `brainy.relate()` | Creating relationships | + +### Legacy Type Migration + +```typescript +// ❌ v2.x way +import { ISenseAugmentation } from '@soulcraft/brainy/types/augmentations' + +// ✅ v3.x way +import { BrainyAugmentation } from '@soulcraft/brainy' +``` + +--- + +## 🎪 Decision Quick Reference + +**Need to add data?** +- 1 item → `add()` +- Many items → `addMany()` +- Streaming → Pipeline + +**Need to find data?** +- Know ID → `get()` +- Natural search → `find("query")` +- Complex filters → `find({ query, where })` +- Similar items → `similar()` + +**Need relationships?** +- Create → `relate()` +- Query → `getRelations()` +- Complex graph → Triple Intelligence + +**Need files?** +- Basic → VFS (standard operations) +- Smart → Semantic VFS (6 dimensional access + neural extraction) + +**Need AI analysis?** +- Patterns → Neural clustering +- Complex queries → Triple Intelligence + +--- + +*This guide covers 95% of use cases. For edge cases or custom requirements, check the [Core API Patterns](./CORE_API_PATTERNS.md) and [Neural API Patterns](./NEURAL_API_PATTERNS.md) guides.* \ No newline at end of file diff --git a/docs/API_REFERENCE.md b/docs/API_REFERENCE.md new file mode 100644 index 00000000..c63ec632 --- /dev/null +++ b/docs/API_REFERENCE.md @@ -0,0 +1,1577 @@ +# 🧠 Brainy 3.0 Complete API Reference + +> The neural database that thinks - Complete API documentation for all public methods + +## Table of Contents + +- [Quick Start](#quick-start) +- [Core API](#core-api) +- [Batch Operations](#batch-operations) +- [Search & Discovery](#search--discovery) +- [Security API](#security-api) +- [Configuration API](#configuration-api) +- [Data Management API](#data-management-api) +- [Query API](#query-api) +- [Neural API](#neural-api) +- [NLP API](#nlp-api) +- [Streaming Pipeline API](#streaming-pipeline-api) +- [Type Definitions](#type-definitions) + +--- + +## Quick Start + +```typescript +import { Brainy, NounType, VerbType } from '@soulcraft/brainy' + +// Initialize +const brain = new Brainy({ + storage: { type: 'filesystem', path: './brainy-data' }, + model: { type: 'fast', precision: 'Q8' } +}) +await brain.init() + +// Add entities +const id = await brain.add({ + data: 'John Smith is a software engineer', + type: NounType.Person, + metadata: { role: 'engineer' } +}) + +// Search +const results = await brain.find('engineers') + +// Clean up +await brain.close() +``` + +--- + +## Core API + +### `new Brainy(config?: BrainyConfig)` +Creates a new Brainy instance. + +**Parameters:** +- `config` - Optional configuration object + +**Returns:** `Brainy` instance + +**Example:** +```typescript +const brain = new Brainy({ + storage: { type: 'filesystem', options: { path: './data' } }, + model: { type: 'accurate', precision: 'FP32' }, + cache: { maxSize: 5000, ttl: 600000 } +}) +``` + +--- + +### `async init(): Promise` +Initializes the brain, loading models and preparing storage. + +**Must be called before any other operations.** + +**Example:** +```typescript +await brain.init() +``` + +--- + +### `async add(params: AddParams): Promise` +Adds a new entity to the brain. + +**Parameters:** +- `data` (required) - Content to embed and store +- `type` (required) - NounType classification +- `metadata` - Custom metadata object +- `id` - Custom ID (auto-generated if not provided) +- `vector` - Pre-computed embedding vector +- `service` - Service name for multi-tenancy +- `writeOnly` - Skip validation for high-speed ingestion + +**Returns:** Entity ID + +**Example:** +```typescript +const id = await brain.add({ + data: 'Important meeting notes from Q4 planning', + type: NounType.Document, + metadata: { + date: '2024-01-15', + author: 'John Smith', + tags: ['planning', 'Q4'] + } +}) +``` + +--- + +### `async get(id: string): Promise` +Retrieves an entity by ID. + +**Parameters:** +- `id` - Entity ID + +**Returns:** Entity object or null if not found + +**Example:** +```typescript +const entity = await brain.get('uuid-1234') +if (entity) { + console.log(entity.type) // NounType.Document + console.log(entity.metadata) // { date: '2024-01-15', ... } +} +``` + +--- + +### `async update(params: UpdateParams): Promise` +Updates an existing entity. + +**Parameters:** +- `id` (required) - Entity ID to update +- `data` - New content (will re-embed) +- `type` - New type classification +- `metadata` - New or partial metadata +- `merge` - Merge metadata (true) or replace (false), default: true +- `vector` - New embedding vector + +**Example:** +```typescript +await brain.update({ + id: 'uuid-1234', + metadata: { status: 'reviewed' }, + merge: true // Keeps existing metadata, adds status +}) +``` + +--- + +### `async delete(id: string): Promise` +Deletes an entity and all its relationships. + +**Parameters:** +- `id` - Entity ID to delete + +**Example:** +```typescript +await brain.delete('uuid-1234') +``` + +--- + +### `async relate(params: RelateParams): Promise` +Creates a relationship between two entities. + +**Parameters:** +- `from` (required) - Source entity ID +- `to` (required) - Target entity ID +- `type` (required) - VerbType enum value +- `weight` - Relationship strength (0-1), default: 1 +- `metadata` - Relationship metadata +- `bidirectional` - Create reverse relationship +- `service` - Service name for multi-tenancy +- `writeOnly` - Skip validation + +**Validation:** +- `from` and `to` must be different (no self-referential relationships) +- `type` must be a valid VerbType enum value +- `weight` if provided must be between 0 and 1 + +**Returns:** Relationship ID + +**Example:** +```typescript +const relationId = await brain.relate({ + from: 'person-123', + to: 'org-456', + type: VerbType.WorksWith, + weight: 0.95, + metadata: { since: '2020-01-01' }, + bidirectional: true +}) +``` + +--- + +### `async unrelate(id: string): Promise` +Removes a relationship. + +**Parameters:** +- `id` - Relationship ID + +**Example:** +```typescript +await brain.unrelate('relation-789') +``` + +--- + +### `async getRelations(params?: GetRelationsParams): Promise` +Gets relationships for entities. + +**Parameters:** +- `from` - Source entity ID +- `to` - Target entity ID +- `type` - Filter by VerbType(s) +- `limit` - Maximum results (default: 100) +- `offset` - Pagination offset +- `service` - Filter by service + +**Example:** +```typescript +// Get all relationships from an entity +const relations = await brain.getRelations({ + from: 'person-123', + type: [VerbType.WorksWith, VerbType.ReportsTo], + limit: 50 +}) +``` + +--- + +### `async close(): Promise` +Shuts down the brain, cleaning up resources. + +**Example:** +```typescript +await brain.close() +``` + +--- + +## Batch Operations + +### `async addMany(params: AddManyParams): Promise` +Adds multiple entities in batch. + +**Parameters:** +- `items` (required) - Array of AddParams +- `parallel` - Process in parallel (default: true) +- `chunkSize` - Batch size (default: 100) +- `continueOnError` - Continue if some fail +- `onProgress` - Progress callback + +**Returns:** BatchResult with successful/failed counts + +**Example:** +```typescript +const result = await brain.addMany({ + items: [ + { data: 'Doc 1', type: NounType.Document }, + { data: 'Doc 2', type: NounType.Document }, + { data: 'Doc 3', type: NounType.Document } + ], + parallel: true, + onProgress: (done, total) => console.log(`${done}/${total}`) +}) + +console.log(`Added: ${result.successful.length}`) +console.log(`Failed: ${result.failed.length}`) +``` + +--- + +### `async updateMany(params: UpdateManyParams): Promise` +Updates multiple entities in batch. + +**Parameters:** +- `updates` (required) - Array of UpdateParams +- `parallel` - Process in parallel +- `continueOnError` - Continue on failures +- `onProgress` - Progress callback + +**Example:** +```typescript +const result = await brain.updateMany({ + updates: ids.map(id => ({ + id, + metadata: { processed: true }, + merge: true + })), + parallel: true +}) +``` + +--- + +### `async deleteMany(params: DeleteManyParams): Promise` +Deletes multiple entities. + +**Parameters:** +- `ids` - Specific IDs to delete +- `type` - Delete all of a type +- `where` - Delete by metadata filter +- `limit` - Maximum to delete (safety limit) +- `onProgress` - Progress callback + +**Example:** +```typescript +// Delete specific IDs +await brain.deleteMany({ + ids: ['id1', 'id2', 'id3'] +}) + +// Delete by type +await brain.deleteMany({ + type: NounType.Document, + where: { status: 'draft' }, + limit: 100 +}) +``` + +--- + +### `async relateMany(params: RelateManyParams): Promise` +Creates multiple relationships in batch. + +**Parameters:** +- `relations` (required) - Array of RelateParams +- `parallel` - Process in parallel +- `continueOnError` - Continue on failures +- `onProgress` - Progress callback + +**Example:** +```typescript +const result = await brain.relateMany({ + relations: [ + { from: 'a', to: 'b', type: VerbType.RelatedTo }, + { from: 'b', to: 'c', type: VerbType.DependsOn }, + { from: 'c', to: 'a', type: VerbType.References } + ] +}) +``` + +--- + +## Search & Discovery + +### `async find(query: string | FindParams): Promise` +Universal search with Triple Intelligence fusion. + +**Parameters:** +- `query` - Natural language query or structured params +- `vector` - Direct vector search +- `type` - Filter by NounType(s) +- `where` - Metadata filters +- `connected` - Graph constraints +- `near` - Proximity search +- `fusion` - Fusion strategy and weights +- `limit` - Maximum results +- `offset` - Pagination offset +- `explain` - Include score explanation +- `service` - Filter by service +- `writeOnly` - Skip validation + +**Returns:** Array of Result objects with scores + +**Example:** +```typescript +// Natural language search +const results = await brain.find('recent product launches') + +// Structured search with fusion +const results = await brain.find({ + query: 'machine learning', + type: [NounType.Document, NounType.Project], + where: { year: 2024 }, + connected: { + to: 'research-dept', + via: VerbType.CreatedBy + }, + fusion: { + strategy: 'adaptive', + weights: { vector: 0.5, graph: 0.3, field: 0.2 } + }, + limit: 20, + explain: true +}) +``` + +--- + +### `async similar(params: SimilarParams): Promise` +Finds similar entities using vector similarity. + +**Parameters:** +- `to` (required) - Entity ID, Entity object, or Vector +- `limit` - Maximum results (default: 10) +- `threshold` - Minimum similarity (0-1) +- `type` - Filter by type(s) +- `where` - Metadata filters + +**Example:** +```typescript +const similar = await brain.similar({ + to: 'doc-123', + limit: 5, + threshold: 0.8, + type: NounType.Document +}) +``` + +--- + +### `async insights(): Promise` +Gets statistics and insights about the data. + +**Returns:** +- `entities` - Total entity count +- `relationships` - Total relationship count +- `types` - Count by NounType +- `services` - List of services +- `density` - Relationships per entity + +**Example:** +```typescript +const insights = await brain.insights() +console.log(`Entities: ${insights.entities}`) +console.log(`Graph density: ${insights.density.toFixed(2)}`) +``` + +--- + +### `async suggest(params?: SuggestParams): Promise` +AI-powered suggestions based on current data. + +**Parameters:** +- `context` - Context for suggestions +- `type` - Filter by type(s) +- `limit` - Maximum suggestions per category +- `service` - Filter by service + +**Returns:** +- `queries` - Suggested search queries +- `connections` - Suggested relationships +- `insights` - Data insights +- `patterns` - Detected patterns + +**Example:** +```typescript +const suggestions = await brain.suggest({ + context: 'product development', + limit: 5 +}) + +// Use suggestions +for (const query of suggestions.queries) { + console.log(`Try searching: ${query}`) +} +``` + +--- + +## Security API + +Access via: `const security = await brain.security()` + +### `async encrypt(data: string): Promise` +Encrypts data using AES-256-CBC. + +**Example:** +```typescript +const encrypted = await security.encrypt('sensitive data') +``` + +--- + +### `async decrypt(encryptedData: string): Promise` +Decrypts previously encrypted data. + +**Example:** +```typescript +const decrypted = await security.decrypt(encrypted) +``` + +--- + +### `async hash(data: string, algorithm?: 'sha256'|'sha512'): Promise` +Creates cryptographic hash. + +**Example:** +```typescript +const hash = await security.hash('password', 'sha512') +``` + +--- + +### `async compare(data: string, hash: string): Promise` +Compares data against hash (constant-time). + +**Example:** +```typescript +const matches = await security.compare('password', hash) +``` + +--- + +### `async generateToken(bytes?: number): Promise` +Generates secure random token. + +**Example:** +```typescript +const token = await security.generateToken(32) +``` + +--- + +### `async deriveKey(password: string, salt?: string): Promise<{key: string, salt: string}>` +Derives key from password using PBKDF2-like iterations. + +**Example:** +```typescript +const { key, salt } = await security.deriveKey('userPassword') +``` + +--- + +### `async sign(data: string, secret?: string): Promise` +Signs data with HMAC-SHA256. + +**Example:** +```typescript +const signature = await security.sign(data, secret) +``` + +--- + +### `async verify(data: string, signature: string, secret: string): Promise` +Verifies HMAC signature. + +**Example:** +```typescript +const valid = await security.verify(data, signature, secret) +``` + +--- + +## Storage Configuration + +Brainy supports multiple storage backends for different deployment scenarios. + +### Storage Types + +#### File System Storage (Recommended for Development) +Persistent local storage using the filesystem. + +```typescript +const brain = new Brainy({ + storage: { + type: 'filesystem', + rootDirectory: './brainy-data' + } +}) +``` + +#### Amazon S3 Storage +Scalable cloud storage with S3. + +```typescript +const brain = new Brainy({ + storage: { + type: 's3', + s3Storage: { + bucketName: 'my-bucket', + region: 'us-east-1', + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY + } + } +}) +``` + +#### Cloudflare R2 Storage +Scalable cloud storage with Cloudflare R2 (S3-compatible). + +```typescript +const brain = new Brainy({ + storage: { + type: 'r2', + r2Storage: { + bucketName: 'my-bucket', + accountId: process.env.CF_ACCOUNT_ID, + accessKeyId: process.env.R2_ACCESS_KEY_ID, + secretAccessKey: process.env.R2_SECRET_ACCESS_KEY + } + } +}) +``` + +#### Google Cloud Storage (Native SDK) 🆕 +**Recommended for GCS deployments.** Uses native `@google-cloud/storage` SDK with automatic authentication. + +**Key Benefits:** +- ✅ Application Default Credentials (ADC) - Zero config in Cloud Run/GCE +- ✅ Better performance with native GCS optimizations +- ✅ No HMAC key management required +- ✅ Automatic service account integration + +**Option 1: Explicit Type (Recommended)** + +With Application Default Credentials (Cloud Run/GCE): +```typescript +const brain = new Brainy({ + storage: { + type: 'gcs-native', // ⚠️ Must be 'gcs-native' for native SDK + gcsNativeStorage: { + bucketName: 'my-bucket' + // No credentials needed - ADC automatic! + } + } +}) +``` + +With Service Account Key File: +```typescript +const brain = new Brainy({ + storage: { + type: 'gcs-native', + gcsNativeStorage: { + bucketName: 'my-bucket', + keyFilename: '/path/to/service-account.json' + } + } +}) +``` + +With Service Account Credentials Object: +```typescript +const brain = new Brainy({ + storage: { + type: 'gcs-native', + gcsNativeStorage: { + bucketName: 'my-bucket', + credentials: { + client_email: 'service@project.iam.gserviceaccount.com', + private_key: process.env.GCS_PRIVATE_KEY + } + } + } +}) +``` + +**Option 2: Auto-Detection** + +You can omit the `type` field and let Brainy auto-detect based on the config object: +```typescript +const brain = new Brainy({ + storage: { + gcsNativeStorage: { + bucketName: 'my-bucket' + // type defaults to 'auto', will use native SDK + } + } +}) +``` + +#### Google Cloud Storage (S3-Compatible) - Legacy +GCS using HMAC keys for S3-compatible access. **Consider migrating to 'gcs-native' for better performance.** + +```typescript +const brain = new Brainy({ + storage: { + type: 'gcs', // ⚠️ Must be 'gcs' for S3-compatible mode + gcsStorage: { // ⚠️ Must use 'gcsStorage' (not 'gcsNativeStorage') + bucketName: 'my-bucket', + region: 'us-central1', + accessKeyId: process.env.GCS_ACCESS_KEY_ID, + secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY, + endpoint: 'https://storage.googleapis.com' + } + } +}) +``` + +**⚠️ Common Mistakes:** +```typescript +// ❌ WRONG - type/config mismatch (will fall back to memory storage) +{ + type: 'gcs', + gcsNativeStorage: { bucketName: 'my-bucket' } +} + +// ❌ WRONG - type/config mismatch (will fall back to memory storage) +{ + type: 'gcs-native', + gcsStorage: { bucketName: 'my-bucket', accessKeyId: '...', secretAccessKey: '...' } +} + +// ✅ CORRECT - type matches config object +{ + type: 'gcs-native', + gcsNativeStorage: { bucketName: 'my-bucket' } +} + +// ✅ CORRECT - auto-detection +{ + gcsNativeStorage: { bucketName: 'my-bucket' } +} +``` + +### Storage Features + +All storage adapters support: +- ✅ **UUID-based sharding** - 256 buckets (00-ff) for scalability +- ✅ **Pagination** - Efficient cursor-based pagination across shards +- ✅ **Statistics** - O(1) count operations +- ✅ **Caching** - Multi-level cache for performance +- ✅ **Backpressure** - Automatic flow control +- ✅ **Throttling detection** - Adaptive retry on rate limits + +### Migration from HMAC to Native GCS + +If you're currently using `type: 'gcs'` with HMAC keys, migrating to `type: 'gcs-native'` is straightforward: + +**Before (S3-Compatible with HMAC):** +```typescript +const brain = new Brainy({ + storage: { + type: 'gcs', // ⚠️ Old: S3-compatible mode + gcsStorage: { // ⚠️ Old: HMAC credentials + bucketName: 'my-bucket', + accessKeyId: process.env.GCS_ACCESS_KEY_ID, + secretAccessKey: process.env.GCS_SECRET_ACCESS_KEY + } + } +}) +``` + +**After (Native SDK with ADC):** +```typescript +const brain = new Brainy({ + storage: { + type: 'gcs-native', // ✅ New: Native SDK mode + gcsNativeStorage: { // ✅ New: ADC authentication + bucketName: 'my-bucket' + // ADC handles authentication automatically + } + } +}) +``` + +**⚠️ Important Migration Notes:** + +1. **Change BOTH the type AND the config object:** + - `type: 'gcs'` → `type: 'gcs-native'` + - `gcsStorage` → `gcsNativeStorage` + +2. **Remove HMAC keys** - Not needed with ADC: + - Remove `accessKeyId` + - Remove `secretAccessKey` + - Remove `region` (optional with native SDK) + +3. **Set up ADC in your environment:** + ```bash + # Cloud Run/GCE: Nothing needed, ADC is automatic + + # Local development: + gcloud auth application-default login + + # Or set GOOGLE_APPLICATION_CREDENTIALS: + export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json" + ``` + +**Data Migration:** +✅ **No data migration required!** Both adapters use the same path structure: +- `entities/nouns/vectors/{shard}/{id}.json` +- `entities/nouns/metadata/{shard}/{id}.json` +- `entities/verbs/vectors/{shard}/{id}.json` + +Simply change your code and restart your application. Existing data will work immediately. + +--- + +## Configuration API + +Access via: `const config = await brain.config()` + +### `async set(params): Promise` +Sets configuration value. + +**Parameters:** +- `key` (required) - Configuration key +- `value` (required) - Value to store +- `encrypt` - Encrypt value + +**Example:** +```typescript +await config.set({ + key: 'api.key', + value: 'secret-key-123', + encrypt: true +}) +``` + +--- + +### `async get(params): Promise` +Gets configuration value. + +**Parameters:** +- `key` (required) - Configuration key +- `decrypt` - Decrypt if encrypted +- `defaultValue` - Default if not found + +**Example:** +```typescript +const apiKey = await config.get({ + key: 'api.key', + decrypt: true, + defaultValue: 'default-key' +}) +``` + +--- + +### `async delete(key: string): Promise` +Deletes configuration key. + +--- + +### `async list(): Promise` +Lists all configuration keys. + +--- + +### `async has(key: string): Promise` +Checks if key exists. + +--- + +### `async clear(): Promise` +Clears all configuration. + +--- + +### `async export(): Promise>` +Exports all configuration. + +--- + +### `async import(config: Record): Promise` +Imports configuration. + +--- + +## Data Management API + +Access via: `const data = await brain.data()` + +### `async backup(options?: BackupOptions): Promise` +Creates backup of all data. + +**Parameters:** +- `includeVectors` - Include embeddings (default: true) +- `compress` - Compress backup (default: false) + +**Example:** +```typescript +const backup = await data.backup({ + includeVectors: true, + compress: true +}) +// Save backup to file +fs.writeFileSync('backup.json', JSON.stringify(backup)) +``` + +--- + +### `async restore(params): Promise` +Restores from backup. + +**Parameters:** +- `backup` (required) - Backup data +- `merge` - Merge with existing data +- `overwrite` - Overwrite conflicts +- `validate` - Validate backup format + +**Example:** +```typescript +const backup = JSON.parse(fs.readFileSync('backup.json')) +await data.restore({ + backup, + merge: false, + overwrite: true +}) +``` + +--- + +### `async clear(params): Promise` +Clears specified data types. + +**Parameters:** +- `entities` - Clear entities +- `relations` - Clear relationships +- `config` - Clear configuration + +**Example:** +```typescript +await data.clear({ + entities: true, + relations: true +}) +``` + +--- + +### `async import(params): Promise` +Imports data from various formats with automatic detection. + +**Parameters:** +- `data` (required) - Data to import (Buffer, string, array, or file path) +- `format` - 'auto' | 'json' | 'csv' | 'excel' | 'pdf' | 'yaml' | 'text' (default: 'auto') +- `mapping` - Field mapping configuration +- `batchSize` - Import batch size (default: 50) +- `validate` - Validate items before import +- `relationships` - Extract relationships automatically (default: true) + +**CSV-specific options:** +- `csvDelimiter` - Column delimiter (auto-detected if not specified) +- `csvHeaders` - First row contains headers (default: true) +- `encoding` - Character encoding (auto-detected if not specified) + +**Excel-specific options:** +- `excelSheets` - Sheet names array or 'all' for all sheets + +**PDF-specific options:** +- `pdfExtractTables` - Extract tables from PDFs (default: true) +- `pdfPreserveLayout` - Preserve text layout (default: true) + +**Examples:** +```typescript +// Import CSV file with auto-detection +const csvResult = await brain.import('customers.csv') +// Auto-detects: format, encoding, delimiter, field types + +// Import Excel workbook +const excelResult = await brain.import('sales-data.xlsx', { + excelSheets: ['Q1', 'Q2'] // Import specific sheets +}) + +// Import PDF with table extraction +const pdfResult = await brain.import('report.pdf', { + pdfExtractTables: true +}) + +// Import data array +const dataResult = await brain.import([ + { name: 'Alice', role: 'Engineer' }, + { name: 'Bob', role: 'Designer' } +], { + batchSize: 100, + relationships: true // Auto-extract relationships +}) + +// Import with custom CSV delimiter +const tsvResult = await brain.import('data.tsv', { + format: 'csv', + csvDelimiter: '\t' +}) +``` + +**Returns:** +```typescript +{ + success: boolean + imported: number // Number of items successfully imported + failed: number // Number of items that failed + entityIds: string[] // IDs of created entities + metadata: { + format: string // Detected format + encoding?: string // Detected encoding (CSV) + delimiter?: string // Detected delimiter (CSV) + sheets?: string[] // Processed sheets (Excel) + pageCount?: number // Number of pages (PDF) + } +} +``` + +--- + +### `async export(params?: ExportOptions): Promise` +Exports data in various formats. + +**Parameters:** +- `format` - 'json' | 'csv' | 'parquet' +- `filter` - Filter options +- `includeVectors` - Include embeddings + +**Example:** +```typescript +const exported = await data.export({ + format: 'csv', + where: { type: NounType.Document }, + includeVectors: false +}) +``` + +--- + +### `getStats(): StatsResult` +Gets complete statistics about entities and relationships. All stats are **O(1) pre-calculated** - updated when entities/relationships are added/removed. + +**Returns:** +```typescript +{ + entities: { + total: number // Total entity count + byType: Record // Entity count by type + } + relationships: { + totalRelationships: number // Total relationship/edge count + relationshipsByType: Record // Relationship count by type + uniqueSourceNodes: number // Number of unique source entities + uniqueTargetNodes: number // Number of unique target entities + totalNodes: number // Total unique entities in relationships + } + density: number // Relationships per entity ratio +} +``` + +**Example:** +```typescript +const stats = brain.getStats() + +// Total counts (O(1) operations) +const totalNouns = stats.entities.total +const totalVerbs = stats.relationships.totalRelationships +const totalRelations = stats.relationships.totalRelationships // alias + +// Counts by type (O(1) operations) +const nounTypes = stats.entities.byType +const verbTypes = stats.relationships.relationshipsByType + +// Graph metrics +console.log(`Entities: ${totalNouns}`) +console.log(`Relationships: ${totalVerbs}`) +console.log(`Density: ${stats.density.toFixed(2)}`) +console.log(`Types:`, Object.keys(nounTypes)) +``` + +**Performance:** +- ✅ All counts pre-calculated in memory +- ✅ O(1) access time +- ✅ Updated automatically on add/remove +- ✅ No expensive full scans required + +**Note:** For more granular counting operations, see the `brain.counts` API below. + +--- + +## Query API + +Access via: `brain.query` + +### `async entities(params?): Promise>` +Query entities with pagination. + +**Parameters:** +- `type` - Filter by type +- `where` - Metadata filters +- `limit` - Page size +- `cursor` - Pagination cursor +- `service` - Filter by service + +**Example:** +```typescript +const result = await brain.query.entities({ + type: NounType.Document, + where: { status: 'published' }, + limit: 50 +}) + +// Get next page +if (result.nextCursor) { + const nextPage = await brain.query.entities({ + cursor: result.nextCursor + }) +} +``` + +--- + +### `async relations(params?): Promise>` +Query relationships with pagination. + +**Parameters:** +- `from` - Source entity +- `to` - Target entity +- `type` - Relationship type +- `limit` - Page size +- `cursor` - Pagination cursor + +--- + +## Neural API + +Access via: `brain.neural()` + +### Similarity Operations + +#### `async similar(a, b, options?): Promise` +Calculates similarity between items. + +**Example:** +```typescript +const similarity = await neural.similar('doc1', 'doc2', { + explain: true +}) +``` + +--- + +### Clustering Operations + +#### `async clusters(options?): Promise` +General purpose clustering. + +**Parameters:** +- `algorithm` - 'hierarchical' | 'kmeans' | 'dbscan' | 'spectral' +- `k` - Number of clusters (for kmeans) +- `threshold` - Distance threshold +- `minPoints` - Minimum points per cluster + +**Example:** +```typescript +const clusters = await neural.clusters({ + algorithm: 'kmeans', + k: 5 +}) +``` + +--- + +#### `async clustersByDomain(params): Promise` +Domain-specific clustering. + +**Example:** +```typescript +const clusters = await neural.clustersByDomain({ + domain: 'technology', + field: 'category', + maxClusters: 10 +}) +``` + +--- + +### Outlier Detection + +#### `async outliers(options?): Promise` +Detects anomalous entities. + +**Parameters:** +- `method` - 'isolation' | 'lof' | 'statistical' | 'autoencoder' +- `threshold` - Outlier threshold (default: 2.5 std deviations) +- `returnScores` - Return anomaly scores + +**Example:** +```typescript +const outliers = await neural.outliers({ + method: 'isolation', + threshold: 3.0, + returnScores: true +}) +``` + +--- + +### Visualization + +#### `async visualize(options?): Promise` +Generates visualization data for entities. + +**Parameters:** +- `layout` - 'force' | 'circular' | 'hierarchical' | 'random' +- `dimensions` - 2D or 3D +- `includeEdges` - Include relationships + +**Example:** +```typescript +const vizData = await neural.visualize({ + layout: 'force', + dimensions: 3, + includeEdges: true +}) +``` + +--- + +## NLP API + +Access via: `brain.nlp()` + +### `async processNaturalQuery(query: string): Promise` +Converts natural language to structured query. + +**Example:** +```typescript +const structured = await nlp.processNaturalQuery( + "Find all documents about AI created last month" +) +// Returns structured query with type, time filters, etc. +``` + +--- + +### `async extract(text: string, options?): Promise` +Extracts entities from text using neural matching to NounTypes. + +**Parameters:** +- `types` - Target NounTypes to extract +- `confidence` - Minimum confidence (0-1) +- `includeVectors` - Include embeddings +- `neuralMatching` - Use neural type matching (default: true) + +**Returns:** Array of extracted entities with proper NounType classification + +**Example:** +```typescript +const entities = await nlp.extract( + "John Smith from Microsoft visited New York on Jan 15", + { + types: [NounType.Person, NounType.Organization, NounType.Location], + confidence: 0.7, + neuralMatching: true + } +) +// Returns: +// [ +// { text: "John Smith", type: NounType.Person, confidence: 0.92 }, +// { text: "Microsoft", type: NounType.Organization, confidence: 0.88 }, +// { text: "New York", type: NounType.Location, confidence: 0.85 } +// ] +``` + +--- + +### `async sentiment(text: string, options?): Promise` +Analyzes text sentiment. + +**Parameters:** +- `granularity` - 'document' | 'sentence' | 'aspect' +- `aspects` - Aspects to analyze + +**Returns:** +- `overall` - Document sentiment (-1 to 1) +- `sentences` - Sentence-level sentiment +- `aspects` - Aspect-based sentiment + +**Example:** +```typescript +const sentiment = await nlp.sentiment( + "The product quality is excellent but the price is too high", + { + granularity: 'aspect', + aspects: ['quality', 'price'] + } +) +// Returns: +// overall: { score: 0.2, label: 'mixed' } +// aspects: { +// quality: { score: 0.9, magnitude: 0.8 }, +// price: { score: -0.7, magnitude: 0.7 } +// } +``` + +--- + +## Streaming Pipeline API + +Access via: `brain.stream()` + +### Source Operations + +#### `source(generator): Pipeline` +Sets data source. + +**Example:** +```typescript +async function* dataGenerator() { + for (let i = 0; i < 100; i++) { + yield { id: i, data: `Item ${i}` } + } +} + +brain.stream() + .source(dataGenerator()) + .map(item => item.data) + .sink(console.log) + .run() +``` + +--- + +### Transform Operations + +#### `map(fn): Pipeline` +Transforms each item. + +#### `flatMap(fn): Pipeline` +Maps and flattens arrays. + +#### `filter(predicate): Pipeline` +Filters items. + +#### `tap(fn): Pipeline` +Side effects without modification. + +#### `retry(fn, maxRetries?, backoff?): Pipeline` +Retries failed operations. + +--- + +### Batching & Windowing + +#### `batch(size, timeoutMs?): Pipeline` +Groups items into batches. + +#### `window(size, type?): Pipeline` +Creates sliding or tumbling windows. + +#### `buffer(size, strategy?): Pipeline` +Buffers with backpressure handling. + +--- + +### Sink Operations + +#### `sink(handler): Pipeline` +Custom sink handler. + +#### `toBrainy(options?): Pipeline` +Sinks data to Brainy. + +**Example:** +```typescript +brain.stream() + .source(dataSource) + .map(transform) + .batch(100) + .toBrainy({ + type: NounType.Document, + metadata: { source: 'stream' } + }) + .run() +``` + +--- + +### Execution + +#### `run(options?): Promise` +Executes the pipeline. + +**Parameters:** +- `workers` - Number of workers or 'auto' +- `monitoring` - Enable monitoring +- `maxThroughput` - Rate limiting +- `backpressure` - 'drop' | 'buffer' | 'pause' +- `errorHandler` - Error callback + +--- + +## Type Definitions + +### Entity Types (NounType) +31 types including: +- `Person`, `Organization`, `Location` +- `Document`, `File`, `Message`, `Content` +- `Product`, `Service`, `Resource` +- `Event`, `Task`, `Project`, `Process` +- `User`, `Role`, `State` +- `Concept`, `Topic`, `Hypothesis` +- And more... + +### Relationship Types (VerbType) +40 types including: +- `RelatedTo`, `Contains`, `PartOf` +- `Creates`, `Modifies`, `Transforms` +- `DependsOn`, `Requires`, `Uses` +- `Owns`, `BelongsTo`, `MemberOf` +- `Supervises`, `ReportsTo`, `WorksWith` +- And more... + +### Core Interfaces + +```typescript +interface Entity { + id: string + vector: Vector + type: NounType + metadata?: T + service?: string + createdAt: number + updatedAt?: number +} + +interface Relation { + id: string + from: string + to: string + type: VerbType + weight?: number + metadata?: T + service?: string + createdAt: number +} + +interface Result { + id: string + score: number + entity: Entity + explanation?: ScoreExplanation +} +``` + +--- + +## Performance Characteristics + +### Speed Benchmarks +- **Add entity**: ~5-10ms (with embedding) +- **Vector search**: ~1-5ms for 1M entities +- **Graph traversal**: ~10-20ms (2-hop) +- **Batch operations**: 1000+ items/second + +### Scalability +- **Entities**: Tested to 10M+ +- **Relationships**: Tested to 100M+ +- **Concurrent operations**: 1000+ parallel +- **Memory usage**: ~1GB per million entities + +### Storage Requirements +- **Per entity**: ~2KB (with vector) +- **Per relationship**: ~200 bytes +- **Indexes**: ~20% overhead + +--- + +## Best Practices + +### 1. Always Initialize +```typescript +const brain = new Brainy() +await brain.init() // Required before operations +``` + +### 2. Use Proper Types +```typescript +// ✅ Good - specific type +await brain.add({ data: 'John', type: NounType.Person }) + +// ❌ Bad - generic type +await brain.add({ data: 'John', type: NounType.Thing }) +``` + +### 3. Batch When Possible +```typescript +// ✅ Good - batch operation +await brain.addMany({ items: documents }) + +// ❌ Bad - individual adds in loop +for (const doc of documents) { + await brain.add(doc) // Slow! +} +``` + +### 4. Use Write-Only for Speed +```typescript +// For high-speed ingestion +await brain.add({ + data: content, + type: NounType.Document, + writeOnly: true // Skip validation +}) +``` + +### 5. Clean Up Resources +```typescript +try { + // Your operations +} finally { + await brain.close() // Always close +} +``` + +--- + +## Input Validation + +Brainy uses a **zero-config validation system** that automatically adapts to your system resources: + +### Auto-Configured Limits +- `limit` parameter maximum: Based on available memory (e.g., 8GB RAM = 80K max limit) +- Query string length: Auto-scaled based on memory +- Vector dimensions: Must be exactly 384 for all-MiniLM-L6-v2 model + +### Common Validation Rules +- **Pagination**: `limit` and `offset` must be non-negative +- **Thresholds**: Values like `weight` and `threshold` must be between 0 and 1 +- **Mutual Exclusion**: Cannot use both `query` and `vector` in same request +- **Type Safety**: `NounType` and `VerbType` must be valid enum values +- **Self-Reference**: Cannot create relationships from an entity to itself + +### Performance Auto-Tuning +The validation system monitors query performance and adjusts limits automatically: +- Fast queries with large results → increases limits +- Slow queries → reduces limits to maintain performance + +## Error Handling + +All methods validate parameters and throw descriptive errors: + +```typescript +try { + await brain.add({ data: '', type: NounType.Document }) +} catch (error) { + // Error: "must provide either data or vector" +} + +try { + await brain.find({ limit: -1 }) +} catch (error) { + // Error: "limit must be non-negative" +} + +try { + await brain.update({ id: 'xyz', metadata: null, merge: false }) +} catch (error) { + // Error: "must specify at least one field to update" + // Note: Use metadata: {} to clear, not null +} +``` + +--- + +## Migration from v2 + +### Old (v2.15) +```typescript +brain.add({ data, type: NounType.Document, metadata }) +brain.find({ query, limit: 10 }) +brain.insights() +``` + +### New (v3.0) +```typescript +brain.add({ data, type: NounType.Document, metadata }) +brain.find({ query, limit: 10 }) +brain.insights() +``` + +--- + +## Support + +- **GitHub**: https://github.com/soulcraft/brainy +- **NPM**: https://www.npmjs.com/package/@soulcraft/brainy +- **Discord**: https://discord.gg/brainy + +--- + +*Brainy v3.0 - The Neural Database That Thinks* \ No newline at end of file diff --git a/docs/BATCHING.md b/docs/BATCHING.md deleted file mode 100644 index e70a9e18..00000000 --- a/docs/BATCHING.md +++ /dev/null @@ -1,468 +0,0 @@ ---- -title: Batch Operations -slug: guides/batching -public: true -category: guides -template: guide -order: 5 -description: Eliminate N+1 query patterns with batchGet() and storage-level batch APIs for fast multi-entity reads against filesystem and memory storage. -next: - - api/reference - - guides/find-system ---- - -# Batch Operations API -> **Production-Ready** | Zero N+1 Query Patterns - -## Overview - -Brainy provides batch operations at the storage layer to eliminate N+1 query patterns for VFS operations, relationship queries, and entity retrieval. - -### Problem Solved - -The naive pattern of looping and calling `brain.get(id)` once per item issues sequential reads through the storage layer. Batched APIs collapse that into a single read pass. - -**IMPORTANT:** The batch optimizations apply **ONLY to `getTreeStructure()`** at the VFS layer and the explicit `batchGet()` / `getNounMetadataBatch()` calls — not to `readFile()` or individual `get()` operations. - ---- - -## New Public APIs - -### 1. `brain.batchGet(ids, options?)` - -Batch retrieval of multiple entities (metadata-only by default). - -```typescript -// Fetch multiple entities in a single batched operation -const ids = ['id1', 'id2', 'id3'] -const results: Map = await brain.batchGet(ids) - -// With vectors (falls back to individual gets) -const resultsWithVectors = await brain.batchGet(ids, { includeVectors: true }) - -// Results map -results.get('id1') // → Entity or undefined -results.size // → 3 (number of found entities) -``` - -**Performance:** -- Memory storage: Instant (parallel reads) -- Filesystem storage: Parallel reads, scales with available IOPS - -**Use Cases:** -- Loading multiple entities for display -- Bulk data export operations -- Relationship traversal (fetch all connected entities) - ---- - -## Storage-Level APIs - -### 2. `storage.getNounMetadataBatch(ids)` - -Batch metadata retrieval with direct O(1) path construction. - -```typescript -const storage = brain.storage as BaseStorage -const ids = ['id1', 'id2', 'id3'] - -const metadataMap: Map = await storage.getNounMetadataBatch(ids) - -for (const [id, metadata] of metadataMap) { - console.log(metadata.noun) // Type: 'document', 'person', etc. - console.log(metadata.data) // Entity data -} -``` - -**Features:** -- ✅ Direct O(1) path construction from ID (no type lookup needed!) -- ✅ Sharding preservation (all paths include `{shard}/{id}`) -- ✅ Write-cache coherent (read-after-write consistency) -- ✅ O(1) path construction — eliminates the per-entity type search of the old type-first layout - -**Performance:** -- Constant-time path construction per ID — no type-cache misses -- Filesystem: parallel reads bounded by IOPS -- No type search delays — every ID maps directly to storage path - ---- - -### 3. `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - -Batch relationship queries by source entity IDs. - -```typescript -const storage = brain.storage as BaseStorage - -// Get all relationships from multiple sources -const results: Map = await storage.getVerbsBySourceBatch([ - 'person1', - 'person2' -]) - -// Filter by verb type -const createsResults = await storage.getVerbsBySourceBatch( - ['person1', 'person2'], - 'creates' -) - -// Process results -for (const [sourceId, verbs] of results) { - console.log(`${sourceId} has ${verbs.length} relationships`) - verbs.forEach(verb => { - console.log(` → ${verb.verb} → ${verb.targetId}`) - }) -} -``` - -**Use Cases:** -- Social graph traversal (fetch all connections for multiple users) -- Knowledge graph queries (find all relationships of specific type) -- Bulk export of relationship data - -**Performance:** -- Memory storage: single in-memory pass over the metadata index -- Filesystem storage: parallel reads through the metadata index - ---- - -## VFS Integration - -VFS operations automatically use batch APIs for maximum performance. - -### Directory Traversal - -```typescript -// Tree traversal uses batched reads under the hood -const tree = await brain.vfs.getTreeStructure('/my-dir') -// ✅ PathResolver.getChildren() uses brain.batchGet() internally -// ✅ Parallel traversal of directories at the same tree level -// ✅ 2-3 batched calls instead of 22 sequential calls -``` - -**Architecture:** - -``` -VFS.getTreeStructure() - ↓ PARALLEL (breadth-first traversal) - → PathResolver.getChildren() [all dirs at level processed in parallel] - ↓ BATCHED - → brain.batchGet(childIds) [1 call instead of N] - ↓ BATCHED - → storage.getNounMetadataBatch(ids) [1 call instead of N] - ↓ ADAPTER - → Filesystem: Promise.all() parallel reads - → Memory: Promise.all() parallel reads -``` - ---- - -## Advanced Features Compatibility - -### ✅ ID-First Storage Architecture - -All batch operations use direct ID-first paths - no type lookup needed! - -**ID-First Path Structure:** -``` -entities/nouns/{SHARD}/{ID}/metadata.json -entities/verbs/{SHARD}/{ID}/metadata.json -``` - -**Direct O(1) Path Construction:** -```typescript -// Every ID maps directly to exactly ONE path - O(1), no type search -const id = 'abc-123' -const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars) -const path = `entities/nouns/${shard}/${id}/metadata.json` - -// No type cache needed! -// No type search needed! -// No multi-type fallback needed! -// Just pure O(1) lookup! -``` - -**Benefits:** -- **O(1)** path lookups (eliminates the 42-type sequential search the old type-first layout required) -- **Simpler code** - removed 500+ lines of type cache complexity -- **Scalable** - works at large scale without type tracking overhead - ---- - -### ✅ Sharding - -All batch paths include shard IDs calculated via `getShardIdFromUuid(id)`: - -```typescript -const id = 'a3c4e5f7-...' -const shard = getShardIdFromUuid(id) // → 'a3' (first 2 hex chars) -const path = `entities/nouns/${shard}/${id}/metadata.json` -``` - -**Distribution:** 256 shards (00-ff) for optimal load distribution. - ---- - -### ✅ Generational MVCC (8.0) - -Batch reads always serve the **live** generation through the fast paths -shown above. Point-in-time reads go through the Db API instead: a pinned -`Db` (`brain.now()`, `brain.asOf()`) resolves changed ids from immutable -generation records and unchanged ids from the same live paths batch reads -use — see the [consistency model](concepts/consistency-model.md). - -```typescript -const db = brain.now() // pinned view -const entity = await db.get(id) // correct at the pinned generation -const results = await brain.batchGet(ids) // live state, batched -await db.release() -``` - ---- - -## Why Batching Is Faster - -Batching's advantage is structural, not a fixed multiplier (the actual speedup -depends on storage backend, IOPS, and batch size): - -- **N+1 elimination** — N sequential reads collapse into a single parallel pass - (`Promise.all` over the batch). -- **O(1) path construction** — every ID maps directly to one storage path, with - no per-type cache lookup. -- **One metadata round-trip** — relationship batches fetch all sources' metadata - in a single pass instead of one query per source. - -The integration test `tests/integration/storage-batch-operations.test.ts` -exercises batch vs. individual reads and asserts that batch retrieval is not -slower than the per-entity loop for large batches; it does not pin a specific -multiplier, since that is hardware- and IOPS-dependent. - ---- - -## Error Handling - -### Partial Batch Failures - -Batch operations gracefully handle missing or invalid entities: - -```typescript -const validId = 'abc-123-...' -const invalidIds = [ - '11111111-1111-1111-1111-111111111111', - '22222222-2222-2222-2222-222222222222' -] - -const results = await brain.batchGet([validId, ...invalidIds]) - -results.size // → 1 (only valid entity) -results.has(validId) // → true -results.has(invalidIds[0]) // → false (silently skipped) -``` - -**Behavior:** -- Invalid UUIDs: Silently skipped (not included in results) -- Missing entities: Silently skipped (not included in results) -- Storage errors: Logged, entity excluded from results -- No exceptions thrown for partial failures - -### Empty Batches - -```typescript -const results = await brain.batchGet([]) -results.size // → 0 (empty map) -``` - -### Duplicate IDs - -```typescript -const results = await brain.batchGet(['id1', 'id1', 'id1']) -results.size // → 1 (deduplicated automatically) -``` - ---- - -## Migration Guide - -### From Individual Gets - -**Before:** -```typescript -const entities = [] -for (const id of ids) { - const entity = await brain.get(id) - if (entity) entities.push(entity) -} -``` - -**After:** -```typescript -const results = await brain.batchGet(ids) -const entities = Array.from(results.values()) -``` - -**Performance Gain:** Replaces N sequential reads with a single batched pass — no fixed multiplier, it scales with storage IOPS. - ---- - -### From Individual Relationship Queries - -**Before:** -```typescript -const allVerbs = [] -for (const sourceId of sourceIds) { - const verbs = await brain.related({ from: sourceId }) - allVerbs.push(...verbs) -} -``` - -**After:** -```typescript -const storage = brain.storage as BaseStorage -const results = await storage.getVerbsBySourceBatch(sourceIds) - -const allVerbs = [] -for (const verbs of results.values()) { - allVerbs.push(...verbs) -} -``` - -**Performance Gain:** One batched metadata fetch instead of one query per source entity. - ---- - -## Best Practices - -### 1. **Use Batching for Multiple Entity Operations** - -```typescript -// ✅ GOOD: Batch fetch -const results = await brain.batchGet(ids) - -// ❌ BAD: Individual gets in loop -for (const id of ids) { - await brain.get(id) -} -``` - -### 2. **Batch Size Recommendations** - -| Storage | Optimal Batch Size | Max Batch Size | -|---------|--------------------|----------------| -| **Memory** | Unlimited | Unlimited | -| **Filesystem** | 100-500 | 1000 | - -**Guideline:** For batches >1000, split into chunks of 500-1000. - -### 3. **Metadata-Only by Default** - -```typescript -// Default: Metadata-only (fast) -const results = await brain.batchGet(ids) // No vectors - -// Only load vectors if needed -const withVectors = await brain.batchGet(ids, { includeVectors: true }) -``` - -### 4. **Error Handling** - -```typescript -// Batch operations never throw for missing entities -const results = await brain.batchGet(ids) - -// Check results -for (const id of ids) { - if (results.has(id)) { - // Entity exists - const entity = results.get(id) - } else { - // Entity missing (not an error) - console.log(`Entity ${id} not found`) - } -} -``` - ---- - -## Testing - -Comprehensive test coverage in `tests/integration/storage-batch-operations.test.ts`: - -```bash -npx vitest run tests/integration/storage-batch-operations.test.ts -``` - -**Test Coverage:** -- ✅ brain.batchGet() high-level API -- ✅ storage.getNounMetadataBatch() with ID-first paths -- ✅ COW integration (branch isolation, inheritance) -- ✅ storage.getVerbsBySourceBatch() relationship queries -- ✅ VFS integration (PathResolver.getChildren()) -- ✅ Performance benchmarks (N+1 elimination) -- ✅ Error handling (partial failures, empty batches, duplicates) -- ✅ ID-first storage verification -- ✅ Sharding preservation - -**Results:** 23 tests passing ✅ - ---- - -## Implementation Details - -### Architecture Layers - -``` -User Code (brain.batchGet) - ↓ -High-Level API (src/brainy.ts) - ↓ -Storage Layer (src/storage/baseStorage.ts) - ↓ -Adapter Layer (readBatchFromAdapter) - ↓ -Storage Adapter (FileSystemStorage / MemoryStorage) -``` - -### Parallel Reads - -Both shipped adapters fall back to `Promise.all` over individual reads: - -```typescript -// BaseStorage.readBatchFromAdapter() -return await Promise.all(resolvedPaths.map(path => this.read(path))) -``` - -**Shipped Adapters:** -- MemoryStorage -- FileSystemStorage - ---- - -## API Summary - -- `brain.batchGet(ids, options?)` - High-level batch entity retrieval -- `storage.getNounMetadataBatch(ids)` - Storage-level metadata batch -- `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries - -**Performance Improvements:** -- VFS operations: single batched pass instead of N sequential reads -- Entity retrieval: N+1 reads collapsed into one batched pass -- Zero N+1 query patterns - -**Compatibility:** -- ✅ ID-first storage -- ✅ Sharding (256 shards) -- ✅ Generational MVCC — batch reads serve the live generation; pinned `Db` views serve the past -- ✅ All indexes respected (vector, metadata, graph adjacency) - ---- - -## Support - -- **Documentation:** `/docs/BATCHING.md`, `/docs/PERFORMANCE.md` -- **Tests:** `/tests/integration/storage-batch-operations.test.ts` -- **Issues:** https://github.com/soulcraft/brainy/issues -- **Discussions:** https://github.com/soulcraft/brainy/discussions - ---- - -**Built with ❤️ for enterprise-scale knowledge graphs** diff --git a/docs/CORE_API_PATTERNS.md b/docs/CORE_API_PATTERNS.md new file mode 100644 index 00000000..717686c8 --- /dev/null +++ b/docs/CORE_API_PATTERNS.md @@ -0,0 +1,643 @@ +# 🧠 Core API Patterns: Modern Brainy v3.x + +> Learn the correct patterns for Brainy's core operations. Avoid v2.x confusion and use modern, efficient APIs. + +## 🚨 Critical: Use v3.x APIs Only + +### ❌ **WRONG - Deprecated v2.x APIs** + +```typescript +// DON'T DO THIS - These methods don't exist in v3.x! +await brain.addNoun(text, type, metadata) // ❌ Removed +await brain.getNouns({ pagination }) // ❌ Removed +await brain.addVerb(source, target, type) // ❌ Removed +await brain.getVerbs() // ❌ Removed +await brain.deleteNoun(id) // ❌ Removed +await brain.deleteVerb(id) // ❌ Removed +``` + +### ✅ **CORRECT - Modern v3.x APIs** + +```typescript +// ✅ Use these modern methods instead +await brain.add({ data, type, metadata }) // Modern unified add +await brain.find({ limit: 100 }) // Natural language search +await brain.relate({ from, to, type }) // Clean relationship creation +await brain.getRelations() // Modern relationship queries +await brain.delete(id) // Unified deletion +// Relationships auto-cascade when entities are deleted +``` + +## 📋 Entity Management Patterns + +### ❌ **WRONG - v2.x Style** + +```typescript +// DON'T DO THIS - Old API patterns +import { Brainy } from 'old-brainy' // ❌ Wrong import + +const brain = new Brainy({ // ❌ Old class name + complexConfig: true +}) + +const id = await brain.addNoun( // ❌ Deprecated method + "John Smith is a developer", + "Person", + { role: "engineer" } +) +``` + +### ✅ **CORRECT - Modern Patterns** + +```typescript +// ✅ Pattern 1: Basic entity creation +import { Brainy, NounType } from '@soulcraft/brainy' + +const brain = new Brainy() // ✅ Zero config +await brain.init() + +const id = await brain.add({ + data: "John Smith is a developer", + type: NounType.Person, + metadata: { role: "engineer", team: "backend" } +}) + +// ✅ Pattern 2: Bulk entity creation +const entities = [ + { data: "React framework", type: NounType.Technology }, + { data: "Vue.js framework", type: NounType.Technology }, + { data: "Angular framework", type: NounType.Technology } +] + +const ids = await Promise.all( + entities.map(entity => brain.add(entity)) +) + +// ✅ Pattern 3: Entity with pre-computed vector +const customVector = await brain.embed("Custom text") +const vectorId = await brain.add({ + data: "Optimized content", + type: NounType.Document, + vector: customVector, // Skip re-embedding + metadata: { source: "api", optimized: true } +}) +``` + +## 🔍 Search & Discovery Patterns + +### ❌ **WRONG - Confusing Old Patterns** + +```typescript +// DON'T DO THIS - Mixing old and new APIs +const results1 = await brain.searchText("query") // ❌ Old method +const results2 = await brain.getNouns({ filter }) // ❌ Doesn't exist +const results3 = await brain.findSimilar(text) // ❌ Unclear naming +``` + +### ✅ **CORRECT - Clean Search Patterns** + +```typescript +// ✅ Pattern 1: Natural language search +const results = await brain.find("React developers working on authentication") + +// ✅ Pattern 2: Structured search with filters +const filteredResults = await brain.find({ + like: "machine learning", // Vector similarity + where: { // Metadata filtering + type: NounType.Document, + year: { $gte: 2020 }, + status: "published" + }, + limit: 50, + orderBy: 'relevance' +}) + +// ✅ Pattern 3: Similarity search +const similarItems = await brain.similar({ + to: existingEntityId, // Find items similar to this + threshold: 0.8, // Minimum similarity + limit: 10, + exclude: [existingEntityId] // Don't include the source +}) + +// ✅ Pattern 4: Advanced search with relationships +const connectedResults = await brain.find({ + like: "frontend frameworks", + connected: { + to: reactId, // Connected to React + via: "related-to", // Through this relationship + depth: 2 // Up to 2 hops away + } +}) +``` + +## 🔗 Relationship Patterns + +### ❌ **WRONG - Old Relationship APIs** + +```typescript +// DON'T DO THIS - Old relationship patterns +await brain.addVerb(sourceId, targetId, "uses", { strength: 0.9 }) // ❌ Old API +const verbs = await brain.getVerbsBySource(sourceId) // ❌ Removed +await brain.deleteVerb(verbId) // ❌ Old pattern +``` + +### ✅ **CORRECT - Modern Relationship Management** + +```typescript +// ✅ Pattern 1: Create relationships +const relationId = await brain.relate({ + from: developerId, + to: frameworkId, + type: VerbType.Uses, + metadata: { + since: "2023-01-01", + proficiency: "expert", + hours_per_week: 40 + } +}) + +// ✅ Pattern 2: Query relationships +const relationships = await brain.getRelations({ + from: developerId, // Relationships from this entity + type: VerbType.Uses, // Of this type + limit: 100 +}) + +// ✅ Pattern 3: Bidirectional relationships +await brain.relate({ + from: projectId, + to: developerId, + type: VerbType.AssignedTo, + bidirectional: true, // Creates reverse relationship + metadata: { role: "lead", start_date: "2024-01-01" } +}) + +// ✅ Pattern 4: Relationship-based discovery +const collaborators = await brain.find({ + connected: { + to: currentProjectId, + via: VerbType.WorksOn, + direction: "incoming" // Who works on this project + } +}) +``` + +## 🗃️ Data Retrieval Patterns + +### ❌ **WRONG - Inefficient Patterns** + +```typescript +// DON'T DO THIS - Loading everything +const everything = await brain.getNouns({ limit: 1000000 }) // ❌ Crashes +const allData = await brain.exportAll() // ❌ Memory explosion +``` + +### ✅ **CORRECT - Efficient Data Access** + +```typescript +// ✅ Pattern 1: Paginated retrieval +async function getAllEntitiesPaginated() { + const pageSize = 100 + let offset = 0 + let allEntities = [] + + while (true) { + const page = await brain.find({ + limit: pageSize, + offset: offset + }) + + if (page.length === 0) break + + allEntities.push(...page) + offset += pageSize + + // Optional: Progress reporting + console.log(`Loaded ${allEntities.length} entities...`) + } + + return allEntities +} + +// ✅ Pattern 2: Streaming large datasets +async function* streamEntities() { + const pageSize = 50 + let offset = 0 + + while (true) { + const page = await brain.find({ + limit: pageSize, + offset: offset + }) + + if (page.length === 0) break + + for (const entity of page) { + yield entity + } + + offset += pageSize + } +} + +// Usage +for await (const entity of streamEntities()) { + await processEntity(entity) +} + +// ✅ Pattern 3: Specific entity retrieval +const entity = await brain.get(entityId) +if (entity) { + console.log('Entity data:', entity.data) + console.log('Metadata:', entity.metadata) +} else { + console.log('Entity not found') +} +``` + +## 🔄 Update & Delete Patterns + +### ❌ **WRONG - Manual Update Patterns** + +```typescript +// DON'T DO THIS - Recreating entities +await brain.delete(oldId) +const newId = await brain.add(updatedData) // ❌ Loses relationships +``` + +### ✅ **CORRECT - Update Operations** + +```typescript +// ✅ Pattern 1: Update entity data +await brain.update(entityId, { + data: "Updated content here", + metadata: { + lastModified: Date.now(), + version: "2.0" + } +}) + +// ✅ Pattern 2: Partial metadata updates +await brain.updateMetadata(entityId, { + status: "published", + tags: ["important", "featured"] + // Merges with existing metadata +}) + +// ✅ Pattern 3: Safe deletion with cascade options +await brain.delete(entityId, { + cascade: true, // Delete related relationships + backup: true // Create backup before deletion +}) + +// ✅ Pattern 4: Bulk operations +const updateOperations = entities.map(entity => ({ + id: entity.id, + changes: { status: "processed" } +})) + +await brain.updateMany(updateOperations) +``` + +## 🧮 Vector & Embedding Patterns + +### ❌ **WRONG - Manual Vector Handling** + +```typescript +// DON'T DO THIS - Manual embedding without understanding +const vector = await brain.embed(text) +// Store vector somewhere manually // ❌ Missing integration +``` + +### ✅ **CORRECT - Smart Vector Operations** + +```typescript +// ✅ Pattern 1: Automatic embedding (recommended) +const id = await brain.add({ + data: "Content to be embedded", + type: NounType.Document + // Vector computed automatically +}) + +// ✅ Pattern 2: Pre-computed vectors for optimization +const texts = ["Text 1", "Text 2", "Text 3"] +const vectors = await Promise.all( + texts.map(text => brain.embed(text)) +) + +const entities = await Promise.all( + texts.map((text, i) => brain.add({ + data: text, + type: NounType.Document, + vector: vectors[i] // Skip re-embedding + })) +) + +// ✅ Pattern 3: Vector similarity search +const queryVector = await brain.embed("search query") +const similar = await brain.similar({ + vector: queryVector, // Use vector directly + threshold: 0.75, + limit: 20 +}) + +// ✅ Pattern 4: Compare vectors directly +const vector1 = await brain.embed("First text") +const vector2 = await brain.embed("Second text") +const similarity = brain.computeSimilarity(vector1, vector2) +console.log(`Similarity: ${similarity}`) +``` + +## 🏗️ Configuration Patterns + +### ❌ **WRONG - Over-Configuration** + +```typescript +// DON'T DO THIS - Complex configurations that break +const brain = new Brainy({ + storage: { + type: 'complex', + options: { + nested: { + configuration: true, + that: "breaks" + } + } + }, + embedding: { + customModel: "broken-model", + dimensions: 999999 + } +}) +``` + +### ✅ **CORRECT - Smart Configuration** + +```typescript +// ✅ Pattern 1: Zero configuration (recommended) +const brain = new Brainy() // Auto-detects everything +await brain.init() + +// ✅ Pattern 2: Simple storage selection +const fsBrain = new Brainy({ + storage: { type: 'filesystem', path: './data' } +}) + +const cloudBrain = new Brainy({ + storage: { type: 's3', bucket: 'my-data' } +}) + +// ✅ Pattern 3: Production configuration +const prodBrain = new Brainy({ + storage: { + type: 's3', + bucket: process.env.BRAINY_BUCKET, + region: process.env.AWS_REGION + }, + silent: true, // No console output + distributed: true, // Enable clustering + cache: { maxSize: 10000 } // Larger cache +}) + +// ✅ Pattern 4: Development vs production +const isDev = process.env.NODE_ENV === 'development' + +const brain = new Brainy({ + storage: isDev + ? { type: 'memory' } // Fast for dev + : { type: 'filesystem', path: './brainy-data' }, // Persistent for prod + silent: !isDev, // Verbose in dev, quiet in prod + cache: { maxSize: isDev ? 100 : 5000 } +}) +``` + +## 🔄 Migration from v2.x + +### ✅ **Migration Patterns** + +```typescript +// If you have old v2.x code, here's how to migrate: + +// OLD v2.x: +// await brain.addNoun(text, type, metadata) +// NEW v3.x: +await brain.add({ data: text, type, metadata }) + +// OLD v2.x: +// await brain.getNouns({ pagination: { limit: 100 } }) +// NEW v3.x: +await brain.find({ limit: 100 }) + +// OLD v2.x: +// await brain.addVerb(sourceId, targetId, verbType, metadata) +// NEW v3.x: +await brain.relate({ from: sourceId, to: targetId, type: verbType, metadata }) + +// OLD v2.x: +// await brain.searchText(query) +// NEW v3.x: +await brain.find(query) // More powerful natural language search +``` + +## 🚀 Performance Patterns + +### ✅ **High-Performance Patterns** + +```typescript +// ✅ Pattern 1: Batch operations +const entities = [/* large array */] +const batchSize = 100 + +for (let i = 0; i < entities.length; i += batchSize) { + const batch = entities.slice(i, i + batchSize) + await Promise.all( + batch.map(entity => brain.add(entity)) + ) + + // Optional: Rate limiting + await new Promise(resolve => setTimeout(resolve, 100)) +} + +// ✅ Pattern 2: Connection pooling for distributed +const brain = new Brainy({ + distributed: true, + connectionPool: { + min: 5, + max: 50, + acquireTimeoutMillis: 30000 + } +}) + +// ✅ Pattern 3: Efficient caching +const brain = new Brainy({ + cache: { + maxSize: 10000, // Number of items + ttl: 300000, // 5 minutes + updateAgeOnGet: true // LRU behavior + } +}) + +// ✅ Pattern 4: Memory-conscious operations +const results = await brain.find({ + query: "large dataset query", + limit: 1000, // Reasonable limit + includeVectors: false // Exclude vectors if not needed +}) +``` + +## 🛡️ Error Handling Patterns + +### ✅ **Robust Error Handling** + +```typescript +// ✅ Pattern 1: Specific error handling +try { + const result = await brain.add({ data, type, metadata }) + return result +} catch (error) { + if (error.code === 'DUPLICATE_ENTITY') { + console.log('Entity already exists, updating instead...') + return await brain.update(error.existingId, { data, metadata }) + } else if (error.code === 'STORAGE_FULL') { + throw new Error('Storage capacity exceeded') + } else if (error.code === 'EMBEDDING_FAILED') { + console.warn('Embedding failed, retrying with simpler text...') + return await brain.add({ + data: data.substring(0, 1000), // Truncate + type, + metadata + }) + } + throw error +} + +// ✅ Pattern 2: Retry with exponential backoff +async function resilientAdd(data: any, maxRetries = 3) { + for (let attempt = 1; attempt <= maxRetries; attempt++) { + try { + return await brain.add(data) + } catch (error) { + if (attempt === maxRetries) throw error + + const delay = Math.pow(2, attempt) * 1000 + console.warn(`Attempt ${attempt} failed, retrying in ${delay}ms...`) + await new Promise(resolve => setTimeout(resolve, delay)) + } + } +} + +// ✅ Pattern 3: Graceful degradation +async function robustSearch(query: string) { + try { + // Try advanced semantic search first + return await brain.find({ + like: query, + threshold: 0.8, + limit: 50 + }) + } catch (error) { + console.warn('Semantic search failed, falling back to basic search:', error.message) + + try { + // Fallback to simple text search + return await brain.find(query) + } catch (fallbackError) { + console.error('All search methods failed:', fallbackError.message) + return [] // Return empty results rather than crash + } + } +} +``` + +## 📊 Monitoring Patterns + +### ✅ **Production Monitoring** + +```typescript +// ✅ Pattern 1: Performance monitoring +const startTime = Date.now() +const result = await brain.add(data) +const duration = Date.now() - startTime + +if (duration > 1000) { + console.warn(`Slow add operation: ${duration}ms`) +} + +// ✅ Pattern 2: Health checks +async function healthCheck() { + try { + // Test basic operations + const testId = await brain.add({ + data: "health check", + type: NounType.System, + metadata: { test: true } + }) + + await brain.get(testId) + await brain.delete(testId) + + return { status: 'healthy', timestamp: Date.now() } + } catch (error) { + return { + status: 'unhealthy', + error: error.message, + timestamp: Date.now() + } + } +} + +// ✅ Pattern 3: Metrics collection +class BrainyMetrics { + private metrics = { + operations: 0, + errors: 0, + totalTime: 0 + } + + async timedOperation(operation: () => Promise): Promise { + const start = Date.now() + try { + const result = await operation() + this.metrics.operations++ + this.metrics.totalTime += Date.now() - start + return result + } catch (error) { + this.metrics.errors++ + throw error + } + } + + getStats() { + return { + ...this.metrics, + avgTime: this.metrics.totalTime / this.metrics.operations || 0, + errorRate: this.metrics.errors / this.metrics.operations || 0 + } + } +} +``` + +## 🎯 Summary: Modern Brainy v3.x Best Practices + +| ❌ **Avoid v2.x** | ✅ **Use v3.x** | +|------------------|----------------| +| `addNoun()` | `add()` | +| `getNouns()` | `find()` | +| `addVerb()` | `relate()` | +| `getVerbs()` | `getRelations()` | +| `deleteNoun()` | `delete()` | +| Complex configs | Zero-config with `new Brainy()` | +| Manual pagination | Built-in smart pagination | +| String-based search | Natural language queries | + +--- + +**🎉 Following these patterns gives you:** +- 🚀 **Modern APIs** that are actively maintained +- ⚡ **Better performance** with intelligent defaults +- 🛡️ **Robust error handling** with specific error types +- 📈 **Scalable patterns** for production applications +- 🧠 **Natural language** search capabilities + +**Next:** [Neural API Patterns →](./NEURAL_API_PATTERNS.md) | [VFS Patterns →](./vfs/COMMON_PATTERNS.md) \ No newline at end of file diff --git a/docs/CREATING-AUGMENTATIONS.md b/docs/CREATING-AUGMENTATIONS.md new file mode 100644 index 00000000..1c7fd04a --- /dev/null +++ b/docs/CREATING-AUGMENTATIONS.md @@ -0,0 +1,435 @@ +# Creating Augmentations for Brainy + +> **Updated for v4.0.0** - Includes metadata structure changes and type system improvements + +## The BrainyAugmentation Interface + +Every augmentation implements this simple yet powerful interface: + +```typescript +interface BrainyAugmentation { + // Identification + name: string // Unique name for your augmentation + + // Execution control + timing: 'before' | 'after' | 'around' | 'replace' // When to execute + operations: string[] // Which operations to intercept + priority: number // Execution order (higher = first) + + // Lifecycle methods + initialize(context: AugmentationContext): Promise + execute(operation: string, params: any, next: () => Promise): Promise + shutdown?(): Promise // Optional cleanup +} +``` + +## v4.0.0 Breaking Changes for Augmentation Developers + +### 1. Metadata Structure Separation +v4.0.0 introduces strict metadata/vector separation for billion-scale performance: + +```typescript +// ✅ v4.0.0: Metadata has required type field +interface NounMetadata { + noun: NounType // Required! Must be a valid noun type + [key: string]: any // Your custom metadata +} + +interface VerbMetadata { + verb: VerbType // Required! Must be a valid verb type + sourceId: string + targetId: string + [key: string]: any +} +``` + +### 2. Storage Adapter Return Types +Storage adapters now return different types at different boundaries: + +```typescript +// Internal methods: Pure structures (no metadata) +abstract _getNoun(id: string): Promise + +// Public API: WithMetadata structures +abstract getNoun(id: string): Promise +``` + +### 3. Verb Property Renamed +The verb relationship field changed from `type` to `verb`: + +```typescript +// ❌ v3.x +verb.type === 'relatedTo' + +// ✅ v4.0.0 +verb.verb === 'relatedTo' +``` + +## Creating a Storage Augmentation + +Storage augmentations are special - they provide the storage backend for Brainy. + +### Important: v4.0.0 Storage Requirements + +Your storage adapter MUST: +1. **Wrap metadata** with required `noun`/`verb` fields +2. **Return pure structures** from internal `_methods` +3. **Return WithMetadata types** from public methods + +```typescript +import { StorageAugmentation } from 'brainy/augmentations' +import { BaseStorageAdapter, HNSWNoun, HNSWNounWithMetadata, NounMetadata } from 'brainy' + +export class MyCustomStorage extends BaseStorageAdapter { + // Internal method: Returns pure structure + async _getNoun(id: string): Promise { + const data = await this.fetchFromDatabase(id) + return data ? { + id: data.id, + vector: data.vector, + nounType: data.type + } : null + } + + // Public method: Returns WithMetadata structure + async getNoun(id: string): Promise { + const noun = await this._getNoun(id) + if (!noun) return null + + // Fetch metadata separately (v4.0.0 pattern) + const metadata = await this.getNounMetadata(id) + + return { + ...noun, + metadata: metadata || { noun: noun.nounType || 'thing' } + } + } + + // CRITICAL: Always save with proper metadata structure + async saveNoun(noun: HNSWNoun, metadata?: NounMetadata): Promise { + // Validate metadata has required 'noun' field + if (!metadata?.noun) { + throw new Error('v4.0.0: NounMetadata requires "noun" field') + } + + await this.database.save({ + id: noun.id, + vector: noun.vector, + nounType: noun.nounType, + metadata: metadata // Stored separately in v4.0.0 + }) + } +} + +export class MyStorageAugmentation extends StorageAugmentation { + private config: MyStorageConfig + + constructor(config: MyStorageConfig) { + super() + this.name = 'my-custom-storage' + this.config = config + } + + // Called during storage resolution phase + async provideStorage(): Promise { + const storage = new MyCustomStorage(this.config) + this.storageAdapter = storage + return storage + } + + // Called during augmentation initialization + protected async onInitialize(): Promise { + await this.storageAdapter!.init() + this.log(`Custom storage initialized`) + } +} +``` + +### Using Your Storage Augmentation + +```typescript +// Register before brain.init() +const brain = new Brainy() +brain.augmentations.register(new MyStorageAugmentation({ + connectionString: 'redis://localhost:6379' +})) +await brain.init() // Will use your storage! +``` + +## Creating a Feature Augmentation + +Here's a complete example of a caching augmentation: + +```typescript +import { BaseAugmentation, BrainyAugmentation } from 'brainy/augmentations' + +export class CachingAugmentation extends BaseAugmentation { + private cache = new Map() + + constructor() { + super() + this.name = 'smart-cache' + this.timing = 'around' // Wrap operations + this.operations = ['search'] // Only cache searches + this.priority = 50 // Mid-priority + } + + async execute(operation: string, params: any, next: () => Promise): Promise { + if (operation === 'search') { + // Check cache + const cacheKey = JSON.stringify(params) + if (this.cache.has(cacheKey)) { + this.log('Cache hit!') + return this.cache.get(cacheKey) + } + + // Execute and cache + const result = await next() + this.cache.set(cacheKey, result) + return result + } + + // Pass through other operations + return next() + } + + protected async onInitialize(): Promise { + this.log('Cache initialized') + } + + async shutdown(): Promise { + this.cache.clear() + await super.shutdown() + } +} +``` + +## The Four Timing Modes + +### 1. `before` - Pre-processing +```typescript +timing = 'before' +async execute(op, params, next) { + // Validate/transform input + const validated = await validate(params) + return next(validated) // Pass modified params +} +``` + +### 2. `after` - Post-processing +```typescript +timing = 'after' +async execute(op, params, next) { + const result = await next() + // Log, analyze, or modify result + console.log(`Operation ${op} returned:`, result) + return result +} +``` + +### 3. `around` - Wrapping (middleware) +```typescript +timing = 'around' +async execute(op, params, next) { + console.log('Starting', op) + try { + const result = await next() + console.log('Success', op) + return result + } catch (error) { + console.log('Failed', op, error) + throw error + } +} +``` + +### 4. `replace` - Complete replacement +```typescript +timing = 'replace' +async execute(op, params, next) { + // Don't call next() - replace entirely! + return myCustomImplementation(params) +} +``` + +## Operations You Can Intercept + +Common operations in Brainy: +- `'storage'` - Storage resolution (special) +- `'add'` - Adding data +- `'search'`, `'similar'` - Searching +- `'update'`, `'delete'` - Modifications +- `'saveNoun'`, `'saveVerb'` - Storage operations +- `'all'` - Intercept everything + +## Context Available to Augmentations + +```typescript +interface AugmentationContext { + brain: Brainy // The brain instance + storage: StorageAdapter // Storage backend + config: BrainyConfig // Configuration + log: (message: string, level?: 'info' | 'warn' | 'error') => void +} +``` + +## Real-World Examples + +### 1. Redis Storage Augmentation +```typescript +export class RedisStorageAugmentation extends StorageAugmentation { + async provideStorage(): Promise { + return new RedisAdapter({ + host: 'localhost', + port: 6379, + // Implement full StorageAdapter interface + }) + } +} +``` + +### 2. Audit Trail Augmentation +```typescript +export class AuditAugmentation extends BaseAugmentation { + timing = 'after' + operations = ['add', 'update', 'delete'] + + async execute(op, params, next) { + const result = await next() + + // Log to audit trail + await this.logAudit({ + operation: op, + params, + result, + timestamp: new Date(), + user: this.context.config.currentUser + }) + + return result + } +} +``` + +### 3. Rate Limiting Augmentation +```typescript +export class RateLimitAugmentation extends BaseAugmentation { + timing = 'before' + operations = ['search'] + private limiter = new RateLimiter({ rps: 100 }) + + async execute(op, params, next) { + await this.limiter.acquire() // Wait if rate limited + return next() + } +} +``` + +## Publishing to Brain Cloud Marketplace + +Future capability for premium augmentations: + +```typescript +// package.json +{ + "name": "@brain-cloud/redis-storage", + "brainy": { + "type": "augmentation", + "category": "storage", + "premium": true + } +} + +// Users can install via: +// brainy augment install redis-storage +``` + +## Best Practices + +### General Practices + +1. **Use BaseAugmentation** - Provides common functionality +2. **Set appropriate priority** - Storage (100), System (80-99), Features (10-50) +3. **Be selective with operations** - Don't use 'all' unless necessary +4. **Handle errors gracefully** - Don't break the chain +5. **Clean up in shutdown()** - Release resources +6. **Log appropriately** - Use context.log() for consistent output +7. **Document your augmentation** - Include examples + +### v4.0.0 Specific Best Practices + +8. **Always include `noun` field** when creating/modifying NounMetadata: + ```typescript + const metadata: NounMetadata = { + noun: 'thing', // REQUIRED! + yourField: 'value' + } + ``` + +9. **Use `verb` property** not `type` when working with relationships: + ```typescript + // ✅ Correct + if (verb.verb === 'relatedTo') { ... } + + // ❌ Wrong (v3.x pattern) + if (verb.type === 'relatedTo') { ... } + ``` + +10. **Access metadata correctly** from storage: + ```typescript + // ✅ Correct - metadata is already structured + const nounType = noun.metadata.noun + + // ⚠️ Fallback pattern for robustness + const nounType = noun.metadata?.noun || 'thing' + ``` + +11. **Respect the two-file storage pattern** - Don't mix vector and metadata operations: + ```typescript + // ✅ Good - Separate concerns + await storage.saveNoun(noun) + await storage.saveMetadata(noun.id, metadata) + + // ❌ Bad - Mixing concerns + await storage.saveNounWithEverything(combinedData) + ``` + +## Testing Your Augmentation + +```typescript +import { Brainy } from 'brainy' +import { MyAugmentation } from './my-augmentation' + +describe('MyAugmentation', () => { + let brain: Brainy + + beforeEach(async () => { + brain = new Brainy() + brain.augmentations.register(new MyAugmentation()) + await brain.init() + }) + + afterEach(async () => { + await brain.destroy() + }) + + it('should enhance searches', async () => { + // Test your augmentation's effect + const results = await brain.search('test') + expect(results).toHaveProperty('enhanced', true) + }) +}) +``` + +## Summary + +Augmentations are Brainy's extension system. They can: +- Replace storage backends +- Add caching layers +- Implement audit trails +- Add rate limiting +- Sync with external systems +- Transform data +- And much more! + +The unified BrainyAugmentation interface makes it easy to create powerful extensions while maintaining consistency across the entire system. \ No newline at end of file diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md deleted file mode 100644 index aa3cd351..00000000 --- a/docs/DATA_MODEL.md +++ /dev/null @@ -1,271 +0,0 @@ -# Data Model - -> How Brainy stores entities and relationships, and the critical distinction between `data` and `metadata`. - ---- - -## Entity (Noun) - -An entity is the fundamental data unit in Brainy. Every entity has: - -| Field | Type | Indexed | Description | -|-------|------|---------|-------------| -| `id` | `string` | Primary key | UUID v4 (auto-generated or custom) | -| `data` | `any` | **HNSW vector index** | Content used for semantic/hybrid search. Strings auto-embed. | -| `metadata` | `object` | **MetadataIndex** | Structured queryable fields (tags, dates, flags, etc.) | -| `type` | `NounType` | MetadataIndex (as `noun`) | Entity type classification | -| `vector` | `number[]` | HNSW | 384-dim embedding (auto-computed from `data` or user-provided) | -| `confidence` | `number` | MetadataIndex | Type classification confidence (0-1) | -| `weight` | `number` | MetadataIndex | Entity importance/salience (0-1) | -| `service` | `string` | MetadataIndex | Multi-tenancy identifier | -| `createdAt` | `number` | MetadataIndex | Creation timestamp (ms since epoch) | -| `updatedAt` | `number` | MetadataIndex | Last update timestamp (ms since epoch) | -| `createdBy` | `object` | MetadataIndex | Source augmentation info | - -### Example - -```typescript -const id = await brain.add({ - data: 'John Smith is a software engineer at Acme Corp', // → embedded into vector - type: NounType.Person, - metadata: { // → indexed, queryable via where filters - role: 'engineer', - department: 'backend', - yearsExperience: 8 - }, - confidence: 0.95, - weight: 0.7 -}) -``` - ---- - -## Relationship (Verb) - -A relationship is a typed, directed edge connecting two entities. - -| Field | Type | Indexed | Description | -|-------|------|---------|-------------| -| `id` | `string` | Primary key | UUID v4 (auto-generated) | -| `from` | `string` | **GraphAdjacencyIndex** | Source entity ID | -| `to` | `string` | **GraphAdjacencyIndex** | Target entity ID | -| `type` | `VerbType` | GraphAdjacencyIndex (as `verb`) | Relationship type classification | -| `data` | `any` | — | Opaque content (overrides auto-computed vector if provided) | -| `metadata` | `object` | — | Structured fields on the edge | -| `weight` | `number` | — | Connection strength (0-1, default: 1.0) | -| `confidence` | `number` | — | Relationship certainty (0-1) | -| `evidence` | `RelationEvidence` | — | Why this relationship was detected | -| `createdAt` | `number` | — | Creation timestamp (ms since epoch) | -| `updatedAt` | `number` | — | Last update timestamp (ms since epoch) | -| `service` | `string` | — | Multi-tenancy identifier | - -### Example - -```typescript -const relId = await brain.relate({ - from: personId, - to: projectId, - type: VerbType.WorksOn, - data: 'Lead engineer on the AI module', // Optional: content for this edge - metadata: { // Optional: queryable edge fields - role: 'lead', - startDate: '2024-01-15' - }, - weight: 0.9 -}) -``` - ---- - -## Data vs Metadata - -This is the most important concept in Brainy's storage model: - -### `data` — Content for Semantic Search - -- Embedded into a 384-dimensional vector via the WASM embedding engine -- Searchable via **semantic similarity** (HNSW vector index) and **hybrid text+semantic** search -- Queried by passing `query` to `find()`: - ```typescript - brain.find({ query: 'machine learning algorithms' }) - ``` -- **NOT** indexed by MetadataIndex — you cannot use `where` filters on `data` -- Stored opaquely: strings, objects, numbers — anything goes - -### `metadata` — Structured Queryable Fields - -- Indexed by MetadataIndex with O(1) lookups per field -- Queryable via `where` filters using [BFO operators](./QUERY_OPERATORS.md): - ```typescript - brain.find({ - where: { - department: 'engineering', - yearsExperience: { greaterThan: 5 }, - tags: { contains: 'senior' } - } - }) - ``` -- **NOT** used for vector/semantic search -- Must be a flat or lightly nested object - -### Quick Reference - -| | `data` | `metadata` | -|---|---|---| -| **Purpose** | Content for embedding / semantic search | Structured fields for filtering | -| **Searched by** | `find({ query })` — vector similarity, hybrid text+semantic | `find({ where })` — exact, range, set operators | -| **Indexed by** | HNSW vector index | MetadataIndex | -| **Queryable with operators?** | No | Yes (`equals`, `greaterThan`, `oneOf`, etc.) | -| **Auto-embedded?** | Yes (strings → 384-dim vectors) | No | -| **Typical content** | Text descriptions, document content | Tags, dates, status flags, categories, numeric fields | - -### Common Pattern - -```typescript -// Add an article -await brain.add({ - data: 'A deep dive into transformer architectures and attention mechanisms', - type: NounType.Document, - metadata: { - title: 'Transformer Deep Dive', - author: 'Dr. Chen', - publishedYear: 2024, - tags: ['AI', 'transformers', 'NLP'], - status: 'published' - } -}) - -// Search by content (semantic — searches data) -const results = await brain.find({ query: 'neural network attention' }) - -// Filter by fields (exact — queries metadata) -const recent = await brain.find({ - where: { - publishedYear: { greaterThan: 2023 }, - status: 'published' - } -}) - -// Combine both (Triple Intelligence) -const precise = await brain.find({ - query: 'attention mechanisms', // Semantic search on data - where: { author: 'Dr. Chen' }, // Metadata filter - connected: { from: authorId, depth: 1 } // Graph traversal -}) -``` - ---- - -## Storage Field Naming - -Internally, Brainy uses different field names in storage vs the public API: - -| Public API (Entity/Relation) | Storage (metadata object) | Notes | -|------------------------------|--------------------------|-------| -| `type` | `noun` | Entity type stored as `noun` | -| `from` | `sourceId` | Relationship source | -| `to` | `targetId` | Relationship target | -| `type` (on Relation) | `verb` | Relationship type stored as `verb` | - -When querying with `find()`, you can use: -- `type` parameter (convenience alias, equivalent to `where.noun`) -- `where.noun` directly - -```typescript -// These are equivalent: -brain.find({ type: NounType.Person }) -brain.find({ where: { noun: NounType.Person } }) -``` - ---- - -## Standard Metadata Fields - -When you add an entity, Brainy stores these standard fields in the metadata object alongside your custom fields: - -| Field | Set By | Description | -|-------|--------|-------------| -| `noun` | System | Entity type (NounType enum value) | -| `subtype` | User | Per-NounType sub-classification (e.g. `'employee'`, `'invoice'`, `'milestone'`). Flat string, no hierarchy. Indexed on the fast path and rolled into per-NounType statistics. | -| `data` | System | The raw `data` value (stored opaquely) | -| `createdAt` | System | Creation timestamp | -| `updatedAt` | System | Last update timestamp | -| `confidence` | User | Type classification confidence | -| `weight` | User | Entity importance | -| `service` | User | Multi-tenancy identifier | -| `createdBy` | User/System | Source augmentation | - -On read, these standard fields are extracted to top-level Entity properties. The `metadata` field on the returned Entity contains **only your custom fields**. - -### Subtype — sub-classification within a NounType - -`type` (NounType) is a stable 42-value enum. `subtype` is the consumer-chosen string vocabulary *within* a type: - -```typescript -// A Person who is an employee: -await brain.add({ - data: 'Avery Brooks — runs the AI lab', - type: NounType.Person, - subtype: 'employee', - metadata: { department: 'ai-lab' } -}) - -// A Document that is an invoice: -await brain.add({ - data: 'INV-2026-001', - type: NounType.Document, - subtype: 'invoice', - metadata: { amount: 1500 } -}) -``` - -`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's how `find({ type, subtype })` routes through the standard-field fast path (column-store hit) instead of the metadata fallback. See **[Subtypes & Facets](./guides/subtypes-and-facets.md)** for the full guide including `trackField()` and `migrateField()`. - -### Subtype — sub-classification within a VerbType (7.30+) - -Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive — a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field on `HNSWVerbWithMetadata` and on the public `Relation`: - -```typescript -await brain.relate({ - from: ceoId, - to: vpId, - type: VerbType.ReportsTo, - subtype: 'direct', // top-level standard field - metadata: { since: '2025-Q1' } // user-custom fields stay in metadata -}) -``` - -Fast-path filter on the verb side: - -```typescript -const direct = await brain.related({ - from: ceoId, - type: VerbType.ReportsTo, - subtype: 'direct' -}) -``` - -The verb-side rollup at `_system/verb-subtype-statistics.json` mirrors the noun-side `_system/subtype-statistics.json` — same shape, same self-heal machinery. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`. - -Verbs and nouns now have full capability parity — every API on the noun side has a verb-side mirror, including the new `brain.updateRelation()` (which closed a pre-7.30 gap where relationships had no update path). - -### Standard verb fields - -The verb-side equivalent of `STANDARD_ENTITY_FIELDS` is `STANDARD_VERB_FIELDS`, exported from `src/coreTypes.ts`. Verb-specific standard fields: - -| Field | Description | -|---|---| -| `verb` | The VerbType enum value | -| `sourceId` / `targetId` | The two endpoints of the relationship | -| `subtype` | Sub-classification within the VerbType (7.30+) | -| `confidence`, `weight`, `createdAt`, `updatedAt`, `service`, `createdBy`, `data` | Same semantics as the noun-side standard fields | - -The companion `resolveVerbField(verb, field)` helper resolves field paths the same way `resolveEntityField` does for nouns: standard fields first, metadata fallback for everything else. - ---- - -## See Also - -- [API Reference](./api/README.md) — Complete API documentation -- [Query Operators](./QUERY_OPERATORS.md) — All BFO operators with examples -- [Find System](./FIND_SYSTEM.md) — Natural language find() details diff --git a/docs/DEVELOPER_LEARNING_PATH.md b/docs/DEVELOPER_LEARNING_PATH.md deleted file mode 100644 index 4134ae63..00000000 --- a/docs/DEVELOPER_LEARNING_PATH.md +++ /dev/null @@ -1,1166 +0,0 @@ -# 🎓 Brainy Developer Learning Path -**From Zero to Hero in 5 Progressive Levels** - -> This guide takes you from your first Brainy query to production-scale neural database mastery. Follow each level in order for the best learning experience. - ---- - -## 📋 Quick Navigation - -- [Level 1: Hello Brainy](#level-1-hello-brainy-15-minutes) - Your first neural database -- [Level 2: Relationships & Batch Operations](#level-2-relationships--batch-operations-30-minutes) - Scale up your data -- [Level 3: Advanced Search & Neural AI](#level-3-advanced-search--neural-ai-45-minutes) - Triple Intelligence -- [Level 4: Virtual Filesystem](#level-4-virtual-filesystem-60-minutes) - Files as intelligent entities -- [Level 5: Production Scale](#level-5-production-scale-90-minutes) - Planet-scale deployment - ---- - -## Level 1: Hello Brainy (15 minutes) - -### What You'll Learn -- Initialize Brainy -- Add your first entity -- Perform semantic search -- Understand basic types - -### Prerequisites -```bash -npm install @soulcraft/brainy -``` - -### Your First Neural Database - -```typescript -import { Brainy, NounType } from '@soulcraft/brainy' - -// Step 1: Create and initialize Brainy -const brain = new Brainy({ - storage: { type: 'memory' } // Start simple - no persistence needed -}) -await brain.init() - -// Step 2: Add some data -const johnId = await brain.add({ - data: 'John Smith is a software engineer at TechCorp', - type: NounType.Person, - metadata: { role: 'Engineer', company: 'TechCorp' } -}) - -const aliceId = await brain.add({ - data: 'Alice Johnson is a product manager at TechCorp', - type: NounType.Person, - metadata: { role: 'Manager', company: 'TechCorp' } -}) - -const projectId = await brain.add({ - data: 'AI-powered customer support system using machine learning', - type: NounType.Project, - metadata: { status: 'active', priority: 'high' } -}) - -// Step 3: Semantic search (this is where magic happens!) -console.log('\n🔍 Searching for "engineers"...') -const engineers = await brain.find({ query: 'engineers' }) -console.log(`Found ${engineers.length} engineers:`) -for (const result of engineers) { - console.log(` - ${result.entity.data} (score: ${result.score.toFixed(2)})`) -} - -// Step 4: Search with filters -console.log('\n🔍 Searching for "people at TechCorp"...') -const techcorpPeople = await brain.find({ - query: 'people', - type: NounType.Person, - where: { company: 'TechCorp' }, - limit: 10 -}) -console.log(`Found ${techcorpPeople.length} people at TechCorp`) - -// Step 5: Get entity by ID -const john = await brain.get(johnId) -console.log('\n👤 John\'s data:', { - type: john?.type, - data: john?.data, - metadata: john?.metadata -}) - -// Step 6: Clean up -await brain.close() -console.log('\n✅ Done! You just created your first neural database!') -``` - -### Key Concepts - -#### 1. **NounType** - Entity Classification -Brainy has 31 built-in types including: -- `Person`, `Organization`, `Location` -- `Document`, `File`, `Content` -- `Product`, `Service`, `Event` -- `Project`, `Task`, `Concept` - -**Why it matters**: Proper typing enables intelligent search and organization. - -#### 2. **Semantic Search** - Understanding Meaning -```typescript -// Traditional search: exact keyword matching -// "engineers" would NOT find "software developer" - -// Semantic search: understands meaning -await brain.find({ query: 'engineers' }) -// ✅ Finds: "software engineer", "developer", "programmer", "coder" -``` - -#### 3. **Metadata Filtering** - Precise Control -```typescript -// Combine semantic search with structured filters -await brain.find({ - query: 'machine learning', // Semantic: finds AI, ML, neural networks - where: { company: 'TechCorp' }, // Structured: exact match - type: NounType.Project // Type filter -}) -``` - -### Practice Exercises - -1. Create a small company directory with 5-10 people -2. Search for "managers", "developers", "designers" -3. Add projects and search for "active projects" -4. Experiment with different metadata filters - -### Next Steps -Once you're comfortable with basic operations, move to **Level 2** to learn about relationships and batch operations. - ---- - -## Level 2: Relationships & Batch Operations (30 minutes) - -### What You'll Learn -- Create relationships between entities -- Batch add/update/delete operations -- Query graph relationships -- Understand VerbTypes - -### Building a Knowledge Graph - -```typescript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' - -const brain = new Brainy({ storage: { type: 'memory' } }) -await brain.init() - -// Batch add multiple entities -console.log('📦 Adding team members...') -const result = await brain.addMany({ - items: [ - { data: 'John Smith - Senior Engineer', type: NounType.Person, metadata: { role: 'Engineer' } }, - { data: 'Alice Johnson - Product Manager', type: NounType.Person, metadata: { role: 'Manager' } }, - { data: 'Bob Wilson - Designer', type: NounType.Person, metadata: { role: 'Designer' } }, - { data: 'TechCorp - Software Company', type: NounType.Organization }, - { data: 'AI Assistant Project', type: NounType.Project, metadata: { status: 'active' } } - ], - parallel: true, - onProgress: (done, total) => console.log(` Progress: ${done}/${total}`) -}) - -console.log(`✅ Added ${result.successful.length} entities`) -const [johnId, aliceId, bobId, techcorpId, projectId] = result.successful - -// Create relationships (building the graph!) -console.log('\n🔗 Creating relationships...') -await brain.relateMany({ - relations: [ - // People work for organization - { from: johnId, to: techcorpId, type: VerbType.WorksWith }, - { from: aliceId, to: techcorpId, type: VerbType.WorksWith }, - { from: bobId, to: techcorpId, type: VerbType.WorksWith }, - - // People work on project - { from: johnId, to: projectId, type: VerbType.WorksOn }, - { from: bobId, to: projectId, type: VerbType.WorksOn }, - - // Alice manages the project - { from: aliceId, to: projectId, type: VerbType.Manages }, - - // Team collaboration - { from: johnId, to: aliceId, type: VerbType.CollaboratesWith, bidirectional: true }, - { from: bobId, to: johnId, type: VerbType.CollaboratesWith, bidirectional: true } - ] -}) - -console.log('✅ Created relationships') - -// Query relationships -console.log('\n🔍 Querying relationships...') - -// Who works for TechCorp? -const techcorpEmployees = await brain.related({ - to: techcorpId, - type: VerbType.WorksWith -}) -console.log(`TechCorp has ${techcorpEmployees.length} employees`) - -// Who works on the AI project? -const projectContributors = await brain.related({ - to: projectId, - type: [VerbType.WorksOn, VerbType.Manages] -}) -console.log(`AI Project has ${projectContributors.length} contributors`) - -// Who does John collaborate with? -const johnsCollaborators = await brain.related({ - from: johnId, - type: VerbType.CollaboratesWith -}) -console.log(`John collaborates with ${johnsCollaborators.length} people`) - -// Get graph statistics -const stats = brain.getStats() -console.log('\n📊 Graph Statistics:', { - entities: stats.entities.total, - relationships: stats.relationships.totalRelationships, - density: stats.density.toFixed(2) -}) - -// Batch update -console.log('\n📝 Updating all team members...') -await brain.updateMany({ - items: [johnId, aliceId, bobId].map(id => ({ - id, - metadata: { team: 'AI Team', updated: new Date().toISOString() }, - merge: true // Merge with existing metadata (don't replace!) - })) -}) - -console.log('✅ Updated team metadata') - -await brain.close() -``` - -### Key Concepts - -#### 1. **VerbType** - Relationship Types -Brainy has 40 relationship types including: -- Work: `WorksWith`, `WorksOn`, `Manages`, `Supervises` -- Structure: `PartOf`, `Contains`, `BelongsTo` -- Knowledge: `RelatedTo`, `DependsOn`, `Requires` -- Creation: `Creates`, `Modifies`, `Transforms` - -#### 2. **Bidirectional Relationships** -```typescript -await brain.relate({ - from: personA, - to: personB, - type: VerbType.CollaboratesWith, - bidirectional: true // Creates A→B AND B→A -}) -``` - -#### 3. **Batch Operations = Performance** -```typescript -// ❌ Slow: 100 individual operations -for (const item of items) { - await brain.add(item) // 100 round trips! -} - -// ✅ Fast: 1 batch operation -await brain.addMany({ items }) // 1 round trip! -``` - -#### 4. **Metadata Merging** -```typescript -// Initial metadata -await brain.add({ - data: 'John', - metadata: { role: 'Engineer', level: 3 } -}) - -// Update with merge: true (default) -await brain.update({ - id: johnId, - metadata: { team: 'AI Team' }, - merge: true // Result: { role: 'Engineer', level: 3, team: 'AI Team' } -}) - -// Update with merge: false -await brain.update({ - id: johnId, - metadata: { team: 'AI Team' }, - merge: false // Result: { team: 'AI Team' } - role and level lost! -}) -``` - -### Practice Exercises - -1. Create an organizational hierarchy (CEO → Managers → Engineers) -2. Build a project dependency graph -3. Model a social network with CollaboratesWith relationships -4. Query "Who reports to Alice?" using related() -5. Batch update all projects to add a "year: 2024" field - -### Next Steps -Ready for AI-powered search and clustering? Move to **Level 3**. - ---- - -## Level 3: Advanced Search & Neural AI (45 minutes) - -### What You'll Learn -- Triple Intelligence (Vector + Metadata + Graph) -- Semantic similarity -- Automatic clustering -- Outlier detection -- Natural language queries - -### Triple Intelligence in Action - -```typescript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' - -const brain = new Brainy({ storage: { type: 'memory' } }) -await brain.init() - -// Create a realistic dataset -console.log('📦 Creating knowledge base...') -const knowledgeBase = await brain.addMany({ - items: [ - // Research papers - { data: 'Deep Learning for Computer Vision using Convolutional Neural Networks', - type: NounType.Document, - metadata: { category: 'AI', year: 2024, citations: 150 } }, - { data: 'Natural Language Processing with Transformer Models', - type: NounType.Document, - metadata: { category: 'AI', year: 2024, citations: 200 } }, - { data: 'Reinforcement Learning for Robotics Applications', - type: NounType.Document, - metadata: { category: 'AI', year: 2023, citations: 80 } }, - - // Different domain - { data: 'Climate Change Impact on Ocean Ecosystems', - type: NounType.Document, - metadata: { category: 'Climate', year: 2024, citations: 120 } }, - { data: 'Renewable Energy Solutions for Urban Planning', - type: NounType.Document, - metadata: { category: 'Energy', year: 2024, citations: 95 } }, - - // Code projects - { data: 'AI-powered code completion tool using GPT', - type: NounType.Project, - metadata: { category: 'Tools', status: 'active' } }, - { data: 'Neural network visualization dashboard', - type: NounType.Project, - metadata: { category: 'Tools', status: 'active' } } - ] -}) - -console.log(`✅ Created ${knowledgeBase.successful.length} entities\n`) - -// 1. VECTOR INTELLIGENCE: Semantic similarity -console.log('🔍 1. VECTOR INTELLIGENCE: Semantic Search') -const aiResults = await brain.find({ - query: 'machine learning and neural networks', // User's natural language - limit: 3 -}) - -console.log('Top 3 semantically similar documents:') -aiResults.forEach((r, i) => { - console.log(` ${i + 1}. [${r.score.toFixed(3)}] ${r.entity.data?.substring(0, 50)}...`) -}) - -// 2. METADATA INTELLIGENCE: Structured filtering -console.log('\n🔍 2. METADATA INTELLIGENCE: Precise Filtering') -const recentHighCitations = await brain.find({ - query: 'artificial intelligence', - where: { - year: 2024, - citations: { $gte: 100 } // Brainy Field Operator: greater than or equal - }, - limit: 10 -}) - -console.log(`Found ${recentHighCitations.length} highly-cited AI papers from 2024`) - -// 3. GRAPH INTELLIGENCE: Relationship-aware search -console.log('\n🔍 3. GRAPH INTELLIGENCE: Relationship-Aware Search') - -// First, create some relationships -const [paper1, paper2] = knowledgeBase.successful -await brain.relate({ - from: paper1, - to: paper2, - type: VerbType.References -}) - -// Search with graph constraints -const connectedDocs = await brain.find({ - query: 'deep learning', - connected: { - to: paper2, - via: VerbType.References - } -}) - -console.log(`Found ${connectedDocs.length} papers that reference the NLP paper`) - -// 4. FUSION: Combine all three intelligences! -console.log('\n🔍 4. TRIPLE INTELLIGENCE FUSION') -const fusionResults = await brain.find({ - query: 'AI research', // Vector: semantic understanding - where: { year: 2024 }, // Metadata: structured filter - type: NounType.Document, // Type constraint - fusion: { - strategy: 'adaptive', // Let Brainy optimize weights - weights: { - vector: 0.5, // 50% semantic similarity - field: 0.3, // 30% metadata match - graph: 0.2 // 20% relationship strength - } - }, - explain: true // See how the score was calculated -}) - -console.log('Fusion search results with score explanations:') -fusionResults.forEach(r => { - console.log(`\n ${r.entity.data?.substring(0, 60)}...`) - console.log(` Total score: ${r.score.toFixed(3)}`) - if (r.explanation) { - console.log(` Vector: ${r.explanation.vector.toFixed(3)}`) - console.log(` Metadata: ${r.explanation.metadata.toFixed(3)}`) - console.log(` Graph: ${r.explanation.graph.toFixed(3)}`) - } -}) - -// 5. SIMILARITY: Find similar documents -console.log('\n\n🔍 SIMILARITY: Find Similar Documents') -const similarTo = await brain.similar({ - to: paper1, // Entity ID of first AI paper - limit: 3, - threshold: 0.5, // Minimum similarity score - type: NounType.Document -}) - -console.log(`Documents similar to "${knowledgeBase.successful[0]}":`) -similarTo.forEach(r => { - console.log(` [${r.score.toFixed(3)}] ${r.entity.data?.substring(0, 50)}...`) -}) - -await brain.close() -``` - -### Key Concepts - -#### 1. **Triple Intelligence Explained** - -``` -Traditional Database: WHERE category = 'AI' (exact match only) - ❌ Misses: "artificial intelligence", "machine learning" - -Vector Search: semantic("AI research") (meaning-based) - ✅ Finds: AI, ML, neural networks, deep learning - ❌ No filtering by year, citations, etc. - -Brainy Triple: semantic("AI") + WHERE year=2024 + CONNECTED TO paper123 - ✅ Finds semantically similar + filters + graph aware -``` - -#### 2. **Score Explanations** -```typescript -const results = await brain.find({ - query: 'AI', - explain: true // Get score breakdown -}) - -// result.explanation shows: -// { -// vector: 0.85, // 85% semantic match -// metadata: 0.90, // 90% field match -// graph: 0.70, // 70% graph relevance -// final: 0.82 // Weighted combination -// } -``` - -#### 3. **Fusion Strategies** -```typescript -// 'adaptive' - Brainy automatically adjusts weights based on query -fusion: { strategy: 'adaptive' } - -// 'balanced' - Equal weights to all signals -fusion: { strategy: 'balanced' } - -// 'custom' - You control the weights -fusion: { - strategy: 'custom', - weights: { vector: 0.7, field: 0.2, graph: 0.1 } -} -``` - -#### 4. **Brainy Field Operators (BFO)** -```typescript -where: { - age: { $gte: 18, $lte: 65 }, // Range - role: { $in: ['Engineer', 'Manager'] }, // One of - name: { $contains: 'John' }, // Substring - active: true, // Exact match - tags: { $includes: 'AI' } // Array contains -} -``` - -### Practice Exercises - -1. Create a document collection and find semantically similar items -2. Use fusion search with custom weights -3. Cluster your data and examine the clusters -4. Find outliers in a dataset -5. Compare results with/without explain: true - -### Next Steps -Want to treat files as intelligent entities? Learn the **Virtual Filesystem** in Level 4. - ---- - -## Level 4: Virtual Filesystem (60 minutes) - -### What You'll Learn -- VFS as knowledge operating system -- Files with semantic understanding -- Semantic file search -- Cross-boundary relationships (VFS ↔ Knowledge) -- VFS filtering architecture - -### Files as Intelligent Entities - -```typescript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' - -const brain = new Brainy({ storage: { type: 'memory' } }) -await brain.init() - -// Initialize VFS -const vfs = brain.vfs() -await vfs.init() - -console.log('📁 Creating semantic filesystem...\n') - -// 1. BASIC FILE OPERATIONS (POSIX-like) -await vfs.mkdir('/projects', { recursive: true }) -await vfs.mkdir('/projects/ai-assistant') -await vfs.mkdir('/docs') - -await vfs.writeFile('/projects/ai-assistant/README.md', ` -# AI Assistant Project - -A neural-powered assistant using transformer models for natural language understanding. - -## Features -- Semantic search -- Context-aware responses -- Multi-turn conversations -`) - -await vfs.writeFile('/projects/ai-assistant/architecture.md', ` -# Architecture - -## Components -- NLP Engine: Transformer-based language model -- Knowledge Graph: Brainy neural database -- API Layer: RESTful endpoints -`) - -await vfs.writeFile('/docs/installation.md', ` -# Installation Guide - -\`\`\`bash -npm install ai-assistant -\`\`\` -`) - -console.log('✅ Created 3 files\n') - -// 2. VFS-ONLY SEMANTIC SEARCH -console.log('🔍 Searching VFS for "neural networks"...') -const vfsFiles = await vfs.search('neural networks', { limit: 5 }) -console.log(`Found ${vfsFiles.length} VFS files:`) -vfsFiles.forEach(f => { - console.log(` [${f.score.toFixed(3)}] ${f.path}`) -}) - -// 3. VFS FILTERING IN KNOWLEDGE QUERIES -console.log('\n🔍 Understanding VFS filtering...\n') - -// Create some knowledge entities -const conceptId = await brain.add({ - data: 'Neural networks are computational models inspired by biological neurons', - type: NounType.Concept, - metadata: { topic: 'AI' } -}) - -const projectId = await brain.add({ - data: 'AI Assistant - conversational AI using transformers', - type: NounType.Project, - metadata: { status: 'active' } -}) - -console.log('Created 2 knowledge entities\n') - -// DEFAULT: Knowledge queries exclude VFS (clean separation!) -console.log('📊 brain.find() - DEFAULT behavior (excludes VFS):') -const knowledgeOnly = await brain.find({ query: 'neural networks' }) -console.log(` Found ${knowledgeOnly.length} entities`) -console.log(` VFS files: ${knowledgeOnly.filter(r => r.metadata?.isVFS).length}`) // 0 -console.log(` Knowledge: ${knowledgeOnly.filter(r => !r.metadata?.isVFS).length}`) - -// OPT-IN: Include VFS when needed -console.log('\n📊 brain.find() with includeVFS: true:') -const everything = await brain.find({ - query: 'neural networks', - includeVFS: true // Opt-in to include VFS files -}) -console.log(` Found ${everything.length} entities`) -console.log(` VFS files: ${everything.filter(r => r.metadata?.isVFS).length}`) -console.log(` Knowledge: ${everything.filter(r => !r.metadata?.isVFS).length}`) - -// VFS-ONLY: Search only files -console.log('\n📊 Searching ONLY VFS files:') -const filesOnly = await brain.find({ - where: { vfsType: 'file', extension: '.md' }, - includeVFS: true // Required to find VFS entities -}) -console.log(` Found ${filesOnly.length} markdown files`) - -// 4. CROSS-BOUNDARY RELATIONSHIPS -console.log('\n\n🔗 Creating cross-boundary relationships...') - -// Link concept to documentation file -const readmeEntity = await brain.find({ - where: { path: '/projects/ai-assistant/README.md' }, - includeVFS: true, - limit: 1 -}) - -if (readmeEntity.length > 0) { - await brain.relate({ - from: conceptId, - to: readmeEntity[0].id, - type: VerbType.DocumentedBy, - metadata: { section: 'Features' } - }) - console.log('✅ Linked concept to README.md') -} - -// Query relationships -const conceptDocs = await brain.related({ - from: conceptId, - type: VerbType.DocumentedBy -}) -console.log(`Concept is documented by ${conceptDocs.length} files`) - -// 5. VFS SEMANTIC FEATURES -console.log('\n\n🔍 VFS Semantic Features:') - -// Find similar files -const similarFiles = await vfs.findSimilar('/projects/ai-assistant/README.md', { - limit: 3, - threshold: 0.5 -}) -console.log(`\nFiles similar to README.md: ${similarFiles.length}`) -similarFiles.forEach(f => { - console.log(` [${f.score.toFixed(3)}] ${f.path}`) -}) - -// Get file stats -const stats = await vfs.stat('/projects/ai-assistant/README.md') -console.log('\nREADME.md stats:', { - size: stats.size, - type: stats.vfsType, - extension: stats.metadata?.extension, - created: new Date(stats.metadata?.createdAt || 0).toLocaleString() -}) - -// Read directory -console.log('\n📁 Directory contents of /projects/ai-assistant:') -const entries = await vfs.readdir('/projects/ai-assistant') -console.log(entries) - -// 6. METADATA & EXTENDED ATTRIBUTES -console.log('\n\n📝 Metadata & Extended Attributes:') - -await vfs.setMetadata('/projects/ai-assistant/README.md', { - author: 'John Smith', - version: '1.0.0', - tags: ['AI', 'documentation', 'project'] -}) - -const metadata = await vfs.getMetadata('/projects/ai-assistant/README.md') -console.log('README metadata:', metadata) - -// Extended attributes (like file properties) -await vfs.setxattr('/projects/ai-assistant/README.md', 'priority', 'high') -await vfs.setxattr('/projects/ai-assistant/README.md', 'reviewStatus', 'approved') - -const xattrs = await vfs.listxattr('/projects/ai-assistant/README.md') -console.log('Extended attributes:', xattrs) - -// 7. FILE OPERATIONS -console.log('\n\n📋 Advanced File Operations:') - -// Copy file -await vfs.copy('/docs/installation.md', '/projects/ai-assistant/INSTALL.md') -console.log('✅ Copied installation.md') - -// Rename -await vfs.rename('/projects/ai-assistant/INSTALL.md', '/projects/ai-assistant/setup.md') -console.log('✅ Renamed to setup.md') - -// Check existence -const exists = await vfs.exists('/projects/ai-assistant/setup.md') -console.log(`setup.md exists: ${exists}`) - -console.log('\n\n✅ VFS Tutorial Complete!') -console.log('\n📚 Key Takeaways:') -console.log(' 1. VFS files have semantic understanding (search by meaning)') -console.log(' 2. brain.find() excludes VFS by default (clean knowledge queries)') -console.log(' 3. Use includeVFS: true to include VFS in knowledge queries') -console.log(' 4. vfs.search() ONLY searches VFS files (never knowledge entities)') -console.log(' 5. Cross-boundary relationships link files to concepts') -console.log(' 6. Every file is a full Brainy entity with vector, metadata, and graph') - -await vfs.close() -await brain.close() -``` - -### Key Concepts - -#### 1. **VFS Filtering Architecture** - -```typescript -// 🎯 DEFAULT BEHAVIOR: Clean Separation -// -// Knowledge queries stay clean (no VFS pollution) -const concepts = await brain.find({ query: 'AI' }) -// Returns: Only NounType.Concept, NounType.Document, etc. -// Excludes: VFS files (no .path property) - -// VFS queries work with VFS only -const files = await vfs.search('documentation') -// Returns: Only VFS files with .path property -// Excludes: Knowledge entities - -// 🔄 CROSS-BOUNDARY: Opt-in when needed -const everything = await brain.find({ - query: 'machine learning', - includeVFS: true // Include both knowledge AND VFS -}) -// Returns: Knowledge entities + VFS files - -// 📁 VFS-ONLY via brain.find() -const markdownFiles = await brain.find({ - where: { vfsType: 'file', extension: '.md' }, - includeVFS: true // Required to find VFS entities -}) -``` - -#### 2. **Cross-Boundary Relationships** - -```typescript -// Files can relate to knowledge entities -await brain.relate({ - from: conceptId, // Knowledge: NounType.Concept - to: fileId, // VFS: File entity - type: VerbType.DocumentedBy -}) - -// Query across boundaries -const conceptDocs = await brain.related({ - from: conceptId, - type: VerbType.DocumentedBy -}) -// Returns: VFS files that document the concept -``` - -#### 3. **VFS vs Traditional Filesystem** - -| Feature | Traditional FS | Brainy VFS | -|---------|---------------|------------| -| Search | Filename only | Semantic content search | -| Organization | Hierarchy only | Hierarchy + Graph | -| Metadata | Limited (size, dates) | Unlimited custom metadata | -| Relationships | None | Full graph relationships | -| Similarity | None | Find similar files | -| Understanding | None | Vector embeddings | - -#### 4. **When to Use What** - -```typescript -// Use vfs.* methods for file operations -await vfs.writeFile('/path/to/file.txt', content) -await vfs.readFile('/path/to/file.txt') -await vfs.search('semantic query') - -// Use brain.* methods for knowledge operations -await brain.add({ data: 'concept', type: NounType.Concept }) -await brain.find({ query: 'concept' }) // Excludes VFS by default - -// Use includeVFS for cross-boundary queries -await brain.find({ - query: 'documentation', - includeVFS: true // Search both knowledge AND files -}) -``` - -### Practice Exercises - -1. Create a project structure with docs, source code, tests -2. Add semantic tags to files -3. Search for "API documentation" and see VFS filtering in action -4. Create relationships between code files and design documents -5. Find files similar to a specific README -6. Compare results with/without includeVFS - -### Next Steps -Ready for production deployment? Level 5 covers **planet-scale architecture**. - ---- - -## Level 5: Production Scale (90 minutes) - -### What You'll Learn -- Production filesystem storage and off-site backup -- Performance optimization -- Batch imports (CSV, Excel, PDF) -- Metadata query optimization -- Production best practices - -### Production-Ready Deployment - -```typescript -import { Brainy, NounType } from '@soulcraft/brainy' - -// 1. PRODUCTION STORAGE - Filesystem with off-site snapshots -console.log('Initializing production storage...\n') - -const brain = new Brainy({ - storage: { - type: 'filesystem', - path: '/var/lib/brainy' - }, - - // Performance tuning - cache: { - maxSize: 10000, // Cache up to 10K entities - ttl: 600000 // 10 minute TTL - }, - - // Monitoring - verbose: process.env.NODE_ENV === 'development' -}) - -await brain.init() -console.log('Brainy initialized with filesystem storage') -console.log('Snapshot /var/lib/brainy off-site via cron with `gsutil rsync` / `aws s3 sync` / `rclone`.\n') - -// 2. BATCH IMPORT - CSV File -console.log('📊 Importing CSV data...\n') - -const csvResult = await brain.import('./data/customers-1000.csv', { - vfsPath: '/imports/customers.csv', // Store in VFS - createEntities: true, // Create knowledge entities - batchSize: 100, // Process in batches of 100 - onProgress: (done, total) => { - console.log(` Progress: ${done}/${total} (${(done/total*100).toFixed(1)}%)`) - } -}) - -console.log('\n📊 Import Results:') -console.log(` Entities created: ${csvResult.stats.graphNodesCreated}`) -console.log(` VFS files created: ${csvResult.stats.vfsFilesCreated}`) -console.log(` Duration: ${csvResult.stats.duration}ms`) - -// 3. METADATA QUERY OPTIMIZATION -console.log('\n\n🔍 Metadata Query Optimization:\n') - -// Discover what fields are available -const fields = await brain.getAvailableFields() -console.log(`Available metadata fields: ${fields.length}`) -console.log(` Top fields: ${fields.slice(0, 10).join(', ')}`) - -// Get field statistics (cardinality, types) -const fieldStats = await brain.getFieldStatistics() -console.log(`\nField statistics:`) -const topFields = Array.from(fieldStats.entries()).slice(0, 5) -topFields.forEach(([field, stats]) => { - console.log(` ${field}: ${stats.cardinality} unique values`) -}) - -// Get optimal query plan -const queryPlan = await brain.getOptimalQueryPlan({ - status: 'active', - year: 2024 -}) -console.log(`\nQuery plan:`) -console.log(` Estimated results: ${queryPlan.estimatedResults}`) -console.log(` Index usage: ${queryPlan.indexUsage.join(', ')}`) -console.log(` Execution time: ~${queryPlan.estimatedMs}ms`) - -// 4. LARGE-SCALE BATCH OPERATIONS -console.log('\n\n📦 Large-Scale Batch Operations:\n') - -// Generate test data -const testItems = Array.from({ length: 1000 }, (_, i) => ({ - data: `Test entity ${i} - Machine learning and artificial intelligence`, - type: NounType.Document, - metadata: { - index: i, - category: i % 5 === 0 ? 'AI' : 'General', - priority: Math.random() > 0.5 ? 'high' : 'normal', - year: 2024 - } -})) - -console.log(`Adding 1000 entities...`) -const startTime = Date.now() - -const batchResult = await brain.addMany({ - items: testItems, - parallel: true, - chunkSize: 100, - onProgress: (done, total) => { - if (done % 200 === 0) console.log(` ${done}/${total}`) - } -}) - -const duration = Date.now() - startTime -console.log(`\n✅ Batch add complete:`) -console.log(` Success: ${batchResult.successful.length}`) -console.log(` Failed: ${batchResult.failed.length}`) -console.log(` Duration: ${duration}ms`) -console.log(` Throughput: ${(batchResult.successful.length / (duration / 1000)).toFixed(0)} entities/sec`) - -// 6. PRODUCTION STATISTICS -console.log('\n\n📊 Production Statistics:\n') - -const stats = brain.getStats() -console.log(`Total Entities: ${stats.entities.total.toLocaleString()}`) -console.log(`Total Relationships: ${stats.relationships.totalRelationships.toLocaleString()}`) -console.log(`Graph Density: ${stats.density.toFixed(4)}`) - -console.log(`\nEntities by Type:`) -Object.entries(stats.entities.byType) - .sort(([, a], [, b]) => (b as number) - (a as number)) - .slice(0, 5) - .forEach(([type, count]) => { - console.log(` ${type}: ${(count as number).toLocaleString()}`) - }) - -// 7. QUERY PERFORMANCE MONITORING -console.log('\n\n⚡ Query Performance:\n') - -const perfStart = Date.now() -const searchResults = await brain.find({ - query: 'artificial intelligence machine learning', - where: { category: 'AI' }, - limit: 100, - explain: true -}) -const perfDuration = Date.now() - perfStart - -console.log(`Query completed in ${perfDuration}ms`) -console.log(` Results: ${searchResults.length}`) -console.log(` Avg score: ${(searchResults.reduce((sum, r) => sum + r.score, 0) / searchResults.length).toFixed(3)}`) - -// Show top result explanation -if (searchResults[0]?.explanation) { - console.log(`\n Top result score breakdown:`) - console.log(` Vector: ${searchResults[0].explanation.vector?.toFixed(3) || 'N/A'}`) - console.log(` Metadata: ${searchResults[0].explanation.metadata?.toFixed(3) || 'N/A'}`) - console.log(` Graph: ${searchResults[0].explanation.graph?.toFixed(3) || 'N/A'}`) -} - -// 8. CLEANUP & BEST PRACTICES -console.log('\n\n🧹 Production Best Practices:\n') - -// Always flush before shutdown -await brain.flush() -console.log('✅ Flushed all data to storage') - -// Get final stats -const finalStats = brain.getStats() -console.log(`✅ Final entity count: ${finalStats.entities.total.toLocaleString()}`) - -// Clean shutdown -await brain.close() -console.log('✅ Brain closed cleanly') - -console.log('\n\n🎓 Production Deployment Complete!') -console.log('\n📚 Key Production Learnings:') -console.log(' 1. Use filesystem storage and snapshot the data directory off-site from your scheduler') -console.log(' 2. Batch operations = 100x faster than individual ops') -console.log(' 3. Metadata query optimization for complex filters') -console.log(' 4. Monitor query performance with explain: true') -console.log(' 5. Always flush() before shutdown') -console.log(' 6. Use getStats() for O(1) counts (no expensive scans)') -console.log(' 7. Stream large imports with progress callbacks') -``` - -### Key Concepts - -#### 1. **Storage Options Comparison** - -| Storage | Use Case | Performance | Setup | -|---------|----------|-------------|-------| -| Memory | Dev/testing | Fastest | Zero config | -| Filesystem | Production | Fast | Local path + scheduled off-site backup | - -#### 2. **Off-Site Backup** - -```bash -# Cron / systemd timer / k8s CronJob — pick whatever you already operate -*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup -*/15 * * * * aws s3 sync /var/lib/brainy s3://my-bucket/brainy-backup -*/15 * * * * gsutil rsync -r /var/lib/brainy gs://my-bucket/brainy-backup -``` - -Brainy itself never reaches out to an object store. Snapshot `path` from your scheduler. - -#### 3. **Performance Optimization** - -```typescript -// 1. Use batch operations -await brain.addMany({ items, parallel: true, chunkSize: 100 }) - -// 2. Enable caching -const brain = new Brainy({ - cache: { maxSize: 10000, ttl: 600000 } -}) - -// 3. Use metadata indexes for filtering -await brain.find({ - where: { status: 'active' }, // Uses MetadataIndexManager - limit: 100 -}) - -// 4. Optimize query plans -const plan = await brain.getOptimalQueryPlan(filters) -// Use plan to choose best query strategy - -// 5. Use writeOnly for bulk imports -await brain.add({ - data, - type, - writeOnly: true // Skip validation for speed -}) -``` - -#### 4. **Import Strategies** - -```typescript -// Small files (<10MB) - Direct import -await brain.import('./data.csv') - -// Large files (>10MB) - Stream with progress -await brain.import('./large-data.csv', { - batchSize: 1000, - onProgress: (done, total) => { - console.log(`${(done/total*100).toFixed(1)}%`) - } -}) - -// Very large files (>100MB) - External pipeline -// Use streaming pipeline API for max control -``` - -#### 5. **Monitoring & Observability** - -```typescript -// 1. Track query performance -const start = Date.now() -const results = await brain.find({ query, explain: true }) -const duration = Date.now() - start -console.log(`Query: ${duration}ms, Results: ${results.length}`) - -// 2. Monitor graph statistics -const stats = brain.getStats() -console.log(`Density: ${stats.density}`) // Relationships per entity - -// 3. Track field cardinality -const fieldStats = await brain.getFieldStatistics() -// High cardinality fields = good for filtering - -// 4. Enable verbose logging in dev -const brain = new Brainy({ verbose: true }) -``` - -### Production Checklist - -#### Before Deployment - -- [ ] Provision a writable `path` on the host -- [ ] Set up an off-site snapshot job (`rclone` / `aws s3 sync` / `gsutil rsync`) -- [ ] Configure caching -- [ ] Test batch operations -- [ ] Benchmark query performance -- [ ] Set up monitoring - -#### During Operation - -- [ ] Monitor query latency -- [ ] Track entity/relationship counts -- [ ] Watch for outliers -- [ ] Optimize slow queries -- [ ] Regular backups - -#### Scaling Considerations - -- [ ] Shard data by service/tenant -- [ ] Use read replicas for queries -- [ ] Implement rate limiting -- [ ] Monitor storage costs -- [ ] Plan for growth - -### Practice Exercises - -1. Deploy Brainy with filesystem storage + scheduled off-site backup -2. Import a 10,000 row CSV file -3. Measure query performance for different filters -4. Optimize a slow query using getOptimalQueryPlan() -5. Set up monitoring dashboard -6. Test restore from off-site snapshot - ---- - -## 🎓 Graduation: You're a Brainy Expert! - -### What You've Mastered - -✅ **Level 1**: Basic operations (add, find, search) -✅ **Level 2**: Relationships & batch operations -✅ **Level 3**: Triple Intelligence & Neural AI -✅ **Level 4**: Virtual Filesystem -✅ **Level 5**: Production deployment - -### Next Steps - -#### Advanced Topics - -- **Multi-instance Deployments**: Run multiple Brainy processes behind your own routing layer -- **Custom Augmentations**: Extend Brainy with plugins -- **Streaming Pipelines**: Real-time data ingestion -- **Security**: Encryption, access control, audit logs -- **Framework Integration**: React, Vue, Next.js, Nuxt - -#### Resources - -- 📚 [API Reference](../api/README.md) - Complete API documentation -- 📁 [VFS Guide](../vfs/VFS_API_GUIDE.md) - Virtual Filesystem deep dive -- 🤖 [Neural API](../guides/neural-api.md) - Advanced neural operations -- 💬 [Discord Community](https://discord.gg/brainy) - Get help, share projects - -#### Share Your Success - -Built something cool with Brainy? Share it with the community! - -- GitHub: https://github.com/soulcraft/brainy -- Twitter: @brainydb -- Discord: https://discord.gg/brainy - ---- - -**Congratulations! You're now a Brainy expert ready to build production neural database applications! 🎉** diff --git a/docs/EXTENDING_STORAGE.md b/docs/EXTENDING_STORAGE.md new file mode 100644 index 00000000..8ecc700a --- /dev/null +++ b/docs/EXTENDING_STORAGE.md @@ -0,0 +1,396 @@ +# 🔌 Extending Brainy Storage with Augmentations + +## Overview + +Brainy's zero-config system is **fully extensible**. Augmentations can register new storage providers, presets, and auto-detection logic that integrates seamlessly with the existing system. + +## How Storage Extensions Work + +### 1. Storage Provider Registration + +When an augmentation is installed, it can register a new storage provider: + +```typescript +import { StorageProvider, registerStorageAugmentation } from '@soulcraft/brainy/config' + +const redisProvider: StorageProvider = { + type: 'redis', + name: 'Redis Storage', + description: 'High-performance in-memory data store', + priority: 10, // Higher priority = checked first in auto-detection + + // Auto-detection logic + async detect(): Promise { + // Check if Redis is available + if (process.env.REDIS_URL) { + try { + const redis = await import('ioredis') + const client = new redis.default(process.env.REDIS_URL) + await client.ping() + await client.quit() + return true + } catch { + return false + } + } + return false + }, + + // Configuration builder + async getConfig(): Promise { + return { + type: 'redis', + redisStorage: { + url: process.env.REDIS_URL, + prefix: 'brainy:', + ttl: 3600 + } + } + } +} + +// Register the provider +registerStorageAugmentation(redisProvider) +``` + +### 2. Using Extended Storage + +Once registered, the new storage type works with zero-config: + +```typescript +// Auto-detection will now check Redis +const brain = new Brainy() // Will use Redis if available! + +// Or explicitly specify +const brain = new Brainy({ storage: 'redis' }) + +// Or with custom config +const brain = new Brainy({ + storage: { + type: 'redis', + redisStorage: { + url: 'redis://localhost:6379', + prefix: 'myapp:' + } + } +}) +``` + +## Real-World Examples + +### Redis Augmentation + +```typescript +// @soulcraft/brainy-redis package +export class RedisStorageAugmentation { + async init() { + // Register the storage provider + registerStorageAugmentation({ + type: 'redis', + name: 'Redis Storage', + priority: 10, + + async detect() { + return !!(process.env.REDIS_URL || process.env.REDIS_HOST) + }, + + async getConfig() { + return { + type: 'redis', + redisStorage: { + url: process.env.REDIS_URL || + `redis://${process.env.REDIS_HOST}:${process.env.REDIS_PORT || 6379}` + } + } + } + }) + + // Register Redis-specific presets + registerPresetAugmentation('redis-cache', { + storage: 'redis', + model: ModelPrecision.Q8, + features: ['core', 'cache'], + distributed: true, + description: 'Redis-backed cache layer', + category: PresetCategory.SERVICE + }) + } +} +``` + +### MongoDB Augmentation + +```typescript +// @soulcraft/brainy-mongodb package +export class MongoStorageAugmentation { + async init() { + registerStorageAugmentation({ + type: 'mongodb', + name: 'MongoDB Storage', + priority: 8, + + async detect() { + return !!(process.env.MONGODB_URI || process.env.MONGO_URL) + }, + + async getConfig() { + return { + type: 'mongodb', + mongoStorage: { + uri: process.env.MONGODB_URI, + database: 'brainy', + collection: 'vectors' + } + } + } + }) + } +} +``` + +### PostgreSQL + pgvector Augmentation + +```typescript +// @soulcraft/brainy-postgres package +export class PostgresStorageAugmentation { + async init() { + registerStorageAugmentation({ + type: 'postgres', + name: 'PostgreSQL + pgvector', + priority: 9, + + async detect() { + const url = process.env.DATABASE_URL + if (url?.includes('postgres')) { + // Check for pgvector extension + const client = new Client({ connectionString: url }) + await client.connect() + const result = await client.query( + "SELECT * FROM pg_extension WHERE extname = 'vector'" + ) + await client.end() + return result.rows.length > 0 + } + return false + }, + + async getConfig() { + return { + type: 'postgres', + postgresStorage: { + connectionString: process.env.DATABASE_URL, + table: 'brainy_vectors' + } + } + } + }) + } +} +``` + +## Auto-Detection Priority + +Storage providers are checked in priority order: + +1. **Custom providers** (highest priority first) +2. **Cloud storage** (S3, GCS, R2) +3. **Database storage** (Redis, MongoDB, PostgreSQL) +4. **Local storage** (filesystem, OPFS) +5. **Memory** (fallback) + +```typescript +// Example priority chain +Redis (priority: 10) → PostgreSQL (9) → MongoDB (8) → S3 (5) → Filesystem (1) → Memory (0) +``` + +## Creating Custom Presets + +Augmentations can also register new presets: + +```typescript +registerPresetAugmentation('redis-cluster', { + storage: 'redis', + model: ModelPrecision.Q8, + features: ['core', 'cache', 'cluster'], + distributed: true, + role: DistributedRole.HYBRID, + cache: { + hotCacheMaxSize: 100000, // Large distributed cache + autoTune: true + }, + description: 'Redis Cluster configuration', + category: PresetCategory.SERVICE +}) + +// Users can then use: +const brain = new Brainy('redis-cluster') +``` + +## Type Safety with Extensions + +To maintain type safety with dynamic storage types: + +```typescript +// Augmentation declares its types +declare module '@soulcraft/brainy' { + interface StorageTypes { + redis: { + url: string + prefix?: string + ttl?: number + } + } + + interface PresetNames { + 'redis-cache': 'redis-cache' + 'redis-cluster': 'redis-cluster' + } +} +``` + +## Best Practices for Storage Augmentations + +1. **Always provide auto-detection** - Check environment variables and connectivity +2. **Set appropriate priority** - Higher for specialized storage, lower for general +3. **Handle failures gracefully** - Return false from detect() if not available +4. **Document requirements** - List required packages and environment variables +5. **Provide presets** - Include common configuration patterns +6. **Maintain compatibility** - Ensure model precision matches across instances + +## Example: Complete Redis Augmentation + +```typescript +import { + StorageProvider, + registerStorageAugmentation, + registerPresetAugmentation, + PresetCategory, + ModelPrecision, + DistributedRole +} from '@soulcraft/brainy/config' +import Redis from 'ioredis' + +export class BrainyRedisAugmentation { + private client: Redis + + async init() { + // Register storage provider + registerStorageAugmentation({ + type: 'redis', + name: 'Redis Vector Storage', + description: 'Redis with RediSearch for vector similarity', + priority: 10, + + requirements: { + env: ['REDIS_URL'], + packages: ['ioredis', 'redis'] + }, + + async detect() { + if (!process.env.REDIS_URL) return false + + try { + const client = new Redis(process.env.REDIS_URL) + + // Check for RediSearch module + const modules = await client.call('MODULE', 'LIST') + const hasRediSearch = modules.some(m => m[1] === 'search') + + await client.quit() + return hasRediSearch + } catch { + return false + } + }, + + async getConfig() { + return { + type: 'redis', + redisStorage: { + url: process.env.REDIS_URL, + prefix: process.env.REDIS_PREFIX || 'brainy:', + index: process.env.REDIS_INDEX || 'brainy-vectors', + ttl: process.env.REDIS_TTL ? parseInt(process.env.REDIS_TTL) : undefined + } + } + } + }) + + // Register presets + this.registerPresets() + } + + private registerPresets() { + // Fast cache preset + registerPresetAugmentation('redis-fast-cache', { + storage: 'redis' as any, + model: ModelPrecision.Q8, + features: ['core', 'cache', 'search'], + distributed: false, + cache: { + hotCacheMaxSize: 10000, + autoTune: true + }, + description: 'Redis-backed fast cache', + category: PresetCategory.SERVICE + }) + + // Distributed cache preset + registerPresetAugmentation('redis-distributed', { + storage: 'redis' as any, + model: ModelPrecision.AUTO, + features: ['core', 'cache', 'search', 'cluster'], + distributed: true, + role: DistributedRole.HYBRID, + cache: { + hotCacheMaxSize: 50000, + autoTune: true + }, + description: 'Redis distributed cache cluster', + category: PresetCategory.SERVICE + }) + + // Session store preset + registerPresetAugmentation('redis-sessions', { + storage: 'redis' as any, + model: ModelPrecision.Q8, + features: ['core', 'cache'], + distributed: false, + cache: { + hotCacheMaxSize: 5000, + autoTune: false + }, + description: 'Redis session storage', + category: PresetCategory.SERVICE + }) + } +} + +// Usage after installing the augmentation: +import { Brainy } from '@soulcraft/brainy' +import '@soulcraft/brainy-redis' // Registers the augmentation + +// Now Redis is automatically detected! +const brain = new Brainy() // Uses Redis if REDIS_URL is set + +// Or use a Redis preset +const brain = new Brainy('redis-fast-cache') + +// Or explicitly configure +const brain = new Brainy({ + storage: 'redis', + model: ModelPrecision.FP32 +}) +``` + +## Summary + +The extensible configuration system allows: + +1. **New storage types** via `registerStorageAugmentation()` +2. **Custom presets** via `registerPresetAugmentation()` +3. **Auto-detection logic** that integrates with zero-config +4. **Type-safe extensions** with TypeScript declarations +5. **Priority-based selection** for intelligent defaults + +This ensures Brainy can grow with new storage technologies while maintaining its zero-configuration philosophy! \ No newline at end of file diff --git a/docs/FIND_SYSTEM.md b/docs/FIND_SYSTEM.md index 1cc38ce9..1f9e2602 100644 --- a/docs/FIND_SYSTEM.md +++ b/docs/FIND_SYSTEM.md @@ -1,49 +1,29 @@ ---- -title: The Find System -slug: guides/find-system -public: true -category: guides -template: guide -order: 3 -description: Complete guide to Brainy's find() method — four intelligence systems, query execution phases, NLP patterns, and performance from 1K to 10M entities. -next: - - concepts/triple-intelligence - - api/reference ---- - # Brainy's Find System - Complete Guide ## Overview Brainy's `find()` method is the most advanced query system in any vector database, combining **Triple Intelligence** (vector + metadata + graph) with **Type-Aware NLP** for natural language understanding. -## Architecture: Four Intelligence Systems +## Architecture: Three Intelligence Systems ### 1. Vector Intelligence (HNSW Index) - **Purpose**: Semantic similarity search using embeddings - **Algorithm**: Hierarchical Navigable Small World (HNSW) -- **Performance**: O(log n) search +- **Performance**: O(log n) search, ~1.8ms typical - **Data Structure**: Multi-layer graph with 16 connections per node - **Use Cases**: "Find similar documents", "Content like this" -### 2. Text Intelligence (Word Index) - **Purpose**: Keyword/exact text matching -- **Algorithm**: Inverted word index with FNV-1a hashing -- **Performance**: O(log C) where C = chunks (~50 values each) -- **Data Structure**: `__words__ → hash → Roaring Bitmap of entity IDs` -- **Use Cases**: "Find exact name", "Keyword search" -- **Integration**: Automatically combined with Vector via RRF fusion - -### 3. Metadata Intelligence (Incremental Indices) +### 2. Metadata Intelligence (Incremental Indices) - **Purpose**: Fast filtering on structured data - **Algorithm**: HashMap for exact matches, Sorted arrays for ranges -- **Performance**: O(1) exact, O(log n) ranges +- **Performance**: O(1) exact, O(log n) ranges, <1ms typical - **Data Structure**: `Map>` + sorted value arrays - **Use Cases**: "Documents from 2023", "Status equals active" -### 4. Graph Intelligence (Adjacency Maps) +### 3. Graph Intelligence (Adjacency Maps) - **Purpose**: Relationship traversal and connection analysis - **Algorithm**: Pure O(1) neighbor lookups via Map operations -- **Performance**: O(1) per hop (measured <1 ms per neighbor lookup, validated up to 1M relationships — `tests/performance/graph-scale-performance.test.ts:238`) +- **Performance**: O(1) per hop, ~0.1ms typical - **Data Structure**: `Map>` - **Use Cases**: "Papers connected to MIT", "Authors who collaborated" @@ -66,29 +46,29 @@ await brain.find("documents by Smith published after 2020 with high citations") ```typescript // Direct query objects with full control await brain.find({ - // Vector search - query: "machine learning research", - - // Metadata filters - where: { - publishDate: { greaterThan: 2020 }, - citations: { between: [50, 1000] }, - status: "published" - }, - - // Type constraints - type: NounType.Document, - - // Graph traversal - connected: { - to: "mit-ai-lab", - via: VerbType.AffiliatedWith, - depth: 2 - }, - - // Control options - limit: 20, - explain: true // Get scoring breakdown + // Vector search + query: "machine learning research", + + // Metadata filters + where: { + publishDate: { greaterThan: 2020 }, + citations: { between: [50, 1000] }, + status: "published" + }, + + // Type constraints + type: NounType.Document, + + // Graph traversal + connected: { + to: "mit-ai-lab", + via: VerbType.AffiliatedWith, + depth: 2 + }, + + // Control options + limit: 20, + explain: true // Get scoring breakdown }) ``` @@ -96,92 +76,14 @@ await brain.find({ ```typescript // Find entities similar to a specific item await brain.find({ - near: { - id: "doc-123", - threshold: 0.8 // Minimum similarity - }, - type: NounType.Document + near: { + id: "doc-123", + threshold: 0.8 // Minimum similarity + }, + type: NounType.Document }) ``` -### 4. Hybrid Search -```typescript -// Zero-config hybrid: automatically combines text + semantic search -await brain.find({ query: "David Smith" }) -// Uses Reciprocal Rank Fusion (RRF) to combine results - -// Force text-only search -await brain.find({ query: "exact keyword", searchMode: 'text' }) - -// Force semantic-only search -await brain.find({ query: "AI concepts", searchMode: 'semantic' }) - -// Custom hybrid weighting (0 = text only, 1 = semantic only) -await brain.find({ query: "search term", hybridAlpha: 0.3 }) -``` - -**How Auto-Alpha Works:** -- Short queries (1-2 words): alpha = 0.3 (favor text matching) -- Medium queries (3-4 words): alpha = 0.5 (balanced) -- Long queries (5+ words): alpha = 0.7 (favor semantic matching) - -### 5. Match Visibility - -Search results include match details showing what matched: - -```typescript -const results = await brain.find({ query: 'david the warrior' }) - -// Each result has: -results[0].textMatches // ["david", "warrior"] - exact words found -results[0].textScore // 0.25 - text match quality (0-1) -results[0].semanticScore // 0.87 - semantic similarity (0-1) -results[0].matchSource // 'both' | 'text' | 'semantic' -``` - -Use this for: -- **Highlighting** exact matches in UI (textMatches) -- **Explaining** why a result was found (matchSource) -- **Debugging** search behavior (separate scores) - -### 6. Semantic Highlighting - -Highlight which concepts/words in text matched your query: - -```typescript -// Find semantically similar words + exact matches -const highlights = await brain.highlight({ - query: "david the warrior", - text: "David Smith is a brave fighter who battles dragons" -}) - -// Returns: -// [ -// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' }, -// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' }, -// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' } -// ] -``` - -**Features:** -- `matchType: 'text'` - Exact word match (score = 1.0) -- `matchType: 'semantic'` - Concept match (score varies) -- `position` - [start, end] for precise highlighting -- `granularity` - 'word' (default), 'phrase', or 'sentence' -- `threshold` - Minimum semantic score (default: 0.5) - -**UI Usage Pattern:** -```typescript -// Highlight search results with different styles -function highlightResult(text: string, highlights: Highlight[]) { - return highlights.map(h => ({ - text: h.text, - position: h.position, - style: h.matchType === 'text' ? 'strong' : 'emphasis' // Different UI styles - })) -} -``` - ## Index Usage in Detail ### Metadata Index Operations @@ -194,7 +96,7 @@ function highlightResult(text: string, highlights: Highlight[]) { // Memory: ~40 bytes per unique field-value combination ``` -#### Sorted Index (Range Queries) +#### Sorted Index (Range Queries) ```typescript // Query: {publishDate: {greaterThan: 2020}} // Index: sortedIndices.get("publishDate") → [[2019, Set], [2020, Set], [2021, Set]] @@ -220,7 +122,7 @@ function highlightResult(text: string, highlights: Highlight[]) { // Query: {query: "machine learning"} // Process: // 1. Embed query text → 384-dimensional vector -// 2. Start at top layer (entry point) +// 2. Start at top layer (entry point) // 3. Greedy search for nearest neighbor at each layer // 4. Move down layers for progressively finer search // 5. Return k nearest neighbors with similarity scores @@ -233,10 +135,10 @@ function highlightResult(text: string, highlights: Highlight[]) { #### O(1) Neighbor Lookups ```typescript // Query: {connected: {to: "entity-123"}} -// Index lookup: -// - Outgoing: sourceIndex.get("entity-123") → Set -// - Incoming: targetIndex.get("entity-123") → Set -// - Both directions: union of both Sets +// Index lookup: +// - Outgoing: sourceIndex.get("entity-123") → Set +// - Incoming: targetIndex.get("entity-123") → Set +// - Both directions: union of both Sets // Performance: O(1) per hop, no matter the graph size // Memory: ~24 bytes per relationship (source + target + metadata) ``` @@ -246,19 +148,19 @@ function highlightResult(text: string, highlights: Highlight[]) { ### Phase 1: Query Parsing ```typescript if (typeof query === 'string') { - // Natural Language Processing - const nlpParams = await this.parseNaturalQuery(query) - - // Type-aware parsing: - // 1. Detect NounType using pre-embedded type vectors - // 2. Get type-specific fields from real data patterns - // 3. Semantic field matching with type affinity boosting - // 4. Validate field-type compatibility - // 5. Generate optimized query plan - - params = nlpParams + // Natural Language Processing + const nlpParams = await this.parseNaturalQuery(query) + + // Type-aware parsing: + // 1. Detect NounType using pre-embedded type vectors + // 2. Get type-specific fields from real data patterns + // 3. Semantic field matching with type affinity boosting + // 4. Validate field-type compatibility + // 5. Generate optimized query plan + + params = nlpParams } else { - params = query // Direct structured query + params = query // Direct structured query } ``` @@ -269,12 +171,12 @@ const searchPromises = [] // Vector search (if query text or vector provided) if (params.query || params.vector) { - searchPromises.push(this.executeVectorSearch(params)) + searchPromises.push(this.executeVectorSearch(params)) } -// Proximity search (if near parameter provided) +// Proximity search (if near parameter provided) if (params.near) { - searchPromises.push(this.executeProximitySearch(params)) + searchPromises.push(this.executeProximitySearch(params)) } // Wait for all searches to complete @@ -285,41 +187,41 @@ const searchResults = await Promise.all(searchPromises) ```typescript // Apply metadata filters using optimized indices if (params.where || params.type || params.service) { - const filter = { - ...params.where, - ...(params.type && { noun: params.type }), - ...(params.service && { service: params.service }) - } - - // Get optimal query plan based on field cardinalities - const queryPlan = await this.getOptimalQueryPlan(filter) - - // Execute filters in optimal order (low cardinality first) - const filteredIds = await this.metadataIndex.getIdsForFilter(filter) - - if (results.length > 0) { - // Intersect with vector search results - results = results.filter(r => filteredIds.includes(r.id)) - } else { - // Create results from metadata matches (metadata-only query) - results = await this.createResultsFromIds(filteredIds) - } + const filter = { + ...params.where, + ...(params.type && { noun: params.type }), + ...(params.service && { service: params.service }) + } + + // Get optimal query plan based on field cardinalities + const queryPlan = await this.getOptimalQueryPlan(filter) + + // Execute filters in optimal order (low cardinality first) + const filteredIds = await this.metadataIndex.getIdsForFilter(filter) + + if (results.length > 0) { + // Intersect with vector search results + results = results.filter(r => filteredIds.includes(r.id)) + } else { + // Create results from metadata matches (metadata-only query) + results = await this.createResultsFromIds(filteredIds) + } } ``` -### Phase 4: Graph Traversal +### Phase 4: Graph Traversal ```typescript // Apply graph constraints using O(1) lookups if (params.connected) { - const connectedIds = await this.graphIndex.getConnectedIds(params.connected) - - if (results.length > 0) { - // Filter existing results to only connected entities - results = results.filter(r => connectedIds.includes(r.id)) - } else { - // Create results from connected entities - results = await this.createResultsFromIds(connectedIds) - } + const connectedIds = await this.graphIndex.getConnectedIds(params.connected) + + if (results.length > 0) { + // Filter existing results to only connected entities + results = results.filter(r => connectedIds.includes(r.id)) + } else { + // Create results from connected entities + results = await this.createResultsFromIds(connectedIds) + } } ``` @@ -327,12 +229,12 @@ if (params.connected) { ```typescript // Combine scores from multiple intelligence sources if (params.fusion && results.length > 0) { - results = this.applyFusionScoring(results, params.fusion) - - // Example fusion strategies: - // - Weighted: vectorScore × 0.6 + metadataScore × 0.2 + graphScore × 0.2 - // - Adaptive: adjust weights based on query characteristics - // - Progressive: prioritize based on result confidence + results = this.applyFusionScoring(results, params.fusion) + + // Example fusion strategies: + // - Weighted: vectorScore × 0.6 + metadataScore × 0.2 + graphScore × 0.2 + // - Adaptive: adjust weights based on query characteristics + // - Progressive: prioritize based on result confidence } // Sort by final score and apply pagination @@ -343,7 +245,7 @@ return results.slice(offset, offset + limit) ## Type-Aware NLP Features ### 1. Dynamic Field Discovery -- **No Hardcoded Fields**: Only NounType/VerbType taxonomies are fixed (42 noun, 127 verb types) +- **No Hardcoded Fields**: Only NounType/VerbType taxonomies are fixed (30+ noun, 40+ verb types) - **Real Data Learning**: Field affinity learned from actual indexed entities - **Semantic Matching**: "by" → "author" via embedding similarity (87% confidence) - **Type Context**: Documents have different fields than Persons or Organizations @@ -363,7 +265,7 @@ return results.slice(offset, offset + limit) ### 3. Field-Type Validation ```typescript // Prevents invalid queries and suggests alternatives: -// "people with publishDate > 2020" +// "people with publishDate > 2020" // → Warning: "Person entities rarely have publishDate field" // → Suggestion: "Did you mean createdAt, updatedAt, or birthDate?" // → Auto-correction: Use most likely alternative based on affinity data @@ -373,40 +275,36 @@ return results.slice(offset, offset + limit) ### Query Performance by Type -| Query Type | Index Used | Complexity | Example | -|------------|------------|------------|---------| -| **Semantic Search** | HNSW Vector | O(log n) | `"AI research papers"` | -| **Exact Metadata** | HashMap | O(1) | `{status: "published"}` | -| **Range Metadata** | Sorted Array | O(log n) | `{year: {greaterThan: 2020}}` | -| **Graph Traversal** | Adjacency Map | O(1) | `{connected: {to: "mit"}}` | -| **Type Detection** | Pre-embedded Types | O(t) | `"documents"` → `NounType.Document` | -| **Field Matching** | Field Embeddings | O(f) | `"by"` → `"author"` | -| **Combined Query** | All Indices | O(log n) | NLP + filters + graph | +| Query Type | Index Used | Performance | Example | +|------------|------------|-------------|---------| +| **Semantic Search** | HNSW Vector | O(log n), ~1.8ms | `"AI research papers"` | +| **Exact Metadata** | HashMap | O(1), ~0.8ms | `{status: "published"}` | +| **Range Metadata** | Sorted Array | O(log n), ~0.6ms | `{year: {greaterThan: 2020}}` | +| **Graph Traversal** | Adjacency Map | O(1), ~0.1ms | `{connected: {to: "mit"}}` | +| **Type Detection** | Pre-embedded Types | O(t), ~0.3ms | `"documents"` → `NounType.Document` | +| **Field Matching** | Field Embeddings | O(f), ~0.1ms | `"by"` → `"author"` | +| **Combined Query** | All Indices | O(log n), ~1.8ms | NLP + filters + graph | Where: - n = number of entities in database -- t = number of types (169 total: 42 noun + 127 verb) +- t = number of types (70 total: 30 noun + 40 verb) - f = number of fields for detected entity type (typically 5-15) -Absolute latencies depend on hardware, embedding model, and storage backend; the columns above describe how each stage scales. The graph stage is the one path with a committed scale benchmark (see [Scalability](#scalability)). - ### Scalability -The cost of each query stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion: - -| Query stage | Complexity | Scaling behavior | -|-------------|------------|------------------| -| Metadata filter (exact) | O(1) | Constant — independent of dataset size | -| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results | -| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers | -| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) | -| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) | +| Database Size | Vector Search | Metadata Filter | Graph Query | Combined | +|---------------|---------------|-----------------|-------------|----------| +| **1K entities** | 0.8ms | 0.3ms | 0.05ms | 1.1ms | +| **10K entities** | 1.2ms | 0.5ms | 0.08ms | 1.5ms | +| **100K entities** | 1.8ms | 0.8ms | 0.1ms | 2.1ms | +| **1M entities** | 2.5ms | 1.2ms | 0.1ms | 2.8ms | +| **10M entities** | 3.8ms | 1.8ms | 0.1ms | 4.2ms | **Key Performance Notes:** - Graph queries stay O(1) regardless of scale - Metadata ranges scale as O(log n), not O(n) - Vector search degrades gracefully due to HNSW -- Type-aware NLP adds minimal overhead (single embedding pass, no full scan) +- Type-aware NLP adds minimal overhead (~0.4ms) ## Example Query Flows @@ -417,27 +315,27 @@ The cost of each query stage is governed by its algorithmic complexity, not a fi // Phase 1: NLP Parsing // - "papers" → NounType.Document (0.94 confidence) // - "Stanford researchers" → entity search + NounType.Person -// - "recent" → publishDate: {greaterThan: 2023} +// - "recent" → publishDate: {greaterThan: 2023} // - "high citations" → citations: {greaterThan: 100} // - "connected to industry" → graph traversal via VerbType.AffiliatedWith // Phase 2: Generated Query { - type: NounType.Document, - where: { - publishDate: { greaterThan: 2023 }, - citations: { greaterThan: 100 } - }, - connected: { - to: ["stanford-researchers"], - via: VerbType.AffiliatedWith, - depth: 2 - } + type: NounType.Document, + where: { + publishDate: { greaterThan: 2023 }, + citations: { greaterThan: 100 } + }, + connected: { + to: ["stanford-researchers"], + via: VerbType.AffiliatedWith, + depth: 2 + } } // Phase 3: Execution Plan // 1. Metadata filter: publishDate > 2023 (O(log n) via sorted index) -// 2. Metadata filter: citations > 100 (O(log n) via sorted index) +// 2. Metadata filter: citations > 100 (O(log n) via sorted index) // 3. Graph traversal: connected to Stanford (O(1) per hop) // 4. Intersection: entities matching all constraints // 5. Sort by relevance score @@ -447,977 +345,19 @@ The cost of each query stage is governed by its algorithmic complexity, not a fi ```typescript // Query: Direct structured query for maximum performance await brain.find({ - type: NounType.Document, // O(1) type filter - where: { - status: "published", // O(1) exact match - year: 2024, // O(1) exact match - citations: { greaterThan: 50 } // O(log n) range query - }, - connected: { - from: "author-123", // O(1) graph lookup - via: VerbType.Creates - }, - limit: 10 + type: NounType.Document, // O(1) type filter + where: { + status: "published", // O(1) exact match + year: 2024, // O(1) exact match + citations: { greaterThan: 50 } // O(log n) range query + }, + connected: { + from: "author-123", // O(1) graph lookup + via: VerbType.Creates + }, + limit: 10 }) -// Executes as O(1) type filter + O(1)/O(log n) metadata + O(1) graph lookup — no full scan +// Total performance: ~1.2ms for 100K entities ``` -## Filter Syntax Reference - -### Where Clause: Complete Operator Guide - -Brainy provides a comprehensive set of operators for filtering entities by metadata fields. All operators work seamlessly with Triple Intelligence (vector + metadata + graph). - -#### Basic Operators - -**Exact Match** (shorthand): -```typescript -await brain.find({ - where: { - status: 'active', // Shorthand for { eq: 'active' } - year: 2024, // Exact match for numbers - verified: true // Boolean matching - } -}) -``` - -**Comparison Operators**: -```typescript -await brain.find({ - where: { - age: { gt: 18 }, // Greater than - score: { gte: 80 }, // Greater than or equal - price: { lt: 100 }, // Less than - stock: { lte: 10 }, // Less than or equal - status: { eq: 'active' }, // Equals (explicit) - role: { ne: 'guest' } // Not equals - } -}) -``` - -**Performance**: O(log n) for comparisons using sorted indices, O(1) for exact matches using hash maps. - -#### Range Operators - -**Between** (inclusive): -```typescript -await brain.find({ - where: { - publishDate: { between: [2020, 2024] }, // Year range - price: { between: [10.00, 99.99] }, // Price range - timestamp: { between: [startMs, endMs] } // Time range - } -}) -``` - -**Performance**: O(log n) for finding range boundaries, O(k) for collecting results where k = matching entities. - -#### Set Membership - -**In/Not In**: -```typescript -await brain.find({ - where: { - category: { in: ['tech', 'science', 'research'] }, - status: { notIn: ['draft', 'deleted'] }, - priority: { in: [1, 2, 3] } - } -}) -``` - -**Performance**: O(1) per set member check via hash lookup, O(m) total where m = set size. - -#### String Matching - -**Contains/Starts/Ends**: -```typescript -await brain.find({ - where: { - title: { contains: 'machine learning' }, // Substring search - email: { startsWith: 'admin@' }, // Prefix match - filename: { endsWith: '.pdf' } // Suffix match - } -}) -``` - -**Performance**: O(n) substring scan (not indexed), best used with additional indexed filters. - -**Note**: For semantic similarity, use `query` parameter instead: -```typescript -// ❌ Slow substring search -where: { description: { contains: 'AI' } } - -// ✅ Fast semantic search -query: 'artificial intelligence' -``` - -#### Existence Checks - -**Exists/Missing**: -```typescript -await brain.find({ - where: { - email: { exists: true }, // Has email field - deletedAt: { exists: false }, // No deletedAt field (not deleted) - profileImage: { exists: true } // Has profile image - } -}) -``` - -**Performance**: O(1) via hash index of fields. - -### Compound Filters - -Combine multiple conditions with boolean logic: - -#### AND Logic (Default) - -All conditions at the same level are implicitly AND: - -```typescript -await brain.find({ - where: { - status: 'published', // AND - year: { gte: 2020 }, // AND - citations: { gte: 50 } // AND - } -}) -// Returns: entities matching ALL three conditions -``` - -**Explicit AND with `allOf`**: -```typescript -await brain.find({ - where: { - allOf: [ - { status: 'published' }, - { year: { gte: 2020 } }, - { citations: { gte: 50 } } - ] - } -}) -``` - -**Performance**: O(log n) total - processes filters in optimal order (low cardinality first). - -#### OR Logic - -Match ANY condition: - -```typescript -await brain.find({ - where: { - anyOf: [ - { status: 'urgent' }, - { priority: { gte: 8 } }, - { assignee: 'admin' } - ] - } -}) -// Returns: entities matching ANY condition -``` - -**Combined AND + OR**: -```typescript -await brain.find({ - where: { - status: 'active', // Must be active - anyOf: [ // AND (urgent OR high priority) - { tags: { contains: 'urgent' } }, - { priority: { gte: 8 } } - ] - } -}) -``` - -**Performance**: O(m × log n) where m = number of OR conditions, results are merged with Set union. - -#### Nested Logic - -Complex boolean expressions: - -```typescript -await brain.find({ - where: { - allOf: [ - { status: 'published' }, - { - anyOf: [ - { featured: true }, - { citations: { gte: 100 } } - ] - } - ] - } -}) -// Returns: published AND (featured OR highly cited) -``` - -### Complete Operator Reference Table - -| **Operator** | **Aliases** | **Description** | **Performance** | **Example** | -|--------------|-------------|-----------------|-----------------|-------------| -| `eq` | `equals` | Exact equality | O(1) | `{ status: { eq: 'active' } }` | -| `ne` | `notEquals` | Not equal | O(n) scan | `{ role: { ne: 'admin' } }` | -| `gt` | `greaterThan` | Greater than | O(log n) | `{ age: { gt: 18 } }` | -| `gte` | `greaterThanOrEqual` | Greater/equal | O(log n) | `{ score: { gte: 80 } }` | -| `lt` | `lessThan` | Less than | O(log n) | `{ price: { lt: 100 } }` | -| `lte` | `lessThanOrEqual` | Less/equal | O(log n) | `{ stock: { lte: 10 } }` | -| `in` | - | In array | O(m) | `{ category: { in: ['A', 'B'] } }` | -| `notIn` | - | Not in array | O(n) scan | `{ status: { notIn: ['draft'] } }` | -| `between` | - | Range (inclusive) | O(log n + k) | `{ year: { between: [2020, 2024] } }` | -| `contains` | - | Substring | O(n) scan | `{ title: { contains: 'AI' } }` | -| `startsWith` | - | Prefix | O(n) scan | `{ email: { startsWith: 'admin' } }` | -| `endsWith` | - | Suffix | O(n) scan | `{ file: { endsWith: '.pdf' } }` | -| `exists` | - | Field exists | O(1) | `{ email: { exists: true } }` | -| `anyOf` | - | OR logic | O(m × log n) | `{ anyOf: [{...}, {...}] }` | -| `allOf` | - | AND logic | O(log n) | `{ allOf: [{...}, {...}] }` | - -**Performance Notes**: -- **O(1)**: Hash index lookup (exact matches, exists) -- **O(log n)**: Sorted index binary search (comparisons, ranges) -- **O(n)**: Full scan (string matching, negations) -- **O(k)**: Result collection where k = matches - -**Optimization Tips**: -1. **Combine fast + slow filters**: Put indexed filters first -2. **Avoid `ne` and `notIn`**: Require full scans, use positive filters when possible -3. **Use `query` for text search**: Semantic search is faster than substring matching -4. **Limit string operations**: `contains`/`startsWith`/`endsWith` are unindexed - -### Type Filtering - -Filter entities by NounType: - -#### Single Type - -```typescript -await brain.find({ - type: NounType.Document, - where: { year: { gte: 2020 } } -}) -``` - -#### Multiple Types - -```typescript -await brain.find({ - type: [NounType.Person, NounType.Organization], - where: { verified: true } -}) -``` - -#### All 42 Available NounTypes - -```typescript -// People & Organizations -NounType.Person, NounType.Organization, NounType.Team, NounType.Role - -// Content -NounType.Document, NounType.Image, NounType.Video, NounType.Audio - -// Knowledge -NounType.Concept, NounType.Topic, NounType.Category, NounType.Tag - -// Technical -NounType.Code, NounType.API, NounType.Database, NounType.Service - -// Events & Time -NounType.Event, NounType.Timeline, NounType.Schedule - -// Location & Physical -NounType.Place, NounType.Building, NounType.Room, NounType.Device - -// Abstract -NounType.Thing, NounType.Entity, NounType.Object - -// And 19 more... (see src/types/graphTypes.ts for complete list) -``` - -**Performance**: O(1) - type stored as indexed metadata field. - -### Graph Query Syntax - -Traverse relationships using the GraphIndex: - -#### Basic Connection - -```typescript -await brain.find({ - connected: { - to: 'entity-id-123', // Connected to this entity - via: VerbType.WorksFor, // Through this relationship type - direction: 'out' // Direction: 'in', 'out', or 'both' - } -}) -``` - -**Performance**: O(1) per hop via adjacency map lookup. - -#### Multi-Hop Traversal - -```typescript -await brain.find({ - connected: { - to: 'research-institution', - via: VerbType.AffiliatedWith, - depth: 2 // Up to 2 hops away - } -}) -``` - -**Performance**: O(d) where d = depth, each hop is O(1). - -#### Combined with Other Filters - -```typescript -await brain.find({ - type: NounType.Person, - where: { - verified: true, - reputation: { gte: 100 } - }, - connected: { - to: 'stanford-ai-lab', - via: VerbType.WorksAt, - direction: 'out' - }, - limit: 20 -}) -// Returns: Verified people with high reputation who work at Stanford AI Lab -``` - -#### Pagination with Graph Queries - -```typescript -// Page through high-degree nodes efficiently -const neighbors = await brain.graphIndex.getNeighbors('hub-entity-id', { - direction: 'out', - limit: 50, - offset: 0 -}) - -// Get verb IDs with pagination -const verbIds = await brain.graphIndex.getVerbIdsBySource('source-id', { - limit: 100, - offset: 0 -}) -``` - -**Performance**: O(1) lookup + O(log k) slice where k = total neighbors. - -**Note**: See `src/graph/graphAdjacencyIndex.ts` for low-level graph operations. - -### Sorting Results - -Sort query results by any field, including timestamps: - -```typescript -// Sort by timestamp (descending - newest first) -await brain.find({ - type: NounType.Document, - orderBy: 'createdAt', - order: 'desc', - limit: 10 -}) - -// Sort by custom field (ascending) -await brain.find({ - where: { status: { eq: 'published' } }, - orderBy: 'priority', - order: 'asc' -}) - -// Sort with filtering and pagination -await brain.find({ - where: { - publishDate: { gte: startDate }, - citations: { gte: 50 } - }, - orderBy: 'citations', - order: 'desc', - limit: 20, - offset: 0 -}) -``` - -**Sorting Performance**: -- **Production-scale**: O(k log k) where k = filtered results -- **Memory**: O(k) for filtered set, independent of total entity count -- **Timestamp fields**: Exact millisecond precision (createdAt, updatedAt) -- **Works with**: Metadata-only queries and vector + metadata queries -- **Default order**: `asc` if not specified - -**Timestamp Sorting**: -```typescript -// Range query + sorting -await brain.find({ - where: { - createdAt: { gte: Date.now() - 86400000 } // Last 24 hours - }, - orderBy: 'createdAt', - order: 'desc' // Newest first -}) - -// Works with updatedAt, accessed, modified -await brain.find({ - orderBy: 'updatedAt', - order: 'desc' -}) -``` - -**Advanced Sorting Examples**: -```typescript -// Sort search results by custom field instead of relevance -await brain.find({ - query: "machine learning", - where: { publishDate: { gte: 2023 } }, - orderBy: 'citations', // Sort by citations, not relevance - order: 'desc' -}) - -// Paginated sorted results -async function getDocumentsByDate(page: number, pageSize: number = 20) { - return await brain.find({ - type: NounType.Document, - where: { status: { eq: 'published' } }, - orderBy: 'publishDate', - order: 'desc', - limit: pageSize, - offset: page * pageSize - }) -} -``` - -## Common Query Patterns - -### Pagination - -**Offset-based pagination**: -```typescript -async function getPaginatedResults(page: number, pageSize: number = 20) { - return await brain.find({ - type: NounType.Document, - where: { status: 'published' }, - orderBy: 'createdAt', - order: 'desc', - limit: pageSize, - offset: page * pageSize - }) -} - -// Usage -const page1 = await getPaginatedResults(0) // First 20 -const page2 = await getPaginatedResults(1) // Next 20 -``` - -**Graph pagination**: -```typescript -// Paginate through high-degree node relationships -async function getNeighborPage(entityId: string, page: number, pageSize: number = 50) { - return await brain.graphIndex.getNeighbors(entityId, { - direction: 'out', - limit: pageSize, - offset: page * pageSize - }) -} -``` - -**Performance**: O(1) for offset calculation, O(k) for slice where k = page size. - -### Time-based Queries - -**Recent entities**: -```typescript -// Last 24 hours -const oneDayAgo = Date.now() - (24 * 60 * 60 * 1000) -await brain.find({ - where: { - createdAt: { gte: oneDayAgo } - }, - orderBy: 'createdAt', - order: 'desc' -}) - -// Last 7 days with additional filters -const oneWeekAgo = Date.now() - (7 * 24 * 60 * 60 * 1000) -await brain.find({ - type: NounType.Document, - where: { - createdAt: { gte: oneWeekAgo }, - status: 'published' - }, - orderBy: 'createdAt', - order: 'desc' -}) -``` - -**Date ranges**: -```typescript -// Specific year -await brain.find({ - where: { - publishDate: { between: [ - new Date('2023-01-01').getTime(), - new Date('2023-12-31').getTime() - ]} - } -}) - -// Quarter -const Q1_2024_start = new Date('2024-01-01').getTime() -const Q1_2024_end = new Date('2024-03-31').getTime() -await brain.find({ - where: { - createdAt: { between: [Q1_2024_start, Q1_2024_end] } - } -}) -``` - -### Combining Vector + Metadata + Graph - -**Triple Intelligence query**: -```typescript -// Find: AI research papers from verified authors at top institutions -const results = await brain.find({ - // Vector search (semantic) - query: 'artificial intelligence machine learning', - - // Metadata filters - type: NounType.Document, - where: { - publishDate: { gte: 2020 }, - citations: { gte: 50 }, - peerReviewed: true - }, - - // Graph traversal - connected: { - to: topInstitutionIds, // Array of institution entity IDs - via: VerbType.AffiliatedWith, - depth: 2 // Authors affiliated with institutions (2 hops) - }, - - // Results - limit: 50, - orderBy: 'citations', - order: 'desc' -}) -``` - -**Performance**: O(log n) vector search + O(log n) metadata filters + O(1) graph traversal — bounded by the vector stage. - -### Excluding Soft-Deleted Entities - -**Common pattern**: -```typescript -// Standard query excludes deleted -await brain.find({ - where: { - deletedAt: { exists: false } // Not soft-deleted - } -}) - -// Or use compound filter -await brain.find({ - where: { - allOf: [ - { status: 'active' }, - { deletedAt: { exists: false } } - ] - } -}) -``` - -**Note**: Consider implementing this as a default filter in your application layer if all queries need it. - -### Finding Similar Entities - -**Semantic similarity**: -```typescript -// Find documents similar to a specific document -await brain.find({ - near: { - id: 'doc-123', - threshold: 0.8 // Minimum 80% similarity - }, - type: NounType.Document, - limit: 10 -}) - -// With metadata constraints -await brain.find({ - near: { id: 'paper-456', threshold: 0.75 }, - where: { - publishDate: { gte: 2020 }, - language: 'en' - } -}) -``` - -**Performance**: O(log n) HNSW search with early termination at threshold. - -### Aggregation Patterns - -**Count matching entities**: -```typescript -// Get total count (metadata-only query is fastest) -const results = await brain.find({ - where: { status: 'published' }, - limit: 1 // We only need the count -}) -// Note: Current API returns results, not counts -// For production, consider caching counts or using metadata indices directly -``` - -**Group by type**: -```typescript -// Find all entities, then group by type in application -const allEntities = await brain.find({ limit: 10000 }) -const byType = allEntities.reduce((acc, entity) => { - const type = entity.noun || 'unknown' - if (!acc[type]) acc[type] = [] - acc[type].push(entity) - return acc -}, {}) -``` - -### Multi-Condition OR Queries - -**Any of multiple values**: -```typescript -await brain.find({ - where: { - anyOf: [ - { priority: 'urgent' }, - { priority: 'high' }, - { assignee: 'admin' }, - { dueDate: { lte: Date.now() } } - ] - } -}) -// Returns: urgent OR high priority OR assigned to admin OR overdue -``` - -**Complex business logic**: -```typescript -// Find: (Premium users OR trial users with activity) AND not banned -await brain.find({ - type: NounType.Person, - where: { - allOf: [ - { - anyOf: [ - { subscription: 'premium' }, - { - allOf: [ - { subscription: 'trial' }, - { lastActive: { gte: Date.now() - 86400000 } } // 24h - ] - } - ] - }, - { banned: { ne: true } } - ] - } -}) -``` - -## Troubleshooting Guide - -### Query Returns No Results - -**Check 1: Verify entity exists** -```typescript -// List all entities of a type -const all = await brain.find({ - type: NounType.Document, - limit: 10 -}) -console.log(`Found ${all.length} documents`) -``` - -**Check 2: Test filters individually** -```typescript -// Remove filters one by one to find the culprit -await brain.find({ where: { status: 'published' } }) // Works? -await brain.find({ where: { year: 2024 } }) // Works? -await brain.find({ where: { - status: 'published', - year: 2024 // Combined - works? -}}) -``` - -**Check 3: Verify field names** -```typescript -// Get a sample entity to see actual field names -const sample = await brain.find({ type: NounType.Document, limit: 1 }) -console.log(Object.keys(sample[0].data)) // Actual fields -``` - -**Common issues**: -- Field name typo: `publishDate` vs `published_date` -- Wrong type: `type: NounType.Document` but entities are `NounType.Paper` -- Case sensitivity: `status: 'Active'` vs `status: 'active'` - -### Slow Query Performance - -**Check 1: Identify slow operation** -```typescript -// Use explain mode (if available) -const results = await brain.find({ - query: 'machine learning', - where: { title: { contains: 'AI' } }, // ⚠️ O(n) substring search - explain: true -}) -``` - -**Check 2: Avoid O(n) operations** -```typescript -// ❌ Slow: Substring search -where: { description: { contains: 'machine' } } - -// ✅ Fast: Semantic search -query: 'machine learning' - -// ❌ Slow: Negation -where: { status: { ne: 'draft' } } - -// ✅ Fast: Positive filter -where: { status: 'published' } -``` - -**Check 3: Optimize filter order** -```typescript -// ❌ Suboptimal: Slow filter first -where: { - description: { contains: 'AI' }, // O(n) - runs first - year: 2024 // O(1) - runs second -} - -// ✅ Optimal: Fast filter first (automatic optimization) -where: { - year: 2024, // O(1) - narrow results - status: 'published' // O(1) - further narrow - // Only then apply O(n) operations if needed -} -``` - -**Performance budget**: -- **< 2ms**: Metadata-only or graph-only queries -- **< 5ms**: Vector search with simple filters -- **< 10ms**: Complex Triple Intelligence queries -- **> 10ms**: Check for O(n) operations or missing indices - -### Type Errors - -**TypeScript type mismatches**: -```typescript -// ❌ Error: Type 'string' is not assignable to type 'NounType' -await brain.find({ type: 'Document' }) - -// ✅ Correct: Use NounType enum -import { NounType } from '@soulcraft/brainy' -await brain.find({ type: NounType.Document }) - -// ❌ Error: Operator not recognized -where: { age: { greaterThan: 18 } } // Old API - -// ✅ Correct: Use canonical operators -where: { age: { gt: 18 } } -``` - -### Graph Traversal Issues - -**No connected entities found**: -```typescript -// Verify relationship exists -const relations = await brain.related({ - from: 'entity-a', - to: 'entity-b' -}) -console.log('Relationships:', relations) - -// Check direction -await brain.find({ - connected: { - to: 'entity-id', - direction: 'in' // Try 'out' or 'both' - } -}) - -// Verify verb type -await brain.find({ - connected: { - to: 'entity-id', - via: VerbType.WorksFor // Correct VerbType? - } -}) -``` - -### Vector Search Not Working - -**Check embeddings**: -```typescript -// Ensure vectors are generated (automatic in v5.0+) -const entity = await brain.get('entity-id') -console.log('Has vector:', !!entity.vector) - -// If missing, entity may predate vector support -// Re-add entity to generate vector -await brain.update(entity.id, { data: entity.data }) -``` - -**Similarity threshold too high**: -```typescript -// ❌ Too strict: May return nothing -await brain.find({ - near: { id: 'doc-123', threshold: 0.95 } -}) - -// ✅ Reasonable: 0.7-0.85 is typical -await brain.find({ - near: { id: 'doc-123', threshold: 0.75 } -}) -``` - -### Unexpected Results - -**Entity appears in wrong type query**: -```typescript -// Check actual entity type -const entity = await brain.get('unexpected-id') -console.log('Entity type:', entity.noun) - -// Verify type filter is working -await brain.find({ - type: NounType.Document, - where: { id: 'unexpected-id' } // Should not return if wrong type -}) -``` - -**Duplicate results**: -```typescript -// Check for duplicate entity IDs -const results = await brain.find({ query: 'test' }) -const ids = results.map(r => r.id) -const uniqueIds = new Set(ids) -console.log(`Results: ${results.length}, Unique: ${uniqueIds.size}`) - -// Brainy should never return duplicates - report if found -``` - -## VFS (Virtual File System) Visibility - -### Default Behavior - -**VFS entities are now part of the knowledge graph** and included in query results by default: - -```typescript -// Default: Searches ALL entities including VFS files -await brain.find({ query: 'authentication setup' }) -// Returns: concepts, papers, AND markdown documentation files -``` - -**Why this change?** VFS files (imported markdown, PDFs, etc.) ARE knowledge entities. When you import documentation or papers into Brainy, you want to search them! - -### Excluding VFS Entities - -If you need to exclude VFS entities from specific queries, use the `excludeVFS` parameter: - -```typescript -// Exclude VFS files from results -await brain.find({ - query: 'machine learning', - excludeVFS: true // Only return non-file entities -}) -``` - -**Alternative**: Use explicit where clause for more control: - -```typescript -// Explicit filtering (same as excludeVFS: true) -await brain.find({ - query: 'machine learning', - where: { vfsType: { exists: false } } -}) - -// Or only search VFS files -await brain.find({ - query: 'setup instructions', - where: { vfsType: 'file' } // Only files -}) -``` - -### Performance - -**VFS filtering is production-scale:** -- Uses MetadataIndex (O(1) for exists checks) -- No performance penalty - same speed as any metadata filter -- Works seamlessly with vector + metadata + graph queries - -## 5. Aggregate Queries - -The `find()` method also supports aggregate queries via the `aggregate` parameter. When set, `find()` bypasses all vector/metadata/graph search paths and returns pre-computed aggregate results from the incremental aggregation engine. - -```typescript -// Define the aggregate (once — survives restarts) -brain.defineAggregate({ - name: 'monthly_spending', - source: { type: NounType.Event, where: { domain: 'financial' } }, - groupBy: ['category', { field: 'date', window: 'month' }], - metrics: { - total: { op: 'sum', field: 'amount' }, - count: { op: 'count' } - } -}) - -// Query it through find() -const results = await brain.find({ - aggregate: 'monthly_spending', - where: { category: 'food' }, // Filters aggregate groups (not raw entities) - orderBy: 'total', - order: 'desc', - limit: 12 -}) -``` - -**Key characteristics:** -- **O(1) read performance** — results come from running totals, no query-time computation -- **Updated incrementally** — every `add()`, `update()`, `delete()` updates matching aggregates -- **Same Result[] format** — aggregate results are returned as `NounType.Measurement` entities -- **Combinable with `where`/`orderBy`/`limit`/`offset`** — standard find() parameters apply to group filtering - -See the **[API Reference → Aggregation Engine](./api/README.md#aggregation-engine)** for the full API. - ---- - -### Migration from v4.6.x - -**BREAKING CHANGE**: The `includeVFS` parameter has been removed: - -```typescript -// ❌ Old (v4.6.x and earlier) -await brain.find({ - query: 'docs', - includeVFS: true // No longer needed! -}) - -// ✅ New -await brain.find({ - query: 'docs' // VFS included by default -}) - -// ✅ To exclude VFS (if needed) -await brain.find({ - query: 'concepts', - excludeVFS: true -}) -``` - -**Why removed?** The old `includeVFS` parameter was: -1. Broken (metadata filter incompatibility with storage adapters) -2. Confusing (double-negative logic) -3. Wrong default (VFS should be searchable) - This system represents the most advanced query intelligence available in any database, combining the speed of specialized indices with the intelligence of natural language understanding and the power of graph relationships. \ No newline at end of file diff --git a/docs/METADATA_CONTRACT_IMPLEMENTATION.md b/docs/METADATA_CONTRACT_IMPLEMENTATION.md new file mode 100644 index 00000000..0020990f --- /dev/null +++ b/docs/METADATA_CONTRACT_IMPLEMENTATION.md @@ -0,0 +1,177 @@ +# Metadata Contract Implementation Plan + +## New Required Interface + +```typescript +export interface BrainyAugmentation { + // Identity + name: string + timing: 'before' | 'after' | 'around' | 'replace' + operations: string[] + priority: number + + // REQUIRED metadata contract + metadata: 'none' | 'readonly' | MetadataAccess + + // Methods + initialize(context: AugmentationContext): Promise + execute(operation: string, params: any, next: () => Promise): Promise + shouldExecute?(operation: string, params: any): boolean + shutdown?(): Promise +} + +interface MetadataAccess { + reads?: string[] | '*' // Fields to read, or '*' for all + writes?: string[] | '*' // Fields to write, or '*' for all + namespace?: string // Optional: custom namespace like '_myAug' +} +``` + +## Augmentation Analysis & Classification + +### Category 1: No Metadata Access ('none') +These augmentations don't read or write metadata at all: + +1. **CacheAugmentation** - Only caches search results +2. **RequestDeduplicatorAugmentation** - Only deduplicates requests +3. **ConnectionPoolAugmentation** - Only manages storage connections +4. **StorageAugmentation** - Base storage layer, metadata handled by Brainy + +### Category 2: Read-Only Access ('readonly') +These augmentations read metadata but never modify it: + +6. **IndexAugmentation** - Reads metadata to build indexes +7. **MonitoringAugmentation** - Reads metadata for monitoring +8. **MetricsAugmentation** - Reads metadata for metrics collection +9. **BatchProcessingAugmentation** - Reads metadata to check for external IDs +10. **EntityRegistryAugmentation** - Reads metadata to register entities +11. **AutoRegisterEntitiesAugmentation** - Reads metadata for auto-registration +12. **ConduitAugmentation** - Reads metadata to pass through operations + +### Category 3: Metadata Writers (needs specific access) +These augmentations modify metadata and need specific field declarations: + +13. **SynapseAugmentation** - Writes to '_synapse' field + ```typescript + metadata: { + reads: '*', + writes: ['_synapse', '_synapseTimestamp'], + namespace: '_synapse' // Uses its own namespace + } + ``` + +14. **IntelligentVerbScoringAugmentation** - Adds scoring to verbs + ```typescript + metadata: { + reads: ['type', 'verb', 'source', 'target'], + writes: ['weight', 'confidence', 'intelligentScoring'] + } + ``` + +15. **ServerSearchAugmentation** - Might add server metadata + ```typescript + metadata: { + reads: '*', + writes: ['_server', '_syncedAt'] + } + ``` + +16. **NeuralImportAugmentation** - Enriches imported data + ```typescript + metadata: { + reads: '*', + writes: ['nounType', 'verbType', '_importedAt', '_enriched'] + } + ``` + +### Category 4: API/Server (needs analysis) +17. **ApiServerAugmentation** - Likely read-only for serving data +18. **StorageAugmentations** (plural) - Collection of storage implementations +19. **ConduitAugmentations** (plural) - Collection of conduit types + +## Implementation Steps + +### Phase 1: Update Base Interface +1. Update `BrainyAugmentation` interface to require `metadata` field +2. Update `BaseAugmentation` class to have abstract `metadata` property +3. Add runtime enforcement in augmentation executor + +### Phase 2: Update Each Augmentation +For each augmentation, add the appropriate metadata declaration: + +#### Example Updates: + +**CacheAugmentation:** +```typescript +export class CacheAugmentation extends BaseAugmentation { + readonly name = 'cache' + readonly metadata = 'none' as const // ✅ No metadata access + // ... rest unchanged +} +``` + +```typescript + readonly metadata = 'readonly' as const // ✅ Only reads for logging + // ... rest unchanged +} +``` + +**SynapseAugmentation:** +```typescript +export abstract class SynapseAugmentation extends BaseAugmentation { + readonly name = 'synapse' + readonly metadata = { + reads: '*', + writes: ['_synapse', '_synapseTimestamp'], + namespace: '_synapse' + } as const + // ... rest unchanged +} +``` + +### Phase 3: Runtime Enforcement +Add a metadata access enforcer that: +1. Wraps metadata objects based on declared access +2. Throws errors if augmentation violates its contract +3. Logs warnings in development mode + +```typescript +class MetadataEnforcer { + enforce(augmentation: BrainyAugmentation, metadata: any): any { + if (augmentation.metadata === 'none') { + return null // No access at all + } + + if (augmentation.metadata === 'readonly') { + return Object.freeze(deepClone(metadata)) // Read-only copy + } + + // For specific access, create proxy that validates + return new Proxy(metadata, { + set(target, prop, value) { + const access = augmentation.metadata as MetadataAccess + if (!access.writes?.includes(String(prop)) && access.writes !== '*') { + throw new Error(`Augmentation '${augmentation.name}' cannot write to field '${String(prop)}'`) + } + target[prop] = value + return true + } + }) + } +} +``` + +## Benefits +1. **Type Safety** - TypeScript enforces metadata declaration +2. **Runtime Safety** - Violations caught immediately +3. **Documentation** - Contract shows exactly what each augmentation does +4. **Brain-cloud Ready** - Registry can validate augmentations +5. **Developer Friendly** - Most use simple 'none' or 'readonly' + +## Migration Checklist +- [ ] Update BrainyAugmentation interface +- [ ] Update BaseAugmentation class +- [ ] Add MetadataEnforcer +- [ ] Update all 19 augmentations with metadata declarations +- [ ] Add tests for metadata enforcement +- [ ] Update documentation \ No newline at end of file diff --git a/docs/MIGRATION-V3-TO-V4.md b/docs/MIGRATION-V3-TO-V4.md deleted file mode 100644 index 29c409ac..00000000 --- a/docs/MIGRATION-V3-TO-V4.md +++ /dev/null @@ -1,569 +0,0 @@ -# Brainy v3 → v4.0.0 Migration Guide - -> **Migration Complexity**: Low -> **Breaking Changes**: None (fully backward compatible) -> **New Features**: Lifecycle management, batch operations, compression, quota monitoring - -## Overview - -Brainy v4.0.0 is a **backward-compatible** release focused on production-ready cost optimization features. Your existing v3 code will continue to work without modifications, but you'll want to enable the new v4.0.0 features for significant cost savings. - -**Key Benefits of Upgrading:** -- 💰 **96% cost savings** with lifecycle policies -- 🚀 **1000x faster** bulk deletions with batch operations -- 📦 **60-80% space savings** with gzip compression -- 📊 **Real-time quota monitoring** for OPFS -- 🎯 **Zero downtime** migration - -## What's New in v4.0.0 - -### 1. Lifecycle Management (Cloud Storage) - -**Automatic tier transitions for massive cost savings:** - -```typescript -// NEW in v4.0.0 -await storage.setLifecyclePolicy({ - rules: [{ - id: 'archive-old-data', - prefix: 'entities/', - status: 'Enabled', - transitions: [ - { days: 30, storageClass: 'STANDARD_IA' }, - { days: 90, storageClass: 'GLACIER' } - ] - }] -}) -``` - -**Supported on:** -- ✅ AWS S3 (Lifecycle + Intelligent-Tiering) -- ✅ Google Cloud Storage (Lifecycle + Autoclass) -- ✅ Azure Blob Storage (Lifecycle policies) - -### 2. Batch Operations - -**1000x faster bulk deletions:** - -```typescript -// v3: Delete one at a time (slow, expensive) -for (const id of idsToDelete) { - await brain.remove(id) // 1000 API calls for 1000 entities -} - -// v4.0.0: Batch delete (fast, cheap) -const paths = idsToDelete.flatMap(id => [ - `entities/nouns/vectors/${id.substring(0, 2)}/${id}.json`, - `entities/nouns/metadata/${id.substring(0, 2)}/${id}.json` -]) -await storage.batchDelete(paths) // 1 API call for 1000 objects (S3) -``` - -**Efficiency gains:** -- S3: 1000 objects per batch -- GCS: 100 objects per batch -- Azure: 256 objects per batch - -### 3. Compression (FileSystem) - -**60-80% space savings for local storage:** - -```typescript -// NEW in v4.0.0 -const brain = new Brainy({ - storage: { - type: 'filesystem', - path: './data', - compression: true // Enable gzip compression - } -}) - -// Automatic compression/decompression on all reads/writes -``` - -### 4. Quota Monitoring (OPFS) - -**Prevent quota exceeded errors in browsers:** - -```typescript -// NEW in v4.0.0 -const status = await storage.getStorageStatus() - -if (status.details.usagePercent > 80) { - console.warn('Approaching quota limit:', status.details) - // Take action: cleanup old data, notify user, etc. -} -``` - -### 5. Tier Management (Azure) - -**Manual or automatic tier transitions:** - -```typescript -// NEW in v4.0.0 -await storage.changeBlobTier(blobPath, 'Cool') // Hot → Cool (50% savings) -await storage.batchChangeTier([blob1, blob2], 'Archive') // 99% savings - -// Rehydrate from Archive when needed -await storage.rehydrateBlob(blobPath, 'High') // 1-hour rehydration -``` - -## Storage Architecture Changes - -### v3.x Storage Structure - -``` -brainy-data/ -├── nouns/ -│ └── {uuid}.json # Single file per entity -├── verbs/ -│ └── {uuid}.json # Single file per relationship -├── metadata/ -│ └── __metadata_*.json # Indexes -└── _system/ - └── statistics.json -``` - -### v4.0.0 Storage Structure (Automatic Migration) - -``` -brainy-data/ -├── entities/ -│ ├── nouns/ -│ │ ├── vectors/ # Vector + HNSW graph (NEW) -│ │ │ ├── 00/ ... ff/ # 256 UUID shards (NEW) -│ │ └── metadata/ # Business data (NEW) -│ │ ├── 00/ ... ff/ # 256 UUID shards (NEW) -│ └── verbs/ -│ ├── vectors/ # Relationship vectors (NEW) -│ │ ├── 00/ ... ff/ -│ └── metadata/ # Relationship data (NEW) -│ ├── 00/ ... ff/ -└── _system/ # Unchanged - └── __metadata_*.json -``` - -**Key Changes:** -1. **Metadata/Vector Separation**: Entities split into 2 files for optimal I/O -2. **UUID-Based Sharding**: 256 shards for cloud storage optimization -3. **Automatic Migration**: Brainy handles migration transparently on first run - -## Migration Steps - -### Step 1: Update Brainy Package - -```bash -npm install @soulcraft/brainy@latest -``` - -**Check your version:** -```bash -npm list @soulcraft/brainy -# Should show: @soulcraft/brainy@4.0.0 -``` - -### Step 2: No Code Changes Required! ✅ - -Your existing v3 code will work without modifications: - -```typescript -// This v3 code works perfectly in v4.0.0 -const brain = new Brainy({ - storage: { type: 'filesystem', path: './data' } -}) - -await brain.init() -await brain.add("content", { type: "entity" }) -const results = await brain.search("query") -``` - -### Step 3: First Run (Automatic Migration) - -On first initialization with v4.0.0: - -1. **Brainy detects v3 storage structure** -2. **Transparently migrates to v4.0.0 structure**: - - Creates `entities/` directory - - Migrates `nouns/` → `entities/nouns/vectors/` + `entities/nouns/metadata/` - - Migrates `verbs/` → `entities/verbs/vectors/` + `entities/verbs/metadata/` - - Applies UUID-based sharding -3. **Old structure preserved** (optional cleanup later) - -**Migration time:** -- 10K entities: ~1 minute -- 100K entities: ~10 minutes -- 1M entities: ~2 hours - -**Zero downtime:** -- Migration happens during init() -- No data loss -- Automatic rollback on error - -### Step 4: Enable v4.0.0 Features (Optional but Recommended) - -#### Enable Lifecycle Policies (Cloud Storage) - -**AWS S3:** -```typescript -// After init() -await storage.setLifecyclePolicy({ - rules: [{ - id: 'optimize-storage', - prefix: 'entities/', - status: 'Enabled', - transitions: [ - { days: 30, storageClass: 'STANDARD_IA' }, - { days: 90, storageClass: 'GLACIER' } - ] - }] -}) - -// Or use Intelligent-Tiering (recommended) -await storage.enableIntelligentTiering('entities/', 'auto-optimize') -``` - -**Google Cloud Storage:** -```typescript -await storage.enableAutoclass({ - terminalStorageClass: 'ARCHIVE' -}) -``` - -**Azure Blob Storage:** -```typescript -await storage.setLifecyclePolicy({ - rules: [{ - name: 'optimize-blobs', - enabled: true, - type: 'Lifecycle', - definition: { - filters: { blobTypes: ['blockBlob'] }, - actions: { - baseBlob: { - tierToCool: { daysAfterModificationGreaterThan: 30 }, - tierToArchive: { daysAfterModificationGreaterThan: 90 } - } - } - } - }] -}) -``` - -#### Enable Compression (FileSystem) - -```typescript -const brain = new Brainy({ - storage: { - type: 'filesystem', - path: './data', - compression: true // NEW: 60-80% space savings - } -}) -``` - -#### Use Batch Operations - -```typescript -// Replace individual deletes with batch delete -const idsToDelete = [/* ... */] -const paths = idsToDelete.flatMap(id => { - const shard = id.substring(0, 2) - return [ - `entities/nouns/vectors/${shard}/${id}.json`, - `entities/nouns/metadata/${shard}/${id}.json` - ] -}) - -await storage.batchDelete(paths) // Much faster! -``` - -#### Monitor Quota (OPFS) - -```typescript -// Periodically check quota in browser apps -setInterval(async () => { - const status = await storage.getStorageStatus() - if (status.details.usagePercent > 80) { - notifyUser('Storage approaching limit') - } -}, 60000) // Check every minute -``` - -## Backward Compatibility - -### Guaranteed to Work (No Changes Needed) - -✅ All v3 APIs remain unchanged -✅ Storage adapters backward compatible -✅ Metadata structure unchanged -✅ Query APIs unchanged -✅ Configuration options unchanged - -### New Optional APIs (Add When Ready) - -- `storage.setLifecyclePolicy()` - NEW in v4.0.0 -- `storage.getLifecyclePolicy()` - NEW in v4.0.0 -- `storage.removeLifecyclePolicy()` - NEW in v4.0.0 -- `storage.enableIntelligentTiering()` - NEW in v4.0.0 (S3) -- `storage.enableAutoclass()` - NEW in v4.0.0 (GCS) -- `storage.batchDelete()` - NEW in v4.0.0 -- `storage.changeBlobTier()` - NEW in v4.0.0 (Azure) -- `storage.getStorageStatus()` - Enhanced in v4.0.0 - -## Testing Your Migration - -### 1. Test in Development First - -```typescript -// Create test brain with v4.0.0 -const testBrain = new Brainy({ - storage: { type: 'filesystem', path: './test-data' } -}) - -await testBrain.init() - -// Verify migration -console.log('Initialization complete') - -// Test basic operations -const id = await testBrain.add("test content", { type: "test" }) -const results = await testBrain.search("test") -console.log('Basic operations working:', results.length > 0) -``` - -### 2. Verify Storage Structure - -```bash -# Check new directory structure -ls -la ./test-data/entities/nouns/vectors/ -# Should see: 00/ 01/ 02/ ... ff/ (256 shards) - -ls -la ./test-data/entities/nouns/metadata/ -# Should see: 00/ 01/ 02/ ... ff/ (256 shards) -``` - -### 3. Verify Data Integrity - -```typescript -// Query all entities -const allEntities = await testBrain.find({}) -console.log('Total entities:', allEntities.length) - -// Verify specific entities -const entity = await testBrain.get(knownEntityId) -console.log('Entity retrieved:', entity !== null) -``` - -### 4. Test Performance - -```typescript -// Benchmark search -const start = Date.now() -const results = await testBrain.search("query") -const duration = Date.now() - start -console.log('Search time:', duration, 'ms') - -// Should be similar or faster than v3 -``` - -## Rollback Procedure (If Needed) - -If you encounter issues, you can rollback: - -### Option 1: Rollback Package - -```bash -# Reinstall v3 -npm install @soulcraft/brainy@^3.50.0 - -# Restart application -``` - -**Important:** v3 can still read v3-structured data (preserved during migration) - -### Option 2: Restore from Backup - -```bash -# If you backed up data before migration -rm -rf ./data -cp -r ./data-backup ./data - -# Reinstall v3 -npm install @soulcraft/brainy@^3.50.0 -``` - -## Common Migration Scenarios - -### Scenario 1: Small Application (<10K Entities) - -**Migration time:** 1 minute -**Recommended approach:** -1. Update npm package -2. Restart application (automatic migration) -3. Enable lifecycle policies immediately - -### Scenario 2: Medium Application (10K-1M Entities) - -**Migration time:** 10 minutes - 2 hours -**Recommended approach:** -1. Backup data -2. Update npm package -3. Schedule maintenance window -4. Restart application (automatic migration) -5. Verify data integrity -6. Enable lifecycle policies - -### Scenario 3: Large Application (1M+ Entities) - -**Migration time:** 2-24 hours -**Recommended approach:** -1. **Backup data** (critical!) -2. Test migration on staging environment -3. Schedule extended maintenance window -4. Update npm package on production -5. Restart application (automatic migration) -6. Monitor migration progress -7. Verify data integrity thoroughly -8. Enable lifecycle policies gradually - -## Cost Savings After Migration - -### Enable All v4.0.0 Features - -**500TB Dataset Example:** - -**Before v4.0.0 (v3 with AWS S3 Standard):** -``` -Storage: $138,000/year -Operations: $5,000/year -Total: $143,000/year -``` - -**After v4.0.0 (with Intelligent-Tiering):** -``` -Storage: $51,000/year (64% savings) -Operations: $5,000/year -Total: $56,000/year -``` - -**After v4.0.0 (with Lifecycle Policies):** -``` -Storage: $5,940/year (96% savings!) -Operations: $5,000/year -Total: $10,940/year -``` - -**Annual Savings: $132,060 (96% reduction)** - -## Troubleshooting - -### Issue: Migration takes too long - -**Solution:** -- Migration is I/O bound -- For 1M+ entities, consider: - - Running during off-peak hours - - Using faster storage (SSD vs HDD) - - Increasing available memory - - Running on more powerful instance - -### Issue: "Storage structure not recognized" - -**Solution:** -```typescript -// Manually trigger migration -await brain.storage.migrateToV4() // If automatic migration fails - -// Or start fresh (data loss warning!) -await brain.storage.clear() -await brain.init() -``` - -### Issue: Lifecycle policy not working - -**Solution:** -```typescript -// Verify policy is set -const policy = await storage.getLifecyclePolicy() -console.log('Active rules:', policy.rules) - -// Cloud providers may take 24-48 hours to start transitions -// Check again after 2 days - -// Verify in cloud console: -// - AWS: S3 → Bucket → Management → Lifecycle -// - GCS: Storage → Bucket → Lifecycle -// - Azure: Storage Account → Lifecycle management -``` - -### Issue: Batch delete not working - -**Solution:** -```typescript -// Ensure storage adapter supports batch delete -const status = await storage.getStorageStatus() -console.log('Storage type:', status.type) - -// Batch delete requires: -// - S3CompatibleStorage ✅ -// - GcsStorage ✅ -// - AzureBlobStorage ✅ -// - FileSystemStorage ✅ -// - OPFSStorage ✅ -// - MemoryStorage ✅ -``` - -## Best Practices - -1. ✅ **Backup before upgrading** (especially for large datasets) -2. ✅ **Test on staging first** (verify migration works) -3. ✅ **Monitor during migration** (watch logs for errors) -4. ✅ **Enable lifecycle policies immediately** (start saving costs) -5. ✅ **Use batch operations** (for any bulk cleanup) -6. ✅ **Monitor quota** (OPFS browser apps) -7. ✅ **Enable compression** (FileSystem storage) - -## Getting Help - -**Documentation:** -- [AWS S3 Cost Optimization Guide](./operations/cost-optimization-aws-s3.md) -- [GCS Cost Optimization Guide](./operations/cost-optimization-gcs.md) -- [Azure Cost Optimization Guide](./operations/cost-optimization-azure.md) -- [Cloudflare R2 Cost Optimization Guide](./operations/cost-optimization-cloudflare-r2.md) - -**Support:** -- GitHub Issues: [https://github.com/soulcraft/brainy/issues](https://github.com/soulcraft/brainy/issues) -- GitHub Discussions: [https://github.com/soulcraft/brainy/discussions](https://github.com/soulcraft/brainy/discussions) - -## Summary - -**Migration Checklist:** -- ✅ Backup data -- ✅ Update npm package (`npm install @soulcraft/brainy@latest`) -- ✅ Restart application (automatic migration) -- ✅ Verify data integrity -- ✅ Enable lifecycle policies -- ✅ Enable compression (FileSystem) -- ✅ Use batch operations -- ✅ Monitor cost savings - -**Expected Results:** -- ✅ Zero downtime migration -- ✅ Full backward compatibility -- ✅ 60-96% cost savings -- ✅ 1000x faster bulk operations -- ✅ 60-80% space savings (with compression) - -**Timeline:** -- Small app (<10K): 1 minute migration -- Medium app (10K-1M): 10 minutes - 2 hours -- Large app (1M+): 2-24 hours - -**Welcome to Brainy v4.0.0! 🎉** - ---- - -**Version**: v4.0.0 -**Migration Difficulty**: Low -**Breaking Changes**: None -**Recommended Upgrade**: Yes (significant cost savings) diff --git a/docs/MODEL_LOADING_QUICK_REFERENCE.md b/docs/MODEL_LOADING_QUICK_REFERENCE.md new file mode 100644 index 00000000..f94e2723 --- /dev/null +++ b/docs/MODEL_LOADING_QUICK_REFERENCE.md @@ -0,0 +1,118 @@ +# 🤖 Model Loading Quick Reference + +## 🚀 Common Scenarios + +### ✅ Development (Zero Config) +```typescript +const brain = new Brainy() +await brain.init() // Downloads automatically (FP32 default) +``` + +### ⚡ Development (Optimized - v2.8.0+) +```typescript +// 75% smaller models, 99% accuracy +const brain = new Brainy({ + embeddingOptions: { dtype: 'q8' } +}) +await brain.init() +``` + +### 🐳 Docker Production +```dockerfile +# Both models (recommended) +RUN npm run download-models + +# Or FP32 only (compatibility) +RUN npm run download-models:fp32 + +# Or Q8 only (space-constrained) +RUN npm run download-models:q8 + +ENV BRAINY_ALLOW_REMOTE_MODELS=false +``` + +### ☁️ Serverless/Lambda +```bash +# Build step +npm run download-models + +# Runtime +export BRAINY_ALLOW_REMOTE_MODELS=false +``` + +### 🔒 Air-Gapped/Offline +```bash +# Connected machine +npm run download-models +tar -czf brainy-models.tar.gz ./models + +# Offline machine +tar -xzf brainy-models.tar.gz +export BRAINY_ALLOW_REMOTE_MODELS=false +``` + +### 🌐 Browser/CDN +```html + + +``` + +## 🚨 Troubleshooting + +| Error | Solution | +|-------|----------| +| "Failed to load embedding model" | `npm run download-models` | +| "ENOENT: no such file" | Check `BRAINY_MODELS_PATH` | +| "Network timeout" | Set `BRAINY_ALLOW_REMOTE_MODELS=false` | +| "Permission denied" | `chmod 755 ./models` | +| "Out of memory" | Increase container memory limit | + +## 🎯 Environment Variables + +| Variable | Values | Purpose | +|----------|--------|---------| +| `BRAINY_ALLOW_REMOTE_MODELS` | `true`/`false` | Allow/block downloads | +| `BRAINY_MODELS_PATH` | `./models` | Model storage path | +| `BRAINY_Q8_CONFIRMED` | `true`/`false` | Silence Q8 compatibility warnings | +| `NODE_ENV` | `production` | Environment detection | + +## 📦 Model Info + +### FP32 (Default) +- **Model**: All-MiniLM-L6-v2 +- **Dimensions**: 384 (fixed) +- **Size**: 90MB +- **Accuracy**: 100% (baseline) +- **Location**: `./models/Xenova/all-MiniLM-L6-v2/onnx/model.onnx` + +### Q8 (Optional - v2.8.0+) +- **Model**: All-MiniLM-L6-v2 (quantized) +- **Dimensions**: 384 (same) +- **Size**: 23MB (75% smaller!) +- **Accuracy**: ~99% (minimal loss) +- **Location**: `./models/Xenova/all-MiniLM-L6-v2/onnx/model_quantized.onnx` + +**⚠️ Important**: FP32 and Q8 create different embeddings and are incompatible! + +## ✅ Verification Commands + +```bash +# Check FP32 model exists +ls ./models/Xenova/all-MiniLM-L6-v2/onnx/model.onnx + +# Check Q8 model exists +ls ./models/Xenova/all-MiniLM-L6-v2/onnx/model_quantized.onnx + +# Test offline mode +BRAINY_ALLOW_REMOTE_MODELS=false npm test + +# Download fresh models (both) +rm -rf ./models && npm run download-models + +# Download specific model variant +rm -rf ./models && npm run download-models:q8 +``` \ No newline at end of file diff --git a/docs/NEURAL_API_PATTERNS.md b/docs/NEURAL_API_PATTERNS.md new file mode 100644 index 00000000..4edd3505 --- /dev/null +++ b/docs/NEURAL_API_PATTERNS.md @@ -0,0 +1,736 @@ +# 🧠 Neural API Patterns: AI-Powered Intelligence + +> Learn the correct patterns for Brainy's Neural API. Avoid performance pitfalls and use AI features effectively. + +## 🚨 Critical: Access Neural APIs Correctly + +### ❌ **WRONG - Outdated Access Patterns** + +```typescript +// DON'T DO THIS - Outdated documentation patterns +import { Brainy } from '@soulcraft/brainy' // ❌ Wrong import +const brain = new Brainy() // ❌ Old class name + +// These may not work as expected: +const neural = brain.neural // ❌ May be undefined +``` + +### ✅ **CORRECT - Modern Neural Access** + +```typescript +// ✅ Use modern Brainy class +import { Brainy } from '@soulcraft/brainy' + +const brain = new Brainy() +await brain.init() + +// ✅ Neural API is available after initialization +const clusters = await brain.neural.clusters() +const similarity = await brain.neural.similar('item1', 'item2') +``` + +## 🔍 Similarity Analysis Patterns + +### ❌ **WRONG - Inefficient Similarity Checks** + +```typescript +// DON'T DO THIS - N² comparisons +const items = await brain.find({ limit: 1000 }) +const similarities = [] + +for (const item1 of items) { + for (const item2 of items) { + if (item1.id !== item2.id) { + const sim = await brain.neural.similar(item1.id, item2.id) // ❌ Millions of calls + similarities.push({ from: item1.id, to: item2.id, score: sim }) + } + } +} +``` + +### ✅ **CORRECT - Efficient Similarity Patterns** + +```typescript +// ✅ Pattern 1: Find neighbors (much more efficient) +const item = await brain.get('target-item-id') +const neighbors = await brain.neural.neighbors(item.id, { + limit: 10, // Top 10 most similar + threshold: 0.7, // Minimum similarity + includeScores: true // Include similarity scores +}) + +console.log(`Found ${neighbors.length} similar items`) + +// ✅ Pattern 2: Batch similarity for specific pairs +const itemPairs = [ + ['item1', 'item2'], + ['item1', 'item3'], + ['item2', 'item3'] +] + +const similarities = await Promise.all( + itemPairs.map(async ([a, b]) => ({ + from: a, + to: b, + score: await brain.neural.similar(a, b) + })) +) + +// ✅ Pattern 3: Text-to-text similarity (no need for IDs) +const textSimilarity = await brain.neural.similar( + "Machine learning is fascinating", + "AI and deep learning are interesting", + { detailed: true } // Get explanation of similarity +) + +console.log(`Similarity: ${textSimilarity.score}`) +console.log(`Explanation: ${textSimilarity.explanation}`) + +// ✅ Pattern 4: Vector-level similarity for optimization +const vector1 = await brain.embed("First concept") +const vector2 = await brain.embed("Second concept") +const vectorSimilarity = await brain.neural.similar(vector1, vector2) +``` + +## 🎯 Clustering Patterns + +### ❌ **WRONG - Uncontrolled Clustering** + +```typescript +// DON'T DO THIS - Clustering everything without limits +const everything = await brain.find({ limit: 100000 }) // ❌ Too much data +const clusters = await brain.neural.clusters() // ❌ May crash or timeout +``` + +### ✅ **CORRECT - Smart Clustering Patterns** + +```typescript +// ✅ Pattern 1: Controlled clustering with limits +const recentItems = await brain.find({ + where: { + createdAt: { $gte: Date.now() - 30 * 24 * 60 * 60 * 1000 } // Last 30 days + }, + limit: 1000 // Reasonable limit +}) + +const clusters = await brain.neural.clusters( + recentItems.map(item => item.id), + { + algorithm: 'kmeans', // Reliable algorithm + maxClusters: 10, // Reasonable number + threshold: 0.75, // High similarity required + iterations: 50 // Convergence limit + } +) + +// ✅ Pattern 2: Domain-specific clustering +const techDocs = await brain.find({ + where: { category: 'technology', type: 'document' }, + limit: 500 +}) + +const techClusters = await brain.neural.clusterByDomain( + 'category', // Group by this field + { + items: techDocs.map(doc => doc.id), + minClusterSize: 3, // Minimum items per cluster + maxClusters: 8 + } +) + +// ✅ Pattern 3: Temporal clustering for time-series data +const timebasedClusters = await brain.neural.clusterByTime( + 'createdAt', // Time field + 'week', // Time window (hour, day, week, month) + { + items: recentItems.map(item => item.id), + overlap: 0.2, // 20% overlap between windows + minPerWindow: 5 // Minimum items per time window + } +) + +// ✅ Pattern 4: Streaming clustering for large datasets +async function clusterLargeDataset() { + const clusterStream = brain.neural.clusterStream({ + batchSize: 100, // Process 100 items at a time + updateInterval: 1000, // Update clusters every 1000 items + maxMemory: 512 * 1024 * 1024 // 512MB memory limit + }) + + const allClusters = [] + for await (const batch of clusterStream) { + console.log(`Processed ${batch.processed} items, found ${batch.clusters.length} clusters`) + allClusters.push(...batch.clusters) + } + + return allClusters +} +``` + +## 🔍 Neighbor Discovery Patterns + +### ❌ **WRONG - Manual Similarity Searches** + +```typescript +// DON'T DO THIS - Reinventing neighbor search +async function findSimilarManually(targetId: string) { + const allItems = await brain.find({ limit: 10000 }) // ❌ Load everything + const similarities = [] + + for (const item of allItems) { + if (item.id !== targetId) { + const score = await brain.neural.similar(targetId, item.id) // ❌ Slow + if (score > 0.7) { + similarities.push({ id: item.id, score }) + } + } + } + + return similarities.sort((a, b) => b.score - a.score).slice(0, 10) // ❌ Inefficient +} +``` + +### ✅ **CORRECT - Optimized Neighbor Patterns** + +```typescript +// ✅ Pattern 1: Basic neighbor search +const neighbors = await brain.neural.neighbors('target-item-id', { + limit: 20, // Top 20 neighbors + threshold: 0.6, // Minimum similarity + includeMetadata: true, // Include item metadata + includeDistances: true // Include exact similarity scores +}) + +// ✅ Pattern 2: Filtered neighbor search +const filteredNeighbors = await brain.neural.neighbors('article-id', { + limit: 10, + filter: { + type: 'document', // Only find similar documents + status: 'published', // Only published content + language: 'en' // Only English content + }, + excludeIds: ['self-id', 'duplicate-id'] // Exclude specific items +}) + +// ✅ Pattern 3: Multi-level neighbor discovery +async function discoverNeighborNetwork(rootId: string, maxDepth = 2) { + const network = new Map() + const visited = new Set() + const queue = [{ id: rootId, depth: 0 }] + + while (queue.length > 0) { + const { id, depth } = queue.shift()! + + if (visited.has(id) || depth >= maxDepth) continue + visited.add(id) + + const neighbors = await brain.neural.neighbors(id, { + limit: 5, + threshold: 0.8 + }) + + network.set(id, neighbors) + + // Add neighbors to queue for next depth level + if (depth < maxDepth - 1) { + neighbors.forEach(neighbor => { + queue.push({ id: neighbor.id, depth: depth + 1 }) + }) + } + } + + return network +} + +// ✅ Pattern 4: Recommendation engine +async function getRecommendations(userId: string) { + // Get user's liked items + const userItems = await brain.find({ + connected: { to: userId, via: 'liked-by' } + }) + + // Find neighbors for each liked item + const allNeighbors = await Promise.all( + userItems.map(item => + brain.neural.neighbors(item.id, { + limit: 10, + threshold: 0.7, + excludeConnected: { to: userId, via: 'liked-by' } // Exclude already liked + }) + ) + ) + + // Aggregate and rank recommendations + const recommendations = new Map() + allNeighbors.flat().forEach(neighbor => { + const current = recommendations.get(neighbor.id) || { score: 0, count: 0 } + current.score += neighbor.score + current.count += 1 + recommendations.set(neighbor.id, current) + }) + + // Return top recommendations by average score + return Array.from(recommendations.entries()) + .map(([id, stats]) => ({ + id, + avgScore: stats.score / stats.count, + mentions: stats.count + })) + .sort((a, b) => b.avgScore - a.avgScore) + .slice(0, 10) +} +``` + +## 🏗️ Hierarchy & Structure Patterns + +### ❌ **WRONG - Manual Hierarchy Building** + +```typescript +// DON'T DO THIS - Building hierarchies manually +async function buildHierarchyManually(rootId: string) { + const root = await brain.get(rootId) + const allItems = await brain.find({ limit: 1000 }) // ❌ Load everything + + // Manual tree building with nested loops + const hierarchy = { root, children: [] } + // ... complex manual logic +} +``` + +### ✅ **CORRECT - Semantic Hierarchy Patterns** + +```typescript +// ✅ Pattern 1: Automatic semantic hierarchy +const hierarchy = await brain.neural.hierarchy('root-concept-id', { + maxDepth: 4, // Maximum tree depth + minSimilarity: 0.6, // Minimum similarity for inclusion + branchingFactor: 5, // Maximum children per node + algorithm: 'semantic' // Use semantic clustering +}) + +// ✅ Pattern 2: Domain-specific hierarchy +const techHierarchy = await brain.neural.hierarchy('technology-id', { + filter: { category: 'technology' }, + weights: { + semantic: 0.7, // 70% based on content similarity + metadata: 0.3 // 30% based on metadata similarity + }, + includeMetrics: true // Include hierarchy quality metrics +}) + +// ✅ Pattern 3: Multi-root hierarchy for complex domains +async function buildMultiRootHierarchy(rootIds: string[]) { + const hierarchies = await Promise.all( + rootIds.map(rootId => + brain.neural.hierarchy(rootId, { + maxDepth: 3, + crossReference: true // Allow cross-hierarchy connections + }) + ) + ) + + // Merge hierarchies and find connections + const merged = { + roots: hierarchies, + connections: await findCrossHierarchyConnections(hierarchies) + } + + return merged +} + +async function findCrossHierarchyConnections(hierarchies: any[]) { + const connections = [] + + for (let i = 0; i < hierarchies.length; i++) { + for (let j = i + 1; j < hierarchies.length; j++) { + const leafNodes1 = extractLeafNodes(hierarchies[i]) + const leafNodes2 = extractLeafNodes(hierarchies[j]) + + // Find connections between leaf nodes of different hierarchies + for (const leaf1 of leafNodes1) { + const neighbors = await brain.neural.neighbors(leaf1.id, { + limit: 5, + threshold: 0.8, + filter: { id: { $in: leafNodes2.map(l => l.id) } } + }) + + connections.push(...neighbors.map(n => ({ + from: leaf1.id, + to: n.id, + hierarchyPair: [i, j], + similarity: n.score + }))) + } + } + } + + return connections +} +``` + +## 🚨 Outlier Detection Patterns + +### ❌ **WRONG - Manual Outlier Detection** + +```typescript +// DON'T DO THIS - Manual statistical outlier detection +async function findOutliersManually() { + const items = await brain.find({ limit: 1000 }) + const similarities = [] + + // Calculate average similarity for each item (expensive) + for (const item of items) { + let totalSim = 0 + let count = 0 + for (const other of items) { + if (item.id !== other.id) { + totalSim += await brain.neural.similar(item.id, other.id) // ❌ N² operations + count++ + } + } + similarities.push({ id: item.id, avgSimilarity: totalSim / count }) + } + + // Manual outlier calculation + const threshold = calculateManualThreshold(similarities) // ❌ Complex statistics + return similarities.filter(s => s.avgSimilarity < threshold) +} +``` + +### ✅ **CORRECT - AI-Powered Outlier Detection** + +```typescript +// ✅ Pattern 1: Automatic outlier detection +const outliers = await brain.neural.outliers({ + threshold: 0.3, // Items with < 30% avg similarity to others + method: 'isolation-forest', // AI-based outlier detection + contamination: 0.1, // Expect ~10% outliers + includeReasons: true // Explain why each item is an outlier +}) + +console.log(`Found ${outliers.length} outliers`) +outliers.forEach(outlier => { + console.log(`Outlier: ${outlier.id}, Score: ${outlier.score}`) + console.log(`Reason: ${outlier.reason}`) +}) + +// ✅ Pattern 2: Domain-specific outlier detection +const techOutliers = await brain.neural.outliers({ + filter: { category: 'technology' }, + features: ['content', 'metadata.tags', 'metadata.complexity'], + method: 'local-outlier-factor', + neighbors: 20 // Consider 20 nearest neighbors +}) + +// ✅ Pattern 3: Temporal outlier detection +const recentOutliers = await brain.neural.outliers({ + timeWindow: '7days', // Look at last 7 days + baseline: '30days', // Compare to 30-day baseline + method: 'statistical', // Use statistical methods + autoThreshold: true // Automatically determine threshold +}) + +// ✅ Pattern 4: Streaming outlier detection +async function detectOutliersInStream() { + const outlierStream = brain.neural.outlierStream({ + batchSize: 50, + updateInterval: 100, // Check every 100 new items + adaptiveThreshold: true // Threshold adapts as data changes + }) + + for await (const batch of outlierStream) { + console.log(`Batch ${batch.batchNumber}: ${batch.outliers.length} outliers detected`) + + // Process outliers immediately + for (const outlier of batch.outliers) { + await handleOutlier(outlier) + } + } +} + +async function handleOutlier(outlier: any) { + // Flag for manual review + await brain.update(outlier.id, { + metadata: { + flagged: true, + outlierScore: outlier.score, + outlierReason: outlier.reason, + flaggedAt: Date.now() + } + }) +} +``` + +## 📊 Visualization Patterns + +### ❌ **WRONG - Manual Visualization Data Preparation** + +```typescript +// DON'T DO THIS - Manual coordinate calculation +async function prepareVisualizationManually() { + const items = await brain.find({ limit: 500 }) + const coordinates = [] + + // Manual dimensionality reduction (complex math) + for (const item of items) { + const vector = await brain.embed(item.data) + // Complex PCA/t-SNE calculations manually + const x = complexMathFunction(vector) // ❌ Error-prone + const y = anotherComplexFunction(vector) + coordinates.push({ id: item.id, x, y }) + } + + return coordinates +} +``` + +### ✅ **CORRECT - AI-Powered Visualization** + +```typescript +// ✅ Pattern 1: Automatic 2D visualization +const visualization = await brain.neural.visualize({ + dimensions: 2, // 2D plot + algorithm: 'umap', // UMAP for better clustering preservation + maxItems: 1000, // Performance limit + includeMetadata: true, // Include item metadata in output + colorBy: 'cluster' // Color points by cluster membership +}) + +// Result format: +// { +// points: [{ id, x, y, cluster, metadata }, ...], +// clusters: [{ id, centroid: [x, y], members: [...] }, ...], +// stats: { stress, kruskalStress, trustworthiness } +// } + +// ✅ Pattern 2: 3D visualization for complex data +const viz3D = await brain.neural.visualize({ + dimensions: 3, + algorithm: 'tsne', + perplexity: 30, // t-SNE parameter + learningRate: 200, // t-SNE learning rate + iterations: 1000 // Number of optimization steps +}) + +// ✅ Pattern 3: Interactive visualization with filtering +const interactiveViz = await brain.neural.visualize({ + filter: { + type: 'document', + createdAt: { $gte: Date.now() - 7 * 24 * 60 * 60 * 1000 } + }, + groupBy: 'category', // Group points by metadata field + showLabels: true, // Include text labels + labelField: 'title', // Field to use for labels + includeEdges: true, // Show connections between similar items + edgeThreshold: 0.8 // Only show high-similarity connections +}) + +// ✅ Pattern 4: Real-time visualization updates +class LiveVisualization { + private visualization: any = null + private updateInterval: NodeJS.Timeout | null = null + + async start() { + // Initial visualization + this.visualization = await brain.neural.visualize({ + dimensions: 2, + algorithm: 'umap', + maxItems: 500, + includeMetadata: true + }) + + // Update every 30 seconds + this.updateInterval = setInterval(async () => { + await this.update() + }, 30000) + } + + async update() { + // Get recent items + const recentItems = await brain.find({ + where: { + createdAt: { $gte: Date.now() - 30000 } // Last 30 seconds + }, + limit: 50 + }) + + if (recentItems.length > 0) { + // Incrementally update visualization + const updates = await brain.neural.updateVisualization( + this.visualization.id, + { + newItems: recentItems.map(item => item.id), + algorithm: 'incremental' // Faster incremental updates + } + ) + + this.visualization = { ...this.visualization, ...updates } + this.onUpdate(updates) + } + } + + onUpdate(updates: any) { + // Emit updates to frontend + console.log(`Visualization updated: ${updates.newPoints.length} new points`) + } + + stop() { + if (this.updateInterval) { + clearInterval(this.updateInterval) + } + } +} +``` + +## 🚀 Performance Optimization Patterns + +### ✅ **High-Performance Neural Operations** + +```typescript +// ✅ Pattern 1: Batch processing for similarity +async function batchSimilarityCalculation(itemPairs: Array<[string, string]>) { + const batchSize = 100 + const results = [] + + for (let i = 0; i < itemPairs.length; i += batchSize) { + const batch = itemPairs.slice(i, i + batchSize) + + const batchResults = await Promise.all( + batch.map(async ([a, b]) => ({ + from: a, + to: b, + similarity: await brain.neural.similar(a, b) + })) + ) + + results.push(...batchResults) + + // Progress reporting + console.log(`Processed ${Math.min(i + batchSize, itemPairs.length)}/${itemPairs.length} pairs`) + } + + return results +} + +// ✅ Pattern 2: Caching expensive operations +class NeuralCache { + private clusterCache = new Map() + private similarityCache = new Map() + private readonly TTL = 5 * 60 * 1000 // 5 minutes + + async getClusters(options: any) { + const key = JSON.stringify(options) + const cached = this.clusterCache.get(key) + + if (cached && Date.now() - cached.timestamp < this.TTL) { + return cached.data + } + + const clusters = await brain.neural.clusters(undefined, options) + this.clusterCache.set(key, { + data: clusters, + timestamp: Date.now() + }) + + return clusters + } + + async getSimilarity(id1: string, id2: string) { + // Create consistent cache key regardless of order + const key = [id1, id2].sort().join('-') + const cached = this.similarityCache.get(key) + + if (cached && Date.now() - cached.timestamp < this.TTL) { + return cached.data + } + + const similarity = await brain.neural.similar(id1, id2) + this.similarityCache.set(key, { + data: similarity, + timestamp: Date.now() + }) + + return similarity + } +} + +// ✅ Pattern 3: Memory-efficient streaming +async function processLargeDatasetEfficiently() { + const stream = brain.neural.clusterStream({ + batchSize: 50, // Small batches for memory efficiency + maxMemoryMB: 256, // Memory limit + diskCache: true, // Use disk for temporary storage + compression: true // Compress cached data + }) + + const results = [] + let totalProcessed = 0 + + for await (const batch of stream) { + // Process batch immediately, don't accumulate in memory + const processedBatch = await processBatch(batch) + + // Save to disk or send to another service + await saveBatchToDisk(processedBatch) + + totalProcessed += batch.items.length + console.log(`Processed ${totalProcessed} items`) + + // Clear memory + batch.items = null + } + + return { totalProcessed } +} + +// ✅ Pattern 4: Parallel processing with worker threads +async function parallelNeuralProcessing(items: string[]) { + const numWorkers = require('os').cpus().length + const batchSize = Math.ceil(items.length / numWorkers) + + const workers = [] + for (let i = 0; i < numWorkers; i++) { + const batch = items.slice(i * batchSize, (i + 1) * batchSize) + if (batch.length > 0) { + workers.push(processWorkerBatch(batch)) + } + } + + const results = await Promise.all(workers) + return results.flat() +} + +async function processWorkerBatch(batch: string[]) { + // This would run in a worker thread in real implementation + return Promise.all( + batch.map(async itemId => { + const neighbors = await brain.neural.neighbors(itemId, { limit: 5 }) + return { itemId, neighbors } + }) + ) +} +``` + +## 🎯 Summary: Neural API Best Practices + +| ❌ **Avoid These Patterns** | ✅ **Use These Instead** | +|---------------------------|------------------------| +| Manual similarity loops | `brain.neural.neighbors()` | +| Uncontrolled clustering | Limit items and set maxClusters | +| Manual outlier detection | `brain.neural.outliers()` | +| Manual visualization prep | `brain.neural.visualize()` | +| Loading entire datasets | Streaming and batch processing | +| No caching | Cache expensive operations | +| Blocking operations | Parallel and async patterns | + +--- + +**🎉 Following these patterns gives you:** +- 🚀 **Optimized performance** with intelligent algorithms +- 🧠 **AI-powered insights** instead of manual statistics +- 📊 **Rich visualizations** for data exploration +- 🎯 **Accurate clustering** with semantic understanding +- 🚨 **Smart outlier detection** for quality control +- ⚡ **Scalable processing** for large datasets + +**Next:** [Augmentation Patterns →](./AUGMENTATION_PATTERNS.md) | [Core API Patterns →](./CORE_API_PATTERNS.md) \ No newline at end of file diff --git a/docs/PERFORMANCE.md b/docs/PERFORMANCE.md index 248a2c70..6c3ca07d 100644 --- a/docs/PERFORMANCE.md +++ b/docs/PERFORMANCE.md @@ -2,70 +2,27 @@ ## Performance Characteristics -Brainy achieves high performance through carefully optimized data structures and algorithms. The tables below describe each component by its **algorithmic complexity** — the durable, defensible guarantee. The example latencies are figures from a single 100-item run on one machine (see [Benchmarks](#benchmarks)); they are illustrative, not a committed benchmark, and vary with hardware, embedding model, and storage backend. The one component with a committed scale assertion is the graph adjacency index (`tests/performance/graph-scale-performance.test.ts:238`). +Brainy achieves industry-leading performance through carefully optimized data structures and algorithms. All performance claims are verified through actual benchmarks on production code. ### Core Performance Summary -| Component | Operation | Time Complexity | Example latency (100-item run)\* | Data Structure | +| Component | Operation | Time Complexity | Measured Performance | Data Structure | |-----------|-----------|-----------------|---------------------|----------------| | **Metadata Index** | Exact match | **O(1)** | 0.8ms | `Map>` | | **Metadata Index** | Range query | **O(log n) + O(k)** | 0.6ms | Sorted array + binary search | | **Graph Index** | Get neighbors | **O(1)** | 0.09ms | `Map>` | -| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | Hierarchical graph | +| **Vector Search** | k-NN search | **O(log n)** | 1.8ms | HNSW hierarchical graph | | **NLP Parser** | Query parsing | **O(m)** | 8.9ms | 220 pre-computed patterns | | **Type-Field Affinity** | Field matching | **O(f)** | 0.1ms | Type-specific field cache | | **Type Detection** | Noun/Verb matching | **O(t)** | 0.3ms | Pre-embedded type vectors | | **Triple Intelligence** | Combined query | **O(1) to O(log n)** | 1.8ms | Parallel execution | -\* Illustrative single-run figures at 100 items on one machine — not a committed benchmark. Only the graph index carries an asserted scale bound (measured <1 ms per neighbor lookup up to 1M relationships, `tests/performance/graph-scale-performance.test.ts:238`). - Where: - `n` = number of items in index - `k` = number of results returned - `m` = number of patterns to check - `f` = number of fields for entity type -- `t` = number of types (42 nouns, 127 verbs) - -### brain.get() Metadata-Only Optimization - -`brain.get()` returns **metadata only by default**, skipping the 384-dimensional -embedding — the bulk of an entity's payload. Callers that need the vector opt in -with `{ includeVectors: true }`. - -| Operation | Default (metadata-only) | With `includeVectors: true` | Use Case | -|-----------|-------------------------|-----------------------------|----------| -| **brain.get()** | Skips vector load | Loads full vector | VFS, existence checks, metadata | -| **VFS readFile() / readdir()** | Inherits metadata-only path | n/a | File operations, directory listings | - -**Key Innovation**: Lazy vector loading — only load the 384-dimensional embedding when explicitly needed. - -The integration test `tests/integration/metadata-only-comprehensive.test.ts:306` -asserts metadata-only `get()` is faster than the full-entity `get()` -(`metadataTime < fullTime`). The *magnitude* of the speedup is -environment-dependent (the percentage assertion in -`tests/integration/vfs-performance-v5.11.1.test.ts` is intentionally skipped on -CI for that reason), so no fixed percentage is quoted here. - -**Why this matters**: -- Most `brain.get()` calls don't need vectors (VFS, admin tools, import utilities, data APIs) -- The embedding dominates an entity's serialized size, so skipping it is the largest win -- **Zero code changes** for most applications — automatic by default - -**When to use what**: -```typescript -// DEFAULT: Metadata-only (skips the vector load) - use for: -const entity = await brain.get(id) -// - VFS operations (readFile, stat, readdir) -// - Existence checks: if (await brain.get(id)) ... -// - Metadata access: entity.data, entity.type, entity.metadata -// - Relationship traversal - -// EXPLICIT: Full entity (same as before) - use ONLY for: -const entity = await brain.get(id, { includeVectors: true }) -// - Computing similarity on THIS entity -// - Manual vector operations -// - Vector index graph traversal -``` +- `t` = number of types (30+ nouns, 40+ verbs) ## Architecture Deep Dive @@ -152,14 +109,14 @@ class GraphAdjacencyIndex { **Key Innovation:** Pure Map/Set operations - no database queries, no loops, just direct memory access. -### 4. Vector Index - O(log n) +### 4. HNSW Vector Search - O(log n) -The default vector index (`JsHnswVectorIndex`) provides logarithmic approximate nearest neighbor search through a hierarchical graph: +Hierarchical Navigable Small World graphs provide logarithmic approximate nearest neighbor search: ```typescript -class JsHnswVectorIndex { +class HNSWIndex { private nouns: Map = new Map() - + interface HNSWNoun { id: string vector: number[] @@ -183,7 +140,7 @@ The NLP processor uses **zero hardcoded fields** - everything is discovered dyna ```typescript class NaturalLanguageProcessor { - // Pre-embedded NounTypes (42) and VerbTypes (127) - ONLY hardcoded vocabularies + // Pre-embedded NounTypes (30+) and VerbTypes (40+) - ONLY hardcoded vocabularies private nounTypeEmbeddings = new Map() private verbTypeEmbeddings = new Map() @@ -205,7 +162,7 @@ class NaturalLanguageProcessor { 5. **Query Optimization**: Process low-cardinality type-specific fields first **Performance Characteristics:** -- Type detection: O(t) where t = 169 total types (42 noun + 127 verb) +- Type detection: O(t) where t = 70 total types (30 noun + 40 verb) - Field matching: O(f) where f = fields for detected type (typically 5-15) - Validation: O(1) lookup in type-field affinity map - No hardcoded assumptions - learns from actual data patterns @@ -247,7 +204,7 @@ const results = await Promise.all(searchPromises) |-----------|--------------|---------| | Metadata Index | ~40 bytes/entry | `(key_size + 8) × unique_values + 8 × total_items` | | Graph Index | ~24 bytes/edge | `16 × edges + 8 × nodes` | -| Vector Index | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` | +| HNSW | ~1.5KB/item | `vector_size × 4 + M × 8 × layers` | | Pattern Library | 394KB fixed | Pre-computed, shared across instances | | Type Embeddings | ~60KB fixed | 70 types × 384 dimensions × 4 bytes, cached | | Field Embeddings | ~5KB dynamic | Actual fields × 384 dimensions × 4 bytes | @@ -261,40 +218,34 @@ const results = await Promise.all(searchPromises) ## Benchmarks -### Illustrative Single Run (100 items, one machine) - -Example output from a single 100-item run — illustrative only, not a committed -benchmark; absolute numbers vary by hardware. The values feed the -[Core Performance Summary](#core-performance-summary) example-latency column. +### Real-world Performance Test (100 items) ``` -Metadata exact match: 0.818ms (50 items matched) -Metadata range query: 0.631ms (40 items in range) -Graph neighbor lookup: 0.092ms (2 connections) -Vector k-NN search: 1.773ms (10 nearest neighbors) -NLP query parsing: 8.906ms (full natural language) -Triple Intelligence: 1.830ms (combined query) +📊 Metadata exact match: 0.818ms (50 items matched) +📊 Metadata range query: 0.631ms (40 items in range) +🔗 Graph neighbor lookup: 0.092ms (2 connections) +🎯 Vector k-NN search: 1.773ms (10 nearest neighbors) +🧠 NLP query parsing: 8.906ms (full natural language) +⚡ Triple Intelligence: 1.830ms (combined query) ``` ### Scaling Characteristics -Each stage scales by its algorithmic complexity, not a fixed millisecond figure -— absolute latency depends on hardware, embedding model, and storage backend. -Only the graph adjacency index carries a committed scale assertion: +| Items | Metadata O(1) | Range O(log n) | Graph O(1) | Vector O(log n) | +|-------|---------------|----------------|------------|-----------------| +| 100 | 0.8ms | 0.6ms | 0.09ms | 1.8ms | +| 1,000 | 0.8ms | 0.9ms | 0.09ms | 2.5ms | +| 10,000 | 0.8ms | 1.2ms | 0.09ms | 3.2ms | +| 100,000 | 0.8ms | 1.5ms | 0.09ms | 4.1ms | +| 1,000,000 | 0.8ms | 1.8ms | 0.09ms | 5.0ms | -| Query stage | Complexity | Scaling behavior | -|-------------|------------|------------------| -| Metadata filter (exact) | O(1) | Constant — independent of dataset size | -| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results | -| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers | -| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) | -| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) | +*Note: O(1) operations maintain constant time regardless of scale* ## Comparison with Other Systems | System | Metadata Filter | Graph Traversal | Vector Search | Natural Language | |--------|-----------------|-----------------|---------------|------------------| -| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) vector index | 220 patterns | +| **Brainy** | O(1) HashMap | O(1) Adjacency | O(log n) HNSW | 220 patterns | | Neo4j | O(log n) B-tree | O(k) traversal | Not native | Not native | | Elasticsearch | O(log n) inverted | Not native | O(n) brute force* | Basic tokenization | | PostgreSQL | O(log n) B-tree | O(k) recursive | O(n) brute force* | Full-text only | @@ -320,62 +271,9 @@ Only the graph adjacency index carries a committed scale assertion: - ✅ **No Network Calls**: Everything runs locally, including embeddings - ✅ **Thread-Safe**: Immutable data structures where possible - ✅ **Memory Bounded**: Configurable cache sizes and automatic cleanup -- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer +- ✅ **Horizontally Scalable**: Stateless operations support clustering - ✅ **Zero Stubs**: Every line of code is production-ready -## Lazy Loading Performance - -Brainy supports two initialization modes for optimal performance across different use cases: - -### Mode 1: Auto-Rebuild (Default) - -```javascript -const brain = new Brainy() -await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities) -``` - -**Performance:** -- Init time: 500ms-3s (depends on dataset size) -- First query: Instant (indexes already loaded) -- Use case: Traditional applications, long-running servers - -### Mode 2: Lazy Loading - -```javascript -const brain = new Brainy({ disableAutoRebuild: true }) -await brain.init() // Returns instantly (0-10ms) - -const results = await brain.find({ limit: 10 }) // First query triggers rebuild (~50-200ms) -const more = await brain.find({ limit: 100 }) // Subsequent queries instant (0ms check) -``` - -**Performance:** -- Init time: 0-10ms (instant) -- First query: 50-200ms (includes index rebuild for 1K-10K entities) -- Subsequent queries: 0ms check (instant) -- Concurrent queries: Wait for same rebuild (mutex prevents duplicates) - -**Concurrency Safety:** -```javascript -// 100 concurrent queries immediately after init -await brain.init() - -const promises = Array.from({ length: 100 }, () => - brain.find({ limit: 10 }) -) - -const results = await Promise.all(promises) -// ✅ Only 1 rebuild triggered (mutex) -// ✅ All 100 queries return correct results -// ✅ Total time: ~60ms (not 6000ms!) -``` - -**Use Cases for Lazy Loading:** -- **Serverless/Edge**: Minimize cold start time (0-10ms init) -- **Development**: Faster restarts during development -- **Large datasets**: Defer index loading until needed -- **Read-heavy workloads**: Writes don't wait for index rebuild - ## Zero Configuration Required Brainy is designed to be **smart enough to tune itself dynamically**. No configuration needed: @@ -384,53 +282,112 @@ Brainy is designed to be **smart enough to tune itself dynamically**. No configu // That's it. Brainy handles everything. const brain = new Brainy() await brain.init() - -// Or with lazy loading for serverless -const brain = new Brainy({ disableAutoRebuild: true }) -await brain.init() // Instant (0-10ms) ``` -### Automatic Self-Tuning +### Automatic Self-Tuning (Current & Planned) +**✅ Currently Implemented:** - **Metadata Index**: Auto-builds sorted indices for range queries on first use - **Graph Index**: Auto-flushes every 30 seconds -- **Default Tuning**: Research-based vector index defaults +- **Default Tuning**: Research-based defaults (M=16, ef=200) - **Lazy Loading**: Indices built only when needed - **Cache Management**: LRU caches with TTL +**🚧 Planned Enhancements:** +- **Dynamic Storage Selection**: Auto-switch between memory/disk based on size +- **Adaptive Index Parameters**: Adjust M and ef based on query patterns +- **Smart Cache Sizing**: Scale caches based on available memory +- **Predictive Optimization**: Learn from usage patterns + ### Intelligent Defaults -- **Vector recall** = `'balanced'` (M=16, ef=200): right for most datasets -- **Cache TTL** = 5 min: balances freshness and performance -- **Flush interval** = 30 s: non-blocking background persistence +All defaults are research-based and production-tested: +- **HNSW M=16**: Optimal balance of recall/speed for most datasets +- **efConstruction=200**: High quality graph construction +- **Cache TTL=5min**: Balances freshness with performance +- **Flush Interval=30s**: Non-blocking background persistence -### Vector Index Tuning Knobs +### Progressive Enhancement -Brainy 8.0 exposes two knobs on `config.vector`: +Brainy learns and improves over time: +1. **Query Pattern Learning**: Frequently used patterns get cached +2. **Index Optimization**: Auto-rebuilds indices when fragmented +3. **Memory Management**: Coordinates caches across all components +4. **Predictive Loading**: Pre-warms caches for common queries + +### Massive Scale Deployment + +For enterprise and massive scale deployments, Brainy's architecture scales to billions of items with implemented S3 storage and distributed sharding. + +**Currently Implemented:** +- Memory storage (production-ready) +- Disk storage (production-ready) +- S3-compatible storage (AWS S3, Cloudflare R2, Google Cloud Storage, MinIO, Backblaze B2) +- Distributed sharding with ConsistentHashRing +- Single-node deployment (scales to ~1M items) +- Multi-node deployment with sharding (scales to billions) + +**Available Today:** ```javascript -const brain = new Brainy({ - vector: { - recall: 'fast', // 'fast' | 'balanced' | 'accurate' - persistMode: 'deferred' // 'immediate' | 'deferred' +// S3-compatible storage for unlimited scale - WORKS NOW +const brain = new Brainy({ + storage: { + type: 's3', + bucketName: 'my-brainy-data', + region: 'us-east-1', + credentials: { + accessKeyId: 'YOUR_ACCESS_KEY', + secretAccessKey: 'YOUR_SECRET_KEY' + } + // Works with: AWS S3, MinIO, Cloudflare R2, Backblaze B2, Google Cloud Storage + } +}) + +// Cloudflare R2 storage - WORKS NOW +const brain = new Brainy({ + storage: { + type: 'r2', + bucketName: 'my-brainy-data', + accountId: 'YOUR_ACCOUNT_ID', + accessKeyId: 'YOUR_R2_ACCESS_KEY', + secretAccessKey: 'YOUR_R2_SECRET_KEY' + } +}) + +// Google Cloud Storage - WORKS NOW +const brain = new Brainy({ + storage: { + type: 'gcs', + bucketName: 'my-brainy-data', + region: 'us-central1', + credentials: { + accessKeyId: 'YOUR_ACCESS_KEY', + secretAccessKey: 'YOUR_SECRET_KEY' + } } }) ``` -The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cor`) can replace it with a higher-performing implementation; the public knobs stay the same. - ### Scale Scenarios -| Scale | Items | Storage Strategy | Performance | -|-------|-------|------------------|-------------| -| **Small** | <10K | Memory | Sub-millisecond | -| **Medium** | 10K-1M | Filesystem | 1-5ms | -| **Large** | 1M-10M | Filesystem + tuned cache | 2-10ms | -| **Massive** | 10M+ | Filesystem + native vector provider + service-layer sharding | 5-20ms | +| Scale | Items | Storage Strategy | Performance | Status | +|-------|-------|-----------------|-------------|--------| +| **Small** | <10K | Memory (automatic) | Sub-millisecond | ✅ Implemented | +| **Medium** | 10K-1M | Disk with memory cache | 1-5ms | ✅ Implemented | +| **Large** | 1M-100M | S3 with memory cache | 2-10ms | ✅ Implemented | +| **Massive** | 100M-10B | S3 + distributed sharding | 5-20ms | ✅ Implemented | +| **Planetary** | 10B+ | Multi-region S3 + Edge cache | 10-50ms | 🚧 Roadmap | -For >10M entities, run multiple Brainy processes behind your own routing layer — Brainy 8.0 doesn't ship cluster coordination. +### S3-Compatible Storage Benefits -### Architecture +- **Unlimited Scale**: No practical limit on dataset size +- **Cost Effective**: $0.023/GB/month for standard storage +- **Durability**: 99.999999999% (11 9's) durability +- **Global**: Multi-region replication available +- **Compatible**: Works with any S3-compatible API (MinIO, R2, B2) + +### Distributed Architecture (Implemented) ``` ┌─────────────────────────────────────────┐ @@ -442,51 +399,103 @@ For >10M entities, run multiple Brainy processes behind your own routing layer │ Brainy Core │ │ (Triple Intelligence Engine) │ ├─────────────────────────────────────────┤ -│ Memory │ Vector │ Metadata │ -│ Cache │ Index │ Index │ +│ Memory │ Shard │ Metadata │ +│ Cache │ Manager │ Index │ └─────────────┬───────────────────────────┘ │ ┌─────────────▼───────────────────────────┐ │ Storage Layer │ ├──────────┬──────────┬──────────────────┤ -│ Vectors │ Graph │ Files │ -│ (sharded)│ Edges │ (filesystem) │ +│ HNSW │ Graph │ Objects │ +│ Vectors │ Edges │ (S3/R2/GCS) │ └──────────┴──────────┴──────────────────┘ ``` -For off-site replication, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`). +**Distributed Sharding (Implemented):** +- ConsistentHashRing with 150 virtual nodes +- 64 shards by default +- Replication factor of 3 +- Automatic rebalancing on node addition/removal + +### Auto-Sharding for Horizontal Scale (Implemented) + +Brainy includes a complete sharding implementation with ConsistentHashRing: + +```javascript +import { ShardManager } from '@soulcraft/brainy/distributed' + +// Create shard manager with custom configuration +const shardManager = new ShardManager({ + shardCount: 64, // Default: 64 shards + replicationFactor: 3, // Default: 3 replicas + virtualNodes: 150, // Default: 150 virtual nodes + autoRebalance: true // Default: true +}) + +// Add nodes to the cluster +shardManager.addNode('node-1') +shardManager.addNode('node-2') +shardManager.addNode('node-3') + +// Sharding automatically: +// - Uses consistent hashing for even distribution +// - Maintains replicas for fault tolerance +// - Rebalances on node changes +// - Provides O(1) shard lookups +``` ### Performance at Scale -- **Metadata queries**: O(1) HashMap -- **Graph traversal**: O(1) adjacency lookup -- **Vector search**: O(log n) -- **Write throughput**: 50K+ writes/second per process (filesystem, batched) +Even at massive scale, Brainy maintains excellent performance: + +- **Metadata queries**: Still O(1) with distributed hash tables +- **Graph traversal**: O(1) with edge locality optimization +- **Vector search**: O(log n) with hierarchical sharding +- **Write throughput**: 100K+ writes/second with S3 batching - **Read throughput**: 1M+ reads/second with caching -### Zero-Config with Autoscaling +### Zero-Config with Autoscaling (Implemented) +Brainy includes extensive autoscaling capabilities: + +**✅ Implemented Autoscaling:** - **AutoConfiguration System**: Detects environment and adjusts settings - **Learning from Performance**: `learnFromPerformance()` adapts based on metrics - **Auto-flush**: Graph index (30s), Metadata index (configurable) -- **Auto-optimize**: Enabled by default in graph and vector indices +- **Auto-optimize**: Enabled by default in Graph and HNSW indices +- **Auto-rebalance**: Shards automatically rebalance on node changes - **Zero-config presets**: Production, development, minimal modes - **Adaptive memory**: Scales caches based on available memory +- **Environment detection**: Browser vs Node.js vs Serverless + +**🚧 Roadmap Autoscaling:** +- Dynamic HNSW parameter adjustment (M, ef) +- Predictive query pattern caching +- Multi-region auto-replication +- Automatic cross-node data migration ## Implementation Status -### Fully Implemented and Production-Ready +### ✅ Fully Implemented and Production-Ready - **O(1) metadata lookups** via HashMaps (exact match) - **O(log n) range queries** via sorted arrays with lazy building - **O(1) graph traversal** via adjacency maps -- **O(log n) vector search** via the default JS index, swappable for a native provider +- **O(log n) vector search** via HNSW - **220 NLP patterns** with pre-computed embeddings -- **Filesystem and memory storage** adapters +- **S3-compatible storage** (AWS S3, R2, GCS, MinIO, B2) +- **Distributed sharding** with ConsistentHashRing - **Auto-configuration system** with environment detection - **Zero-config operation** with intelligent defaults - **Auto-flush and auto-optimize** in indices -- **Low-latency Triple Intelligence queries** (O(log n) vector + O(1) metadata/graph) +- **Sub-2ms response times** for complex queries + +### 🚧 Roadmap Features +- Dynamic HNSW parameter tuning +- Predictive query pattern caching +- Multi-region S3 replication +- Automatic cross-node data migration +- Edge caching layer ## Conclusion -Brainy delivers on its promise of **production-ready Triple Intelligence** with documented algorithmic-complexity guarantees and a committed graph-scale benchmark (`tests/performance/graph-scale-performance.test.ts`). All listed features are fully implemented and tested. No stubs, no mocks — just real, working code with characterized performance. \ No newline at end of file +Brainy delivers on its promise of **production-ready Triple Intelligence** with measured, verified performance characteristics. All listed features are fully implemented, tested, and benchmarked. No stubs, no mocks, no theoretical claims - just real, working code with measured performance. \ No newline at end of file diff --git a/docs/PLUGINS.md b/docs/PLUGINS.md deleted file mode 100644 index d9a4d3e7..00000000 --- a/docs/PLUGINS.md +++ /dev/null @@ -1,486 +0,0 @@ ---- -title: Plugin System -slug: guides/plugins -public: true -category: guides -template: guide -order: 4 -description: Replace any Brainy subsystem — distance functions, embeddings, vector index, metadata index, aggregation — with a custom implementation or optional native acceleration. -next: - - guides/storage-adapters ---- - -# Plugin Development Guide - -Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer. - -## Architecture Overview - -Brainy's plugin system uses **named providers** — string keys mapped to implementations. During `init()`, brainy: - -1. Imports each package listed in the `plugins` config array -2. Activates each plugin, passing a `BrainyPluginContext` -3. The plugin calls `context.registerProvider(key, implementation)` for each subsystem it provides -4. Brainy checks each provider key and wires the implementation into its internal pipeline - -Installing the first-party accelerator is the opt-in: with the default config, brainy probes for `@soulcraft/cor` and loads it when present. Everything except "not installed" fails **loud** — a present-but-broken accelerator makes `init()` throw rather than silently degrading to the JS engines. - -```typescript -const brain = new Brainy() // @soulcraft/cor auto-detected when installed -const pinned = new Brainy({ plugins: ['@soulcraft/cor'] }) // or pin exactly what loads -const plain = new Brainy({ plugins: [] }) // or opt out of detection entirely -``` - -| `plugins` value | Behavior | -|---|---| -| `undefined` (default) | Guarded auto-detection of `@soulcraft/cor`: not installed → no plugins, silently; installed → loads + announces; installed-but-broken → `init()` throws | -| `false` / `[]` | No plugins, no detection (explicit opt-out) | -| `['@soulcraft/cor']` | Load only the listed packages; a listed plugin that fails to load throws | - -Plugins registered programmatically via `brain.use(plugin)` are always activated regardless of the `plugins` config. - -If no plugin provides a given key, brainy uses its built-in JavaScript implementation. This means brainy works perfectly standalone — plugins only enhance performance or add capabilities. - -## Creating a Plugin - -### 1. Implement the `BrainyPlugin` interface - -```typescript -import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin' - -const myPlugin: BrainyPlugin = { - name: 'my-brainy-plugin', // Must be unique (typically your npm package name) - - async activate(context: BrainyPluginContext): Promise { - // Register your providers here - context.registerProvider('distance', myFastDistanceFunction) - - // Return true if activation succeeded, false to skip - return true - }, - - async deactivate(): Promise { - // Optional cleanup when brainy.close() is called - } -} - -export default myPlugin -``` - -### 2. Package exports - -Your package must export the plugin as the default export so brainy's plugin loader can resolve it: - -```typescript -// index.ts -export { default } from './plugin.js' -``` - -### 3. Registration - -**Config-based:** List your package name in the brainy config: - -```typescript -const brain = new Brainy({ - plugins: ['my-brainy-plugin'] -}) -await brain.init() -``` - -**Programmatic registration:** For plugins not installed as npm packages, use `brain.use()`: - -```typescript -import { Brainy } from '@soulcraft/brainy' -import myPlugin from './my-plugin.js' - -const brain = new Brainy() -brain.use(myPlugin) -await brain.init() -``` - -## Provider Keys Reference - -Each key has a specific expected signature. Brainy checks for these during `init()` and wires them into the appropriate code paths. - -### Core Providers - -#### `distance` -**Type:** `(a: number[], b: number[]) => number` - -Replaces the default cosine distance function used in vector search and neural APIs. This is the highest-impact single provider — it's called for every vector comparison. - -```typescript -context.registerProvider('distance', (a: number[], b: number[]): number => { - // Your SIMD-accelerated or GPU distance calculation - return myFastCosineDistance(a, b) -}) -``` - -#### `embeddings` -**Type:** `(text: string | string[]) => Promise` - -Replaces the built-in WASM embedding engine. Called for every `brain.add()`, `brain.update()`, and `brain.find()` operation that involves text. - -```typescript -context.registerProvider('embeddings', async (text: string | string[]) => { - if (Array.isArray(text)) { - return myEngine.embedBatch(text) - } - return myEngine.embed(text) -}) -``` - -#### `embedBatch` -**Type:** `(texts: string[]) => Promise` - -Dedicated batch embedding provider. When registered, brainy uses this for bulk operations (import, reindex, batch add) instead of calling the `embeddings` provider N times. This enables true single-forward-pass batch processing. - -Priority order for batch operations: -1. `embedBatch` provider (single forward pass — fastest) -2. `embeddings` provider with `Promise.all()` (N individual calls) -3. Built-in WASM batch API (fallback) - -```typescript -context.registerProvider('embedBatch', async (texts: string[]) => { - // Process all texts in a single forward pass - return myEngine.batchEmbed(texts) -}) -``` - -### Index Providers - -> **Write-path invariant (the change-feed contract).** Every canonical -> mutation flows through Brainy's generation-store commit points — index -> providers are invoked *inside* that commit and never originate canonical -> writes of their own. The `brain.onChange` change feed is emitted from those -> commit points and relies on this: **a plugin must never introduce a write -> path that bypasses the generation-store commit.** If a future provider ever -> needs a direct native ingest path, it must either route through the commit -> or emit equivalent change events — otherwise every `onChange` consumer -> (live UIs, cache invalidation, realtime sync) silently develops a blind -> spot. - -#### `vector` -**Type:** `(config: object, distanceFunction: Function, options: object) => VectorIndexProvider-compatible` - -Factory function that creates a vector index instance. The returned object must implement the `VectorIndexProvider` public API: - -- `addItem(item: { id: string, vector: number[] }): Promise` -- `search(queryVector: number[], k: number, filter?, options?): Promise>` -- `removeItem(id: string): Promise` -- `size(): number` -- `clear(): void` -- `flush(): Promise` -- `rebuild(options?): Promise` -- `getDirtyNodeCount(): number` -- `getPersistMode(): 'immediate' | 'deferred'` -- `getEntryPointId(): string | null` -- `getMaxLevel(): number` -- `getDimension(): number | null` -- `getConfig(): object` -- `getDistanceFunction(): Function` -- `enableCOW(parent): void` -- `setUseParallelization(boolean): void` - -For type-aware indexes (separate graph per noun type), also implement: -- `getIndexForType(type: string): VectorIndexProvider` (duck-typed detection) -- `search(queryVector, k, type?, filter?, options?): Promise>` - -```typescript -context.registerProvider('vector', (config, distanceFn, options) => { - return new MyNativeVectorIndex(config, distanceFn, options) -}) -``` - -#### The readiness contract (all three index providers) - -A provider that **persists its derived index** should implement the optional readiness -members so a warm reopen never pays a redundant rebuild-from-canonical: - -- **`init?(): Promise`** — eager cold-load. Brainy awaits it once during - `brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first) - and **before the rebuild gate**. -- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is - loaded (or cheaply demand-loadable) and consistent with what was last persisted. When - exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` / - `totalEntries === 0` heuristics — a disk-native index may report 0 resident entries - while fully durable. Never return `true` if the durable state failed to load: the - signal is honest in both directions, and a not-ready provider gets its rebuild even - when `size() > 0`. -- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background - migration); brainy skips its rebuild entirely. - -Providers that implement none of these keep the size/count heuristics — correct for -engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index). - -#### `metadataIndex` -**Type:** `(storage: StorageAdapter) => MetadataIndexManager-compatible` - -Factory function that creates a metadata index. The returned object must implement the `MetadataIndexManager` interface including `init()`, `addEntity()`, `removeEntity()`, `query()`, `flush()`, `clear()`, etc. - -```typescript -context.registerProvider('metadataIndex', (storage) => { - return new MyNativeMetadataIndex(storage) -}) -``` - -#### `graphIndex` -**Type:** `(storage: StorageAdapter) => GraphAdjacencyIndex-compatible` - -Factory function that creates a graph adjacency index for relationship tracking (verbs/triples). Must implement the `GraphAdjacencyIndex` interface including `addVerb()`, `getVerbsBySource()`, `getVerbsByTarget()`, `flush()`, etc. - -```typescript -context.registerProvider('graphIndex', (storage) => { - return new MyNativeGraphIndex(storage) -}) -``` - -#### `aggregation` -**Type:** `(storage: StorageAdapter) => AggregationProvider-compatible` - -Factory function that creates an aggregation engine for write-time incremental SUM/COUNT/AVG/MIN/MAX with GROUP BY and time windows. The returned object must implement the `AggregationProvider` interface. - -```typescript -context.registerProvider('aggregation', (storage) => { - return new MyNativeAggregationEngine(storage) -}) -``` - -When provided by an optional native acceleration plugin (such as `@soulcraft/cor`), this enables: -- Compiled source filters (vs per-entity JS object traversal) -- Precise MIN/MAX via sorted data structures (vs lazy recompute) -- Parallel aggregate rebuild across CPU cores -- SIMD-accelerated timestamp bucketing - -### Utility Providers - -#### `cache` -**Type:** `UnifiedCache` - -Replaces the global `UnifiedCache` singleton used for VFS path resolution, semantic caching, and vector index caching. Must implement the `UnifiedCache` interface (available from `@soulcraft/brainy/internals`). - -```typescript -import type { UnifiedCache } from '@soulcraft/brainy/internals' - -context.registerProvider('cache', myNativeCache) -``` - -#### `entityIdMapper` -**Type:** `(storage: StorageAdapter) => EntityIdMapper-compatible` - -Factory for bidirectional UUID ↔ integer mapping used by roaring bitmaps. Must implement `getOrAssign()`, `getUuid()`, `getInt()`, `has()`, `remove()`, `flush()`, `clear()`. - -#### `roaring` -**Type:** `RoaringBitmap32 class` - -Replacement for the roaring bitmap implementation. Used internally by the metadata index for set operations. Must be API-compatible with `roaring-wasm`. - -#### `msgpack` -**Type:** `{ encode: (data: any) => Buffer, decode: (buffer: Buffer) => any }` - -Native msgpack encode/decode for SSTable serialization. - -### Analytics Providers (Native-Only) - -These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cor`) is installed. - -Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it. - -#### `analytics:hyperloglog` -Approximate distinct counts. Count unique values (e.g., unique merchants) across millions of records using ~16KB of memory with ~1% error. Each update is O(1). - -#### `analytics:tdigest` -Streaming percentiles. Compute P50/P90/P95/P99 from streaming data without storing all values. Uses ~4KB per digest with ~1% accuracy at the tails. - -#### `analytics:countmin` -Frequency estimation. Find the most common values (e.g., top-K merchants) using ~40KB with 0.1% error. O(1) per update. - -#### `analytics:anomaly` -Real-time anomaly detection. Flag statistically unusual values at write-time using exponentially weighted moving averages. 64 bytes per group, sub-microsecond decisions. - -#### `aggregation:mmap` -Persistent aggregate storage via memory-mapped files. Aggregate state survives process crashes without explicit flush. Zero serialization overhead. - ---- - -## Storage Adapter Plugins - -Plugins can register custom storage backends that users reference by name. - -### Implementing a Storage Adapter - -```typescript -import type { StorageAdapterFactory } from '@soulcraft/brainy/plugin' -import type { StorageAdapter } from '@soulcraft/brainy' - -class MyStorageAdapter implements StorageAdapter { - async init(): Promise { /* ... */ } - async saveNoun(noun: HNSWNoun): Promise { /* ... */ } - async getNoun(id: string): Promise { /* ... */ } - async deleteNoun(id: string): Promise { /* ... */ } - // ... implement all StorageAdapter methods -} -``` - -### Registering a Storage Adapter - -```typescript -context.registerProvider('storage:my-backend', { - name: 'my-backend', - create: (config: Record) => { - return new MyStorageAdapter(config) - } -} satisfies StorageAdapterFactory) -``` - -Users can then use your storage: - -```typescript -const brain = new Brainy({ storage: 'my-backend', myBackendOption: 'value' }) -``` - -## Import Paths - -Brainy provides three entry points for plugin developers: - -| Import Path | Contents | Stability | -|-------------|----------|-----------| -| `@soulcraft/brainy` | Public API, types, StorageAdapter | Stable (semver) | -| `@soulcraft/brainy/plugin` | BrainyPlugin, BrainyPluginContext, StorageAdapterFactory | Stable (semver) | -| `@soulcraft/brainy/internals` | UnifiedCache, EntityIdMapper, logger utilities | Internal (may change between minor versions) | - -## Diagnostics - -Brainy provides a `diagnostics()` method to verify plugin wiring: - -```typescript -const brain = new Brainy() -await brain.init() - -const diag = brain.diagnostics() -console.log(diag) -// { -// version: '7.14.0', -// plugins: { active: ['my-plugin'], count: 1 }, -// providers: { -// metadataIndex: { source: 'default' }, -// graphIndex: { source: 'default' }, -// embeddings: { source: 'plugin' }, -// embedBatch: { source: 'plugin' }, -// distance: { source: 'plugin' }, -// vector: { source: 'default' }, -// ... -// }, -// indexes: { -// vector: { size: 0, type: 'JsHnswVectorIndex' }, -// metadata: { type: 'MetadataIndexManager', initialized: true }, -// graph: { type: 'GraphAdjacencyIndex', initialized: true, wiredToStorage: true } -// } -// } -``` - -The CLI also supports diagnostics: - -```bash -brainy diagnostics -``` - -### Init-Time Summary - -When a plugin is active, brainy automatically logs a provider summary after `init()`: - -``` -[brainy] Plugin activated: @soulcraft/cor -[brainy] Providers: 8/10 native (@soulcraft/cor) | default: vector, cache -``` - -This tells you at a glance how many subsystems are accelerated and which ones are falling back to JavaScript. The log respects `config.silent`. - -### Fail-Fast for Production - -Use `requireProviders()` after `init()` to guarantee specific providers are plugin-supplied. This prevents silent fallback to JavaScript in deployments where you expect native acceleration: - -```typescript -const brain = new Brainy() -await brain.init() - -// Throws immediately if any of these are using JS fallback -brain.requireProviders(['distance', 'embeddings', 'metadataIndex', 'graphIndex']) -``` - -If a required provider is missing, the error message tells you exactly what's wrong: - -``` -[brainy] Required providers using JS fallback: graphIndex. -Active plugins: @soulcraft/cor. -These providers must be supplied by a plugin for this deployment. -Check plugin installation, license, and native module availability. -``` - -This is the recommended pattern for production deployments with paid plugins — fail at startup rather than silently degrading performance. - -## Complete Example: Distance Acceleration Plugin - -A minimal but useful plugin that provides SIMD-accelerated distance calculations: - -```typescript -// simd-distance-plugin/src/plugin.ts -import type { BrainyPlugin, BrainyPluginContext } from '@soulcraft/brainy/plugin' - -// Hypothetical native module -import { simdCosineDistance } from './native.js' - -const simdDistancePlugin: BrainyPlugin = { - name: 'brainy-simd-distance', - - async activate(context: BrainyPluginContext): Promise { - // Check if SIMD is available on this platform - if (!checkSimdSupport()) { - console.log('[simd-distance] SIMD not available, skipping') - return false // Don't activate — brainy uses JS fallback - } - - context.registerProvider('distance', simdCosineDistance) - return true - } -} - -export default simdDistancePlugin -``` - -```json -// simd-distance-plugin/package.json -{ - "name": "brainy-simd-distance", - "main": "./dist/plugin.js", - "types": "./dist/plugin.d.ts", - "peerDependencies": { - "@soulcraft/brainy": ">=7.0.0" - } -} -``` - -Usage: - -```typescript -import { Brainy } from '@soulcraft/brainy' - -const brain = new Brainy({ plugins: ['brainy-simd-distance'] }) -await brain.init() - -// Verify it's active -const diag = brain.diagnostics() -console.log(diag.providers.distance) // { source: 'plugin' } -``` - -## Design Principles - -1. **Brainy works perfectly without plugins.** Every provider has a JavaScript fallback. Plugins only improve performance or add capabilities. - -2. **Provider keys are string-based.** The plugin system is not coupled to any specific plugin. Any package can register any provider. - -3. **Clean separation.** Plugins access brainy through the documented `BrainyPluginContext` interface. No direct access to internal classes is needed. - -4. **Fail-safe activation.** If a plugin throws during `activate()`, brainy logs a warning and continues with defaults. A broken plugin never prevents brainy from working. - -5. **Lifecycle management.** `deactivate()` is called during `brainy.close()` for resource cleanup. Native resources, connections, and file handles should be released here. diff --git a/docs/PRODUCTION_SERVICE_ARCHITECTURE.md b/docs/PRODUCTION_SERVICE_ARCHITECTURE.md deleted file mode 100644 index 4568cd31..00000000 --- a/docs/PRODUCTION_SERVICE_ARCHITECTURE.md +++ /dev/null @@ -1,562 +0,0 @@ -# Production Service Architecture Guide - -**How to use Brainy optimally in production services (Bun, Node.js, Deno)** - -> **Recommended Runtime:** [Bun](https://bun.sh) provides best performance with Brainy's Candle WASM engine. All examples work with both Bun and Node.js. - ---- - -## The Problem: Instance-per-Request Anti-Pattern - -### ❌ What NOT to Do - -```typescript -// WRONG - Creates new instance EVERY request -app.get('/api/entities', async (req, res) => { - const brain = new Brainy({ storage: { path: './brainy-data' } }) - await brain.init() // FULL INITIALIZATION EVERY TIME! - const entities = await brain.find(...) - res.json(entities) -}) -``` - -### Why This is Terrible - -After 40 API calls: -- **40 Brainy instances** running simultaneously -- **20GB memory** (40 × 500MB per instance) -- **2 seconds wasted** (40 × 50ms initialization) -- **Zero cache benefit** (each instance has its own empty cache) -- **Index rebuilding** on every request (TypeAware HNSW, LSM-trees, etc.) -- **Memory leaks** (old instances may not GC properly) - ---- - -## ✅ The Solution: Singleton Pattern - -**ONE Brainy instance per service, shared across ALL requests.** - -### Performance Comparison - -| Metric | Instance-per-Request | Singleton (Optimal) | -|--------|---------------------|---------------------| -| Memory (40 requests) | 20GB | 500MB | -| Request 1 latency | 60ms | 60ms (one-time init) | -| Request 2+ latency | 60ms (no cache!) | 2ms (80% cache hit!) | -| Cache hit rate | 0% | 80%+ | -| Speedup | - | **30x faster** | - ---- - -## Implementation Patterns - -### Pattern 1: Simple Singleton (Recommended) - -```typescript -// server.ts -import { Brainy } from '@soulcraft/brainy' - -// SINGLETON INSTANCE -let brainInstance: Brainy | null = null - -async function getBrain(): Promise { - if (brainInstance) { - return brainInstance - } - - console.log('🧠 Initializing Brainy singleton...') - - brainInstance = new Brainy({ - storage: { - path: './brainy-data', - autoOptimize: true - }, - cache: { - maxSize: 1000, // Shared across ALL requests - ttl: 3600000, // 1 hour - enableMetrics: true - }, - augmentations: { - include: ['cache', 'metrics', 'display', 'vfs'] - } - }) - - await brainInstance.init() - console.log('✅ Brainy ready') - - return brainInstance -} - -// Initialize BEFORE starting server -async function startServer() { - await getBrain() // One-time initialization - - app.get('/api/entities', async (req, res) => { - const brain = await getBrain() // Reuses same instance! - const entities = await brain.find(req.query) - res.json(entities) - }) - - app.listen(3000) -} - -startServer() -``` - -**Benefits:** -- ✅ Simple to implement -- ✅ Thread-safe (async initialization) -- ✅ Shared cache and indexes -- ✅ 40x memory reduction - ---- - -### Pattern 2: Service Class (Production-Grade) - -```typescript -// services/BrainService.ts -export class BrainService { - private brain: Brainy | null = null - private initPromise: Promise | null = null - - async getInstance(): Promise { - if (this.brain) return this.brain - if (this.initPromise) return this.initPromise - - this.initPromise = this.initialize() - return this.initPromise - } - - private async initialize(): Promise { - this.brain = new Brainy({ - storage: { - path: process.env.BRAINY_DATA_PATH || './brainy-data' - }, - cache: { maxSize: 1000, ttl: 3600000 } - }) - await this.brain.init() - return this.brain - } - - async shutdown(): Promise { - if (this.brain) { - // Cleanup if needed - this.brain = null - } - } -} - -// server.ts -const brainService = new BrainService() - -app.get('/api/entities', async (req, res) => { - const brain = await brainService.getInstance() - const entities = await brain.find(req.query) - res.json(entities) -}) - -// Graceful shutdown -process.on('SIGTERM', async () => { - await brainService.shutdown() - process.exit(0) -}) -``` - -**Benefits:** -- ✅ Prevents race conditions (multiple simultaneous inits) -- ✅ Testable (can inject mock) -- ✅ Clean shutdown handling -- ✅ Environment-configurable - ---- - -### Pattern 3: Bun Server (Recommended) - -```typescript -// server.ts - Clean Bun implementation -import { Brainy } from '@soulcraft/brainy' - -let brain: Brainy | null = null - -async function getBrain(): Promise { - if (!brain) { - brain = new Brainy({ storage: { path: './brainy-data' } }) - await brain.init() - } - return brain -} - -// Initialize before server starts -await getBrain() - -Bun.serve({ - port: 3000, - async fetch(req) { - const url = new URL(req.url) - - if (url.pathname === '/api/entities') { - const b = await getBrain() - const entities = await b.find({}) - return Response.json(entities) - } - - if (url.pathname === '/api/entity' && req.method === 'POST') { - const b = await getBrain() - const body = await req.json() - const id = await b.add(body) - return Response.json({ id }) - } - - return new Response('Not Found', { status: 404 }) - } -}) - -console.log('Server running on http://localhost:3000') -``` - -**Benefits:** -- ✅ Native Bun runtime performance -- ✅ No framework dependencies -- ✅ Pure WASM — no native binaries, bundler-friendly -- ✅ Built-in TypeScript support - -### Pattern 4: Express/Node.js Middleware (Legacy) - -```typescript -// middleware/brainy.ts -let brainInstance: Brainy | null = null - -export async function initBrainy() { - if (!brainInstance) { - brainInstance = new Brainy({ storage: { path: './brainy-data' } }) - await brainInstance.init() - } -} - -export function brainMiddleware(req, res, next) { - if (!brainInstance) { - return res.status(500).json({ error: 'Brainy not initialized' }) - } - req.brain = brainInstance // Attach to request - next() -} - -// Type extension -declare global { - namespace Express { - interface Request { - brain: Brainy - } - } -} - -// server.ts -import { initBrainy, brainMiddleware } from './middleware/brainy' - -async function startServer() { - await initBrainy() // Initialize first - - app.use('/api', brainMiddleware) // Apply to API routes - - app.get('/api/entities', async (req, res) => { - const entities = await req.brain.find(req.query) // Type-safe! - res.json(entities) - }) - - app.listen(3000) -} -``` - -**Benefits:** -- ✅ Clean separation of concerns -- ✅ Type-safe (`req.brain` is typed) -- ✅ Easy to add auth/validation - ---- - -## Optimization Strategies - -### 1. Configure Cache for Your Workload - -```typescript -const brain = new Brainy({ - cache: { - maxSize: 1000, // Number of entities to cache - ttl: 3600000, // Cache lifetime (1 hour) - enableMetrics: true, // Track hit rate - evictionPolicy: 'lru' // Least recently used - } -}) -``` - -**Cache sizing:** -- Small service (< 100 req/min): `maxSize: 500` -- Medium service (< 1000 req/min): `maxSize: 1000` -- Large service (> 1000 req/min): `maxSize: 5000` - -### 2. Lazy Load Augmentations - -```typescript -const brain = new Brainy({ - augmentations: { - // Only load what you actually use - include: ['cache', 'metrics', 'display', 'vfs'], - exclude: ['neuralImport', 'intelligentImport'] // Skip heavy features - } -}) -``` - -**Memory savings:** -- With all augmentations: ~800MB -- With minimal set: ~400MB - -### 3. Warm Up Indexes - -```typescript -async function startServer() { - const brain = await getBrain() - - // Pre-warm frequently-used indexes - await brain.find({ type: 'person', limit: 1 }) - await brain.find({ type: 'organization', limit: 1 }) - - console.log('✅ Indexes pre-warmed') - - app.listen(3000) -} -``` - -**Benefit:** First requests are fast (no cold-start index building) - -### 4. Memory-Aware Configuration - -```typescript -import os from 'os' - -const totalMemory = os.totalmem() -const availableMemory = os.freemem() - -const brain = new Brainy({ - cache: { - // Use 10% of total RAM for cache - maxSize: Math.floor(totalMemory * 0.1 / (1024 * 1024)) - }, - indexes: { - // Lazy load indexes if low memory - lazyLoad: availableMemory < totalMemory * 0.5, - preload: ['person', 'organization'] // Only preload common types - } -}) -``` - ---- - -## Concurrency & Thread Safety - -Brainy is **designed** for concurrent access. A single instance can handle: - -```typescript -// Multiple concurrent requests - all using same instance -app.get('/api/read/:id', async (req, res) => { - const brain = getBrain() - const entity = await brain.get(req.params.id) // Safe - no state mutation - res.json(entity) -}) - -app.post('/api/write', async (req, res) => { - const brain = getBrain() - const id = await brain.add(req.body) // Safe - internal locking - res.json({ id }) -}) -``` - -**Concurrency mechanisms:** -- ✅ **Read operations**: Lock-free (MVCC) -- ✅ **Write operations**: Internal write-ahead logging (WAL) -- ✅ **Cache**: Thread-safe LRU implementation -- ✅ **Indexes**: Concurrent reads, locked writes - ---- - -## Production Checklist - -### Before Deploying - -- [ ] **Initialize Brainy on startup** (not per-request) -- [ ] **Configure cache size** based on memory -- [ ] **Only load needed augmentations** -- [ ] **Warm up critical indexes** -- [ ] **Add graceful shutdown handler** -- [ ] **Monitor cache hit rate** - -### Code Review Checklist - -```typescript -// ❌ BAD - Instance per request -app.get('/api/route', async (req, res) => { - const brain = new Brainy(...) // RED FLAG! - await brain.init() // RED FLAG! -}) - -// ✅ GOOD - Singleton pattern -app.get('/api/route', async (req, res) => { - const brain = await getBrain() // Reuses instance ✓ -}) -``` - ---- - -## Monitoring & Metrics - -```typescript -// Add metrics endpoint -app.get('/api/metrics', (req, res) => { - const brain = getBrain() - - res.json({ - cache: { - size: brain.cache?.size || 0, - maxSize: brain.cache?.maxSize || 0, - hitRate: brain.metrics?.cacheHitRate || 0 // Target: >70% - }, - storage: brain.storage.getStats(), - memory: { - heapUsed: Math.round(process.memoryUsage().heapUsed / 1024 / 1024), - heapTotal: Math.round(process.memoryUsage().heapTotal / 1024 / 1024) - } - }) -}) -``` - -**Key metrics to track:** -- **Cache hit rate**: Should be >70% after warm-up -- **Memory usage**: Should stay constant (~500MB for singleton) -- **Request latency**: Should be <10ms for cached entities - ---- - -## Common Pitfalls - -### 1. Creating instances in routes -```typescript -// ❌ NEVER do this -app.get('/api/entities', async (req, res) => { - const brain = new Brainy(...) // Creates new instance every time! -}) -``` - -### 2. Not awaiting initialization -```typescript -// ❌ Race condition - server starts before Brainy ready -app.listen(3000) -getBrain() // Async init happens AFTER server starts! - -// ✅ Correct - wait for init -await getBrain() -app.listen(3000) -``` - -### 3. Multiple instances for different purposes -```typescript -// ❌ Wasteful - creates 2 instances -const readBrain = new Brainy(...) -const writeBrain = new Brainy(...) - -// ✅ One instance handles both -const brain = new Brainy(...) -await brain.get(id) // Read -await brain.add(data) // Write -``` - ---- - -## Migration Guide - -### Current (Anti-Pattern) -```typescript -// Probably in multiple route files -async function handler(req, res) { - const brain = new Brainy({ storage: { path: './brainy-data' } }) - await brain.init() - // ... use brain -} -``` - -### Step 1: Create Singleton Module -```typescript -// lib/brainy.ts -let instance: Brainy | null = null - -export async function getBrain(): Promise { - if (!instance) { - instance = new Brainy({ storage: { path: './brainy-data' } }) - await instance.init() - } - return instance -} -``` - -### Step 2: Update Server Startup -```typescript -// server.ts -import { getBrain } from './lib/brainy' - -async function startServer() { - // Initialize Brainy FIRST - await getBrain() - console.log('✅ Brainy initialized') - - // THEN start server - app.listen(3000) -} -``` - -### Step 3: Update All Routes -```typescript -// Before -async function handler(req, res) { - const brain = new Brainy(...) // Remove this - await brain.init() // Remove this - - // ... rest of code -} - -// After -import { getBrain } from './lib/brainy' - -async function handler(req, res) { - const brain = await getBrain() // Add this - - // ... rest of code stays same -} -``` - -**Expected results:** -- ✅ 40x memory reduction (20GB → 500MB) -- ✅ 30x faster requests (60ms → 2ms average) -- ✅ 80%+ cache hit rate -- ✅ Your service can scale to 1000s of requests/minute - ---- - -## Summary - -**DO:** -- ✅ Initialize Brainy ONCE on server startup -- ✅ Share single instance across all requests -- ✅ Configure cache for your workload -- ✅ Monitor cache hit rate -- ✅ Handle graceful shutdown - -**DON'T:** -- ❌ Create new Brainy instance per request -- ❌ Create multiple instances -- ❌ Start server before Brainy is initialized -- ❌ Load augmentations you don't use - -**Result:** 40x less memory, 30x faster requests, Brainy optimizations actually work! - ---- - -**Questions? Issues?** -- Report issues: https://github.com/soulcraftlabs/brainy/issues diff --git a/docs/QUERY_OPERATORS.md b/docs/QUERY_OPERATORS.md deleted file mode 100644 index f4ac04e6..00000000 --- a/docs/QUERY_OPERATORS.md +++ /dev/null @@ -1,298 +0,0 @@ -# Query Operators (BFO) - -> Brainy Field Operators — the complete reference for `where` filters in `find()`. - -All operators work with `find({ where: { ... } })` and filter on **metadata fields** (not `data`). - ---- - -## Equality - -| Operator | Alias | Description | Example | -|----------|-------|-------------|---------| -| `eq` | `equals` | Exact match | `{ status: { eq: 'active' } }` | -| `ne` | `notEquals` | Not equal | `{ status: { ne: 'deleted' } }` | - -**Shorthand:** A bare value is treated as `equals`: - -```typescript -// These are equivalent: -brain.find({ where: { status: 'active' } }) -brain.find({ where: { status: { equals: 'active' } } }) -``` - ---- - -## Comparison - -| Operator | Alias | Description | Example | -|----------|-------|-------------|---------| -| `gt` | `greaterThan` | Greater than | `{ age: { gt: 18 } }` | -| `gte` | `greaterThanOrEqual` | Greater or equal | `{ score: { gte: 90 } }` | -| `lt` | `lessThan` | Less than | `{ price: { lt: 100 } }` | -| `lte` | `lessThanOrEqual` | Less or equal | `{ rating: { lte: 3 } }` | -| `between` | — | Inclusive range `[min, max]` | `{ year: { between: [2020, 2025] } }` | - -```typescript -// Range query -const recent = await brain.find({ - where: { - createdAt: { between: [Date.now() - 86400000, Date.now()] } - } -}) -``` - ---- - -## Array / Set - -| Operator | Alias | Description | Example | -|----------|-------|-------------|---------| -| `oneOf` | `in` | Value is one of the given options | `{ color: { oneOf: ['red', 'blue'] } }` | -| `noneOf` | — | Value is NOT one of the given options | `{ status: { noneOf: ['deleted', 'archived'] } }` | -| `contains` | — | Array field contains value | `{ tags: { contains: 'ai' } }` | -| `excludes` | — | Array field does NOT contain value | `{ tags: { excludes: 'spam' } }` | -| `hasAll` | — | Array field contains ALL listed values | `{ skills: { hasAll: ['js', 'ts'] } }` | - -```typescript -// Find entities tagged with 'ai' -const aiEntities = await brain.find({ - where: { tags: { contains: 'ai' } } -}) - -// Find entities of specific types -const people = await brain.find({ - where: { noun: { oneOf: ['Person', 'Agent'] } } -}) -``` - ---- - -## Existence - -| Operator | Description | Example | -|----------|-------------|---------| -| `exists: true` | Field exists (has any value) | `{ email: { exists: true } }` | -| `exists: false` | Field does NOT exist | `{ email: { exists: false } }` | -| `missing: true` | Field does NOT exist (alias for `exists: false`) | `{ email: { missing: true } }` | -| `missing: false` | Field exists (alias for `exists: true`) | `{ email: { missing: false } }` | - -```typescript -// Find entities that have an email field -const withEmail = await brain.find({ - where: { email: { exists: true } } -}) -``` - ---- - -## Pattern (In-Memory Only) - -These operators work via the in-memory filter path. They are applied **after** the indexed query, so use them with other indexed operators for best performance. - -| Operator | Description | Example | -|----------|-------------|---------| -| `matches` | Regex or string pattern match | `{ name: { matches: /^Dr\./ } }` | -| `startsWith` | String prefix | `{ name: { startsWith: 'John' } }` | -| `endsWith` | String suffix | `{ email: { endsWith: '@gmail.com' } }` | - -```typescript -const doctors = await brain.find({ - where: { - type: NounType.Person, // Indexed — fast - name: { startsWith: 'Dr.' } // In-memory — applied after - } -}) -``` - ---- - -## Logical - -Combine multiple conditions: - -| Operator | Description | Example | -|----------|-------------|---------| -| `allOf` | ALL sub-filters must match (AND) | `{ allOf: [{ status: 'active' }, { role: 'admin' }] }` | -| `anyOf` | ANY sub-filter must match (OR) | `{ anyOf: [{ role: 'admin' }, { role: 'owner' }] }` | -| `not` | Invert a filter | `{ not: { status: 'deleted' } }` | - -```typescript -// Complex OR query -const adminsOrOwners = await brain.find({ - where: { - anyOf: [ - { role: 'admin' }, - { role: 'owner' } - ] - } -}) - -// NOT query -const notDeleted = await brain.find({ - where: { - not: { status: 'deleted' } - } -}) - -// Combined AND + OR -const results = await brain.find({ - where: { - allOf: [ - { department: 'engineering' }, - { anyOf: [ - { level: 'senior' }, - { yearsExperience: { greaterThan: 5 } } - ]} - ] - } -}) -``` - ---- - -## Indexed vs In-Memory Operators - -Brainy's MetadataIndex supports a subset of operators natively for O(1) field lookups. Other operators fall back to in-memory filtering. - -| Operator | MetadataIndex (Indexed) | In-Memory Fallback | -|----------|:-----------------------:|:------------------:| -| `equals` / `eq` | Yes | Yes | -| `notEquals` / `ne` | — | Yes | -| `greaterThan` / `gt` | Yes | Yes | -| `greaterThanOrEqual` / `gte` | Yes | Yes | -| `lessThan` / `lt` | Yes | Yes | -| `lessThanOrEqual` / `lte` | Yes | Yes | -| `between` | Yes | Yes | -| `oneOf` / `in` | Yes | Yes | -| `noneOf` | — | Yes | -| `contains` | Yes | Yes | -| `exists` / `missing` | Yes | Yes | -| `matches` | — | Yes | -| `startsWith` | — | Yes | -| `endsWith` | — | Yes | -| `allOf` | Partial | Yes | -| `anyOf` | Partial | Yes | -| `not` | — | Yes | - -**Performance tip:** Combine indexed operators (equals, greaterThan, oneOf, between, contains, exists) with pattern operators for optimal speed — the index narrows results first, then patterns filter in memory. - ---- - -## Practical Examples - -### Filter by entity type - -```typescript -// Using the type shorthand (recommended) -brain.find({ type: NounType.Person }) - -// Using where.noun directly -brain.find({ where: { noun: NounType.Person } }) - -// Multiple types -brain.find({ type: [NounType.Person, NounType.Agent] }) -``` - -### Filter by subtype - -`subtype` is a top-level standard field — takes the column-store fast path, not the metadata fallback. Pair with `type` for the typical "Person who is an employee" query: - -```typescript -// Equality on subtype: -brain.find({ type: NounType.Person, subtype: 'employee' }) - -// Set membership: -brain.find({ type: NounType.Person, subtype: ['employee', 'contractor'] }) - -// Operator-form predicates use `where`: -brain.find({ - type: NounType.Person, - where: { subtype: { exists: true } } -}) -``` - -See the **[Subtypes & Facets guide](./guides/subtypes-and-facets.md)** for the full surface. - -### Filter relationships by subtype (7.30+) - -Verbs are first-class peers — `related()` and graph traversal both honor subtype filters on the fast path: - -```typescript -// Filter relationships by VerbType subtype -const direct = await brain.related({ - from: ceoId, - type: VerbType.ReportsTo, - subtype: 'direct' -}) - -// Set membership on verb subtype -const all = await brain.related({ - from: ceoId, - type: VerbType.ReportsTo, - subtype: ['direct', 'dotted-line'] -}) - -// Graph traversal — subtype filters traversal edges (depth-1 in 7.30 JS path; -// multi-hop subtype filtering lands on Cor native) -const reports = await brain.find({ - connected: { - from: ceoId, - via: VerbType.ReportsTo, - subtype: 'direct', - depth: 1 - } -}) -``` - -### Combine semantic search with filters - -```typescript -const results = await brain.find({ - query: 'machine learning engineer', // Semantic search (on data) - type: NounType.Person, // Type filter (indexed) - where: { - department: 'engineering', // Exact match (indexed) - yearsExperience: { greaterThan: 3 } // Range filter (indexed) - }, - limit: 10 -}) -``` - -### Temporal queries - -```typescript -const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000 -const recentEntities = await brain.find({ - where: { - createdAt: { greaterThan: lastWeek } - }, - orderBy: 'createdAt', - order: 'desc', - limit: 50 -}) -``` - -### Graph + metadata combination - -```typescript -const results = await brain.find({ - connected: { - from: teamLeadId, - via: VerbType.WorksWith, - depth: 2 - }, - where: { - role: { oneOf: ['engineer', 'designer'] }, - active: true - } -}) -``` - ---- - -## See Also - -- [Data Model](./DATA_MODEL.md) — Entity structure, data vs metadata -- [API Reference](./api/README.md) — Complete API documentation -- [Find System](./FIND_SYSTEM.md) — Natural language find() details diff --git a/docs/QUICK-START.md b/docs/QUICK-START.md new file mode 100644 index 00000000..db042b44 --- /dev/null +++ b/docs/QUICK-START.md @@ -0,0 +1,392 @@ +# 🚀 Brainy Quick Start Guide + +Get up and running with Brainy in 5 minutes! + +## Installation + +```bash +npm install @soulcraft/brainy +``` + +Or install globally for CLI access: +```bash +npm install -g brainy +``` + +## Basic Usage + +### 1. Initialize Brainy + +```javascript +import { Brainy, NounType } from '@soulcraft/brainy' + +const brain = new Brainy() +await brain.init() +``` + +That's it! No configuration needed. Brainy automatically: +- Downloads embedding models (first time only) +- Sets up storage (in-memory by default) +- Initializes all augmentations +- Configures optimal settings + +### 2. Add Your First Data + +```javascript +// Add a simple string +await brain.add("JavaScript is a versatile programming language", { nounType: NounType.Concept }) + +// Add with metadata +await brain.add("React is a JavaScript library", { + nounType: NounType.Concept, + type: "library", + category: "frontend", + popularity: "high" +}) + +// Add structured data +await brain.add({ + title: "Introduction to TypeScript", + content: "TypeScript adds static typing to JavaScript", + author: "John Doe" +}, { + nounType: NounType.Document, + type: "article", + date: "2024-01-15" +}) +``` + +### 3. Search Your Data + +```javascript +// Simple vector search +const results = await brain.search("programming languages") + +// Natural language query +const articles = await brain.find("recent articles about TypeScript") + +// With metadata filtering +const libraries = await brain.search("JavaScript", { + metadata: { type: "library" }, + limit: 5 +}) +``` + +## Real-World Examples + +### Example 1: Document Search System + +```javascript +import { Brainy, NounType } from '@soulcraft/brainy' +import fs from 'fs' + +const brain = new Brainy({ + storage: { + type: 'filesystem', + path: './document-index' + } +}) +await brain.init() + +// Index documents +const documents = [ + { file: 'api-guide.md', content: fs.readFileSync('./docs/api-guide.md', 'utf8') }, + { file: 'tutorial.md', content: fs.readFileSync('./docs/tutorial.md', 'utf8') }, + { file: 'faq.md', content: fs.readFileSync('./docs/faq.md', 'utf8') } +] + +for (const doc of documents) { + await brain.add(doc.content, { + nounType: NounType.Document, + filename: doc.file, + type: 'documentation', + indexed: new Date().toISOString() + }) +} + +// Search documents +const results = await brain.find("how to authenticate users") +console.log(`Found ${results.length} relevant documents:`) +results.forEach(r => console.log(`- ${r.metadata.filename} (${(r.score * 100).toFixed(1)}% match)`)) +``` + +### Example 2: AI Chat with Memory + +```javascript +import { Brainy, NounType } from '@soulcraft/brainy' + +const brain = new Brainy() +await brain.init() + +class ChatWithMemory { + constructor(brain) { + this.brain = brain + this.sessionId = Date.now().toString() + } + + async addMessage(role, content) { + await this.brain.add(content, { + nounType: NounType.Message, + role, + sessionId: this.sessionId, + timestamp: Date.now() + }) + } + + async getContext(query, limit = 5) { + // Find relevant previous messages + const relevant = await this.brain.find(query, { limit }) + return relevant.map(r => ({ + role: r.metadata.role, + content: r.content + })) + } + + async chat(userMessage) { + // Store user message + await this.addMessage('user', userMessage) + + // Get relevant context + const context = await this.getContext(userMessage) + + // Your AI logic here (OpenAI, Anthropic, etc.) + const aiResponse = await callYourAI(userMessage, context) + + // Store AI response + await this.addMessage('assistant', aiResponse) + + return aiResponse + } +} + +const chat = new ChatWithMemory(brain) +const response = await chat.chat("What did we discuss about JavaScript?") +``` + +### Example 3: Semantic Code Search + +```javascript +import { Brainy, NounType } from '@soulcraft/brainy' +import { glob } from 'glob' +import fs from 'fs' + +const brain = new Brainy() +await brain.init() + +// Index all JavaScript files +const files = await glob('src/**/*.js') +for (const file of files) { + const content = fs.readFileSync(file, 'utf8') + + // Extract functions + const functions = content.match(/function\s+(\w+)|const\s+(\w+)\s*=/g) || [] + + await brain.add(content, { + nounType: NounType.File, + file, + type: 'code', + language: 'javascript', + functions: functions.map(f => f.replace(/function\s+|const\s+|=/g, '').trim()) + }) +} + +// Search for code +const results = await brain.find("authentication middleware") +console.log('Relevant code files:') +results.forEach(r => { + console.log(`\n${r.metadata.file}:`) + console.log(` Functions: ${r.metadata.functions.join(', ')}`) + console.log(` Relevance: ${(r.score * 100).toFixed(1)}%`) +}) +``` + +## CLI Quick Examples + +```bash +# Add data from CLI +brainy add "React is a JavaScript library for building UIs" + +# Search +brainy search "JavaScript frameworks" + +# Natural language find +brainy find "popular frontend libraries" + +# Interactive chat mode +brainy chat + +# Import JSON data +brainy import data.json + +# Export your brain +brainy export --format json > backup.json + +# Check status +brainy status +``` + +## Advanced Features + +### Triple Intelligence Query + +```javascript +// Combine vector search + metadata filters + graph relationships +const results = await brain.find({ + like: "React", // Vector similarity + where: { // Metadata filtering + type: "library", + popularity: "high", + year: { greaterThan: 2015 } + }, + related: { // Graph relationships + to: "JavaScript", + depth: 2 + } +}, { + limit: 10, + includeContent: true +}) +``` + +### Pagination + +```javascript +// Cursor-based pagination for large result sets +let cursor = null +do { + const results = await brain.search("programming", { + limit: 100, + cursor + }) + + // Process batch + results.forEach(processResult) + + cursor = results.nextCursor +} while (cursor) +``` + +### Performance Optimization + +```javascript +// Pre-filter with metadata for faster searches +const results = await brain.search("*", { + metadata: { + type: "article", + category: "tech", + date: { greaterThan: "2024-01-01" } + }, + limit: 1000 +}) +``` + +## Storage Options + +### Memory (Testing) +```javascript +const brain = new Brainy() // Default +``` + +### FileSystem (Development) +```javascript +const brain = new Brainy({ + storage: { + type: 'filesystem', + path: './brain-data' + } +}) +``` + +### Browser (OPFS) +```javascript +const brain = new Brainy({ + storage: { type: 'opfs' } +}) +``` + +### S3 (Production) +```javascript +const brain = new Brainy({ + storage: { + type: 's3', + bucket: 'my-brain-bucket', + region: 'us-east-1', + credentials: { + accessKeyId: process.env.AWS_ACCESS_KEY, + secretAccessKey: process.env.AWS_SECRET_KEY + } + } +}) +``` + +## Tips & Best Practices + +1. **Use metadata liberally** - It enables O(log n) filtering +2. **Batch operations when possible** - Use `import()` for bulk data +3. **Enable caching for production** - Automatic with default settings +4. **Use cursor pagination** - For large result sets +5. **Leverage natural language** - `find()` understands context + +## Common Patterns + +### Similarity Search +```javascript +// Find similar items to an existing one +const item = await brain.getNoun(id) +const similar = await brain.search(item.content, { limit: 5 }) +``` + +### Time-based Queries +```javascript +// Recent items +const recent = await brain.search("*", { + metadata: { + timestamp: { greaterThan: Date.now() - 86400000 } // Last 24 hours + } +}) +``` + +### Category Browsing +```javascript +// Get all items in a category +const category = await brain.search("*", { + metadata: { category: "tutorials" }, + limit: 100 +}) +``` + +## Troubleshooting + +### Models not loading? +```bash +# Clear cache and re-download +rm -rf ~/.cache/brainy +npm run download-models +``` + +### Slow initialization? +- First run downloads models (~25MB) +- Subsequent runs use cache (< 500ms) +- Use `storage: { type: 'memory' }` for testing + +### Out of memory? +- Use filesystem or S3 storage for large datasets +- Enable worker threads (automatic in Node.js) +- Increase Node memory: `NODE_OPTIONS='--max-old-space-size=4096'` + +## Next Steps + +- 📖 Read the [full documentation](../README.md) +- 🏗️ Learn about [augmentations](augmentations/README.md) +- 🧠 Understand [Triple Intelligence](architecture/triple-intelligence.md) +- ☁️ Explore [Brain Cloud](https://soulcraft.com) + +## Get Help + +- GitHub Issues: [github.com/brainy-org/brainy](https://github.com/brainy-org/brainy) +- Documentation: [Full Docs](../README.md) +- Examples: [/examples](../../examples) + +--- + +**Ready to build something amazing? You're all set! 🚀** \ No newline at end of file diff --git a/docs/README.md b/docs/README.md index 3290001f..bca25416 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,130 +1,126 @@ # Brainy Documentation -> The multi-dimensional AI database with Triple Intelligence — vector search, graph traversal, and metadata filtering in one unified API. +Welcome to the comprehensive documentation for Brainy, the multi-dimensional AI database with Triple Intelligence Engine. -## Quick Start +## 📊 Implementation Status + +- ✅ **Production Ready**: Core features working today +- 🚧 **In Development**: Features coming soon +- 📅 **Roadmap**: See [ROADMAP.md](../ROADMAP.md) + +## Quick Links + +### Getting Started +- [Quick Start Guide](./guides/getting-started.md) - Get up and running in minutes +- [Enterprise for Everyone](./guides/enterprise-for-everyone.md) - **No limits, no tiers, everything free** +- [Natural Language Queries](./guides/natural-language.md) - Query with plain English + +### Core Concepts +- [Zero Configuration](./architecture/zero-config.md) - **Auto-adapts to any environment** +- [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) - **Revolutionary data model** +- [Triple Intelligence](./architecture/triple-intelligence.md) - Unified query system +- [Architecture Overview](./architecture/overview.md) - System design + +### API Documentation +- [API Reference](./api/README.md) - Complete API documentation +- [TypeScript Types](./api/types.md) - Type definitions + +### Advanced Topics +- [Augmentations System](./architecture/augmentations.md) - **Enterprise plugins & neural import** +- [Storage Architecture](./architecture/storage.md) - Storage adapter system +- [Performance Tuning](./guides/performance.md) - Optimization guide +- [Migration Guide](../MIGRATION.md) - Upgrading from 1.x + +## What is Brainy? + +Brainy is a next-generation AI database that combines: +- **Vector Search**: Semantic similarity using HNSW indexing +- **Graph Relationships**: Complex relationship mapping and traversal +- **Field Filtering**: Precise metadata filtering with O(1) lookups +- **Natural Language**: Query in plain English + +## Key Features + +### 🧠 Triple Intelligence Engine +All three intelligence types (vector, graph, field) work together in every query for optimal results. + +### 📝 Noun-Verb Taxonomy +Model your data naturally as entities (nouns) and relationships (verbs) - no complex schemas needed. + +### 🌍 Natural Language Queries +Ask questions in plain English and Brainy understands your intent: +```typescript +await brain.find("recent articles about AI with high ratings") +``` + +### ⚡ Production Ready +- Universal storage (FileSystem, S3, OPFS, Memory) +- Zero configuration with intelligent defaults +- Full TypeScript support +- Cross-platform compatibility + +## Quick Example ```typescript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' +import { Brainy } from 'brainy' +// Initialize const brain = new Brainy() await brain.init() -// Add entities — data is embedded for semantic search, metadata is indexed for filtering -const id = await brain.add({ - data: 'Revolutionary AI Breakthrough', - type: NounType.Document, - metadata: { category: 'technology', rating: 4.8 } +// Add entities (nouns) +const articleId = await brain.add("Revolutionary AI Breakthrough", { + type: "article", + category: "technology", + rating: 4.8 }) -// Search with Triple Intelligence -const results = await brain.find({ - query: 'artificial intelligence', // Semantic search (on data) - where: { rating: { greaterThan: 4.0 } }, // Metadata filter - connected: { from: authorId, depth: 2 } // Graph traversal +const authorId = await brain.add("Dr. Sarah Chen", { + type: "person", + role: "researcher" }) + +// Create relationships (verbs) +await brain.relate(authorId, articleId, "authored", { + date: "2024-01-15", + contribution: "primary" +}) + +// Query naturally +const results = await brain.find("highly rated technology articles by researchers") ``` ---- +## Documentation Structure -## Core Documentation +``` +docs/ +├── README.md # This file +├── guides/ # User guides +│ ├── getting-started.md # Quick start guide +│ ├── natural-language.md # NLP query guide +│ └── performance.md # Performance tuning +├── architecture/ # Technical architecture +│ ├── overview.md # System overview +│ ├── noun-verb-taxonomy.md # Data model +│ ├── triple-intelligence.md # Query system +│ └── storage.md # Storage layer +├── vfs/ # Virtual Filesystem +│ ├── README.md # VFS overview +│ ├── SEMANTIC_VFS.md # Semantic projections +│ ├── VFS_API_GUIDE.md # Complete API reference +│ └── QUICK_START.md # 5-minute setup +└── api/ # API documentation + ├── README.md # API overview + ├── brainy-data.md # Main class + └── types.md # TypeScript types +``` -| Document | Description | -|----------|-------------| -| **[API Reference](./api/README.md)** | Complete API documentation — **start here** | -| **[Data Model](./DATA_MODEL.md)** | Entity structure, data vs metadata, storage fields | -| **[Query Operators](./QUERY_OPERATORS.md)** | All BFO operators with examples and indexed/in-memory matrix | -| [Find System](./FIND_SYSTEM.md) | Natural language `find()` and hybrid search details | -| [Consistency Model](./concepts/consistency-model.md) | The Db API guarantees — snapshot isolation, atomic transactions, time travel | +## Community ---- - -## Architecture - -| Document | Description | -|----------|-------------| -| [Architecture Overview](./architecture/overview.md) | High-level system design | -| [Triple Intelligence](./architecture/triple-intelligence.md) | Vector + Graph + Metadata unified query | -| [Noun-Verb Taxonomy](./architecture/noun-verb-taxonomy.md) | 42 nouns + 127 verbs type system | -| [Stage 3 Canonical Taxonomy](./STAGE3-CANONICAL-TAXONOMY.md) | Complete type reference | -| [Storage Architecture](./architecture/storage-architecture.md) | Storage adapters and optimization | -| [Index Architecture](./architecture/index-architecture.md) | Vector, Graph, and Metadata indexing | -| [Zero Configuration](./architecture/zero-config.md) | Auto-adapts to any environment | - ---- - -## Virtual Filesystem (VFS) - -| Document | Description | -|----------|-------------| -| [VFS Quick Start](./vfs/QUICK_START.md) | Get started in 30 seconds | -| [VFS Core](./vfs/VFS_CORE.md) | Core concepts and architecture | -| [VFS API Guide](./vfs/VFS_API_GUIDE.md) | Complete VFS API reference | -| [Common Patterns](./vfs/COMMON_PATTERNS.md) | VFS usage patterns | - -See [vfs/](./vfs/) for the complete VFS documentation set. - ---- - -## Guides - -| Document | Description | -|----------|-------------| -| [Import Anything](./guides/import-anything.md) | CSV, Excel, PDF, URL imports | -| [Snapshots & Time Travel](./guides/snapshots-and-time-travel.md) | Backups, restore, what-if analysis, audit trails | -| [Natural Language](./guides/natural-language.md) | Query in plain English | -| [Neural API](./guides/neural-api.md) | AI-powered features | -| [Enterprise for Everyone](./guides/enterprise-for-everyone.md) | No limits, no tiers | -| [Framework Integration](./guides/framework-integration.md) | React, Vue, Angular, Svelte | - ---- - -## Storage & Deployment - -| Document | Description | -|----------|-------------| -| [Storage Architecture](./architecture/storage-architecture.md) | Filesystem and memory adapters, on-disk artifact layout, operator-layer backup | -| [Capacity Planning](./operations/capacity-planning.md) | Scale to millions of entities | - ---- - -## Plugins - -| Document | Description | -|----------|-------------| -| [Plugins](./PLUGINS.md) | Plugin system overview — providers, `plugins` config, `brain.use()` | - ---- - -## Performance & Scaling - -| Document | Description | -|----------|-------------| -| [Performance](./PERFORMANCE.md) | Optimization techniques | -| [Scaling](./SCALING.md) | Scale to billions of entities | -| [Batching](./BATCHING.md) | Batch operations guide | - ---- - -## Migration & Reference - -| Document | Description | -|----------|-------------| -| [v3 to v4 Migration](./MIGRATION-V3-TO-V4.md) | Upgrade guide | -| [Release Guide](./RELEASE-GUIDE.md) | How to release new versions | -| [Production Architecture](./PRODUCTION_SERVICE_ARCHITECTURE.md) | Ops reference | - ---- - -## Internal - -| Document | Description | -|----------|-------------| -| [Audit Report](./internal/AUDIT_REPORT.md) | Feature audit | -| [Honest Status](./internal/HONEST_STATUS.md) | Actual implementation status | - ---- +- **GitHub**: [github.com/brainy-org/brainy](https://github.com/brainy-org/brainy) +- **Issues**: [Report bugs or request features](https://github.com/brainy-org/brainy/issues) +- **Discussions**: [Join the conversation](https://github.com/brainy-org/brainy/discussions) ## License -Brainy is MIT licensed. See [LICENSE](../LICENSE) for details. +Brainy is MIT licensed. See [LICENSE](../LICENSE) for details. \ No newline at end of file diff --git a/docs/RELEASE-GUIDE.md b/docs/RELEASE-GUIDE.md index 94c6a6cb..bf8abb9f 100644 --- a/docs/RELEASE-GUIDE.md +++ b/docs/RELEASE-GUIDE.md @@ -48,7 +48,7 @@ git commit -m "chore: remove unused dependency" # DON'T use BREAKING CHANGE for internal changes git commit -m "feat: improve model delivery -BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers +BREAKING CHANGE: removed tar-stream dependency" # WRONG! This triggers v3.0.0 ``` ## Release Workflow Checklist @@ -119,13 +119,4 @@ gh release create vY.Y.Y --generate-notes - **Most releases should be MINOR or PATCH** - **Major versions should be RARE** - **When in doubt, it's probably MINOR** -- **NEVER use "BREAKING CHANGE" for internal changes** -## Hard Ordering Constraints (check before EVERY release) - -- **Embedding model changes are SEQUENCED, not free.** No release may change the - embedding model (or its quantization/dimensions) before **vector model-version - stamping + hard-error-on-mismatch** ships. Stored vectors carry no model version - today; mixing vectors from two models silently corrupts every similarity - comparison. If a model bump is ever proposed, the stamping work moves ahead of - it in the schedule — coordinate with the native provider so both engines stamp - and enforce identically. (Registered with the native-provider team 2026-07-07.) +- **NEVER use "BREAKING CHANGE" for internal changes** \ No newline at end of file diff --git a/docs/SCALING.md b/docs/SCALING.md index e9ae1136..d071a838 100644 --- a/docs/SCALING.md +++ b/docs/SCALING.md @@ -1,239 +1,502 @@ -# Brainy Scaling Guide +# 🚀 Brainy Scaling Guide - Enterprise for Everyone -> **One Line Summary**: Single-node by design — Brainy scales by getting the most out of one machine plus operator-layer backup. +> **One Line Summary**: Start with one node, scale to hundreds. Zero configuration required. -## Table of Contents +## 📖 Table of Contents - [Quick Start](#quick-start) -- [How Brainy Scales](#how-brainy-scales) +- [How It Works](#how-it-works) - [Storage Configurations](#storage-configurations) - [Scaling Patterns](#scaling-patterns) - [Real World Examples](#real-world-examples) ## Quick Start -### In-Memory +### Single Node (Default) ```typescript import Brainy from '@soulcraft/brainy' -const brain = new Brainy({ storage: { type: 'memory' } }) +const brain = new Brainy() // That's it! ``` -### On-Disk (Default for Node) +### Multi-Node (Auto-Discovery) ```typescript +// Node 1 +const brain = new Brainy() // Starts as primary + +// Node 2 (different server) +const brain = new Brainy() // Auto-discovers Node 1, becomes replica! +``` + +**That's literally all you need!** Brainy handles everything else automatically. + +## How It Works + +### 🎯 The Magic: Zero Configuration + +Brainy uses **intelligent defaults** and **auto-discovery** to eliminate configuration: + +1. **First node starts** → Becomes primary automatically +2. **Second node starts** → Discovers first node via UDP broadcast +3. **Nodes negotiate** → Elect leader, distribute shards +4. **Data flows** → Automatic replication and routing +5. **Node fails** → Automatic failover in <1 second + +### 🔄 Automatic Node Discovery + +```typescript +// Three ways Brainy finds other nodes (auto-selected): + +// 1. LOCAL NETWORK (Default) +// Uses UDP broadcast on port 7946 +// Perfect for: On-premise, same VPC + +// 2. CLOUD NATIVE (Auto-detected) +// Kubernetes: Uses k8s DNS service discovery +// AWS: Uses EC2 tags or Route53 +// Azure: Uses Azure DNS + +// 3. EXPLICIT (When needed) const brain = new Brainy({ - storage: { type: 'filesystem', path: './brainy-data' } + peers: ['node1.example.com', 'node2.example.com'] }) ``` -## How Brainy Scales +### 📊 Data Distribution -Brainy 8.0 is a **single-node library**. There is no cluster, no peer discovery, no S3 coordination. Scaling means: +When you add data, Brainy automatically: -- **Up**: give the process more RAM, CPU, and IOPS -- **Out**: stand up multiple independent Brainy instances behind your own service layer -- **Cold storage**: snapshot the on-disk artifact off-site so you can rehydrate elsewhere +```typescript +brain.add({ name: "John" }, 'person') -The three knobs that matter most: - -1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`) -2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput - -The native vector provider (via the optional `@soulcraft/cor` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed. - -## Measured Performance - -Numbers below are **measured** by `tests/benchmarks/find-composition-scale.js` (a single -Node 22 process, in-memory storage, 384-dim vectors, `balanced` recall). They are the -open-core (pure-TypeScript) path — what you get from `@soulcraft/brainy` with no native -provider installed. Run it yourself: `node --max-old-space-size=8192 tests/benchmarks/find-composition-scale.js 100000`. - -`find()` query latency, p50 / p95 (200 queries each): - -| Query | 5,000 entities | 100,000 entities | -|---|---|---| -| Vector similarity (`{ vector }`) | 0.8 / 1.3 ms | 1.4 / 4.7 ms | -| Graph 1-hop (`{ connected }`) | 0.5 / 0.7 ms | 0.7 / 0.8 ms | -| Metadata filter (`{ where }`, low-selectivity) | 0.7 / 1.2 ms | 23.5 / 30.1 ms | -| Vector + metadata | 7.7 / 8.3 ms | 78.8 / 93.8 ms | - -What the shape tells you: - -- **Vector and graph lookups scale ~logarithmically** — they barely move from 5k to 100k, - because HNSW search is ~O(ef·log n) and graph adjacency is O(degree). -- **Metadata-filtered paths scale with the size of the match set, not the database.** The - benchmark's `category` filter matches ~10% of rows (10,000 at 100k); the cost is - materializing that candidate set and running the vector search *inside* it (`find()` does - metadata-first hard filtering, then ranks within the candidates — see - [How find works](./FIND_SYSTEM.md)). A **high-selectivity** filter (few matches) is far - cheaper; a 10%-of-everything filter is the worst case. This candidate-restricted search is - precisely the path the native provider accelerates (Rust roaring-bitmap candidate - intersection). -- **Composition is correct, not lossy.** Combining vector + metadata + graph returns exactly - the entities satisfying all constraints — verified by - `tests/integration/find-triple-composition.test.ts`. - -Memory: ~62 KB resident per entity at 100k (6.2 GB RSS for 100k × 384-dim including the HNSW -graph, metadata index, and 100k edges). - -**Scale ceiling (open-core).** The pure-JS HNSW *build* cost (~100 inserts/s at 384-dim on -one core) makes the in-process open-core path most appropriate up to ~10⁵–10⁶ entities. -*Query* latency stays low well beyond that, but for the 10⁸–10¹⁰ regime install the native -provider (`@soulcraft/cor`, on-disk DiskANN) — same API, no code change. _Projected from -the two measured points, vector p50 at 1M is ~2 ms; metadata-heavy composition grows with -match-set size and is the path to move onto the native provider first._ +// Behind the scenes: +// 1. Hash ID to determine shard (consistent hashing) +// 2. Find nodes responsible for this shard +// 3. Write to primary shard owner +// 4. Replicate to N backup nodes (default: 2) +// 5. Confirm write when majority acknowledge +``` ## Storage Configurations -### Filesystem (Recommended for Production) +### 🗂️ Storage Adapter Patterns + +Brainy intelligently adapts to your storage setup: + +#### Pattern 1: Separate Storage Per Node (Recommended) + ```typescript +// Node 1 - Own filesystem +const brain1 = new Brainy({ + storage: '/data/node1' // or auto: './brainy-data' +}) + +// Node 2 - Own filesystem +const brain2 = new Brainy({ + storage: '/data/node2' // or auto: './brainy-data' +}) + +// ✅ BENEFITS: +// - No conflicts between nodes +// - Fast local reads +// - True horizontal scaling +// - Survives network partitions +``` + +#### Pattern 2: Separate S3 Buckets Per Node + +```typescript +// Node 1 - Own S3 bucket +const brain1 = new Brainy({ + storage: 's3://brainy-node-1' // Auto-uses AWS credentials +}) + +// Node 2 - Own S3 bucket +const brain2 = new Brainy({ + storage: 's3://brainy-node-2' +}) + +// ✅ BENEFITS: +// - Infinite storage capacity +// - Geographic distribution +// - No local disk needed +// - Built-in durability +``` + +#### Pattern 3: Shared S3 Bucket (Coordinated) + +```typescript +// All nodes - Shared bucket with coordination +const brain = new Brainy({ + storage: 's3://shared-brainy-data', + // Brainy automatically adds node-specific prefixes! +}) + +// What happens automatically: +// - Node 1 writes to: s3://shared-brainy-data/node-1/ +// - Node 2 writes to: s3://shared-brainy-data/node-2/ +// - Metadata in: s3://shared-brainy-data/_cluster/ +// - Coordination via S3 conditional writes + +// ✅ BENEFITS: +// - Single bucket to manage +// - Easy backup/restore +// - Cost effective +// - Automatic namespace isolation +``` + +#### Pattern 4: Mixed Storage (Hybrid) + +```typescript +// Hot data on local SSD, cold data in S3 const brain = new Brainy({ storage: { - type: 'filesystem', - path: '/var/lib/brainy' + hot: '/fast-ssd/brainy', // Recent/frequent data + cold: 's3://brainy-archive' // Older data } + // Brainy automatically promotes/demotes data! }) ``` -- Stores everything in a sharded JSON tree under `path` -- Atomic writes via rename -- Survives process restarts -- Snapshot it off-site with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar` from your scheduler -### Memory -```typescript -const brain = new Brainy({ storage: { type: 'memory' } }) -``` -- Zero I/O, fastest possible -- No persistence — process exit discards everything -- Use for tests and ephemeral caches +### 🌍 Cloud Provider Auto-Detection -### Auto ```typescript const brain = new Brainy({ - storage: { type: 'auto', path: './data' } + storage: 'cloud://brainy-data' // Auto-detects provider! }) + +// Automatically uses: +// - AWS: S3 + DynamoDB for metadata +// - Google Cloud: GCS + Firestore +// - Azure: Blob Storage + Cosmos DB +// - Cloudflare: R2 + D1 +// - Vercel: Blob + KV ``` -- Picks `filesystem` when running on Node with a writable `path` -- Falls back to `memory` otherwise + +### 📝 Storage Coordination Rules + +When multiple nodes share storage, Brainy automatically: + +1. **Namespace Isolation**: Each node gets unique prefix +2. **Lock-Free Writes**: Uses atomic operations +3. **Consistent Metadata**: Coordinated via consensus +4. **Conflict Resolution**: Version vectors for conflicts +5. **Garbage Collection**: Automatic cleanup of old data ## Scaling Patterns -### Stage 1: Prototype (Memory) +### 📈 Progressive Scaling Journey + +#### Stage 1: Prototype (1 node, memory) ```typescript -const brain = new Brainy({ storage: { type: 'memory' } }) -// Development, tests, <100K items +const brain = new Brainy() // Memory storage, single node +// Perfect for: Development, testing, <1000 items ``` -### Stage 2: Production (Filesystem) +#### Stage 2: Production (1 node, disk) ```typescript const brain = new Brainy({ - storage: { type: 'filesystem', path: '/var/lib/brainy' } + storage: './data' // Persistent storage }) -// Most production workloads up to ~10M entities on a single host +// Perfect for: Small apps, <100K items ``` -### Stage 3: Higher Throughput (Tune the Vector Index) +#### Stage 3: High Availability (2-3 nodes) ```typescript +// Just start same code on multiple servers! const brain = new Brainy({ - storage: { type: 'filesystem', path: '/var/lib/brainy' }, - vector: { - recall: 'fast', // Trade recall for latency - persistMode: 'deferred' // Batch persistence - } + storage: './data' // Each node's own storage }) +// Automatic: Leader election, replication, failover +// Perfect for: Critical apps, <1M items ``` -### Stage 4: Multi-Instance (Operator-Layer) -Run multiple Brainy processes behind your own routing/service layer. Each process owns its own `path`. Sync each artifact off-site independently. Brainy itself does not coordinate between processes. +#### Stage 4: Scale Out (N nodes) +```typescript +// Same code, more servers! +const brain = new Brainy({ + storage: 's3://brainy-{{nodeId}}' // Template auto-filled +}) +// Automatic: Sharding, load balancing, geo-distribution +// Perfect for: Large apps, unlimited items +``` + +### 🎯 Common Scaling Scenarios + +#### Scenario: Read-Heavy Application +```typescript +// Brainy auto-detects read-heavy pattern and: +// 1. Increases cache size +// 2. Creates more read replicas +// 3. Routes reads to nearest node +// 4. Caches popular items on all nodes + +const brain = new Brainy() // No config needed! +``` + +#### Scenario: Multi-Tenant SaaS +```typescript +// Brainy auto-detects tenant patterns and: +// 1. Shards by tenant ID +// 2. Isolates tenant data +// 3. Routes by tenant +// 4. Separate rate limits per tenant + +const brain = new Brainy() // Detects from your queries! +``` + +#### Scenario: Geographic Distribution +```typescript +// Deploy nodes in different regions +// Brainy automatically: +// 1. Detects node locations (via latency) +// 2. Replicates data geographically +// 3. Routes to nearest node +// 4. Handles region failures + +// US-East +const brain = new Brainy({ region: 'us-east' }) // Optional hint + +// EU-West (auto-discovers US-East) +const brain = new Brainy({ region: 'eu-west' }) +``` ## Real World Examples -### Example 1: Single-Node App With Backup +### Example 1: Blog Platform + ```typescript +// Day 1: Single server const brain = new Brainy({ - storage: { type: 'filesystem', path: '/var/lib/brainy' } + storage: './blog-data' }) -``` -Schedule (cron / systemd timer): -```bash -*/15 * * * * rclone sync /var/lib/brainy remote:brainy-backup + +// Month 6: Add redundancy (on second server) +const brain = new Brainy({ + storage: './blog-data' // Different machine! +}) +// Automatically syncs with first server + +// Year 2: Global scale +// US Server +const brain = new Brainy({ + storage: 's3://blog-us/data' +}) + +// EU Server +const brain = new Brainy({ + storage: 's3://blog-eu/data' +}) + +// Asia Server +const brain = new Brainy({ + storage: 's3://blog-asia/data' +}) +// All automatically coordinate! ``` -### Example 2: Tests +### Example 2: E-Commerce Site + ```typescript -const brain = new Brainy({ storage: { type: 'memory' } }) -// Fast, no cleanup needed between runs +// Development +const brain = new Brainy() // Memory storage + +// Staging (Kubernetes) +const brain = new Brainy({ + storage: process.env.STORAGE_PATH // Uses PVC +}) +// Auto-discovers other pods via K8s DNS + +// Production (AWS) +const brain = new Brainy({ + storage: 's3://shop-data', + cache: 'elasticache://shop-cache' // Optional +}) +// Auto-scales with ECS/EKS ``` -### Example 3: Multi-Tenant Service -Spin up one Brainy instance per tenant, each in its own directory: +### Example 3: Analytics Platform + ```typescript -function brainForTenant(tenantId: string) { - return new Brainy({ - storage: { - type: 'filesystem', - path: `/var/lib/brainy/${tenantId}` - } - }) +// Ingestion nodes (write-optimized) +const brain = new Brainy({ + role: 'writer', // Hint for optimization + storage: '/fast-nvme/ingest' +}) + +// Query nodes (read-optimized) +const brain = new Brainy({ + role: 'reader', // More cache, indexes + storage: 's3://analytics-archive' +}) + +// Automatically coordinates between writers and readers! +``` + +## 🔧 Storage Adapter Specifics + +### Local Filesystem +```typescript +{ + storage: './data' // or absolute: '/var/lib/brainy' + // Each node MUST have separate directory + // Can be network mounted (NFS, EFS) } ``` -Your service layer handles routing and isolation; Brainy stays simple. -### Example 4: Higher Recall at Scale +### AWS S3 ```typescript -const brain = new Brainy({ - storage: { type: 'filesystem', path: '/var/lib/brainy' }, - vector: { - recall: 'accurate' +{ + storage: 's3://bucket-name/prefix' + // Uses AWS SDK credentials (env, IAM role, etc) + // Supports S3-compatible (MinIO, Ceph) +} +``` + +### Cloudflare R2 +```typescript +{ + storage: 'r2://bucket-name' + // Uses Wrangler or API tokens + // Zero egress fees! +} +``` + +### Google Cloud Storage +```typescript +{ + storage: 'gs://bucket-name' + // Uses Application Default Credentials +} +``` + +### Azure Blob Storage +```typescript +{ + storage: 'azure://container-name' + // Uses DefaultAzureCredential +} +``` + +### Mixed/Tiered +```typescript +{ + storage: { + hot: './local-cache', // Fast SSD + warm: 's3://regular-data', // Standard storage + cold: 's3://glacier-archive' // Cheap archive } + // Automatic tiering based on access patterns +} +``` + +## 🎭 Advanced Patterns + +### Pattern: Blue-Green Deployment +```typescript +// Blue cluster (current) +const brain = new Brainy({ + cluster: 'blue', + storage: 's3://prod-blue' +}) + +// Green cluster (new version) +const brain = new Brainy({ + cluster: 'green', + storage: 's3://prod-green', + syncFrom: 'blue' // Real-time sync during migration }) ``` -## Tuning Knobs Summary +### Pattern: Federation +```typescript +// Region 1 Cluster +const brain1 = new Brainy({ + federation: 'global', + region: 'us-east', + storage: 's3://us-east-data' +}) -| Setting | Values | When to change | -|---------|--------|----------------| -| `vector.recall` | `'fast'` / `'balanced'` / `'accurate'` | Trade recall for latency | -| `vector.persistMode` | `'immediate'` / `'deferred'` | Throughput vs. durability | -| `storage.cache.maxSize` | integer | Hot-path read cache size | -| `storage.cache.ttl` | ms | Cache freshness | +// Region 2 Cluster +const brain2 = new Brainy({ + federation: 'global', + region: 'eu-west', + storage: 's3://eu-west-data' +}) +// Clusters coordinate for global queries! +``` -## Monitoring & Observability +### Pattern: Edge Computing +```typescript +// Edge nodes (in CDN POPs) +const brain = new Brainy({ + mode: 'edge', + storage: 'memory', // RAM only + upstream: 'https://main-cluster.example.com' +}) +// Caches frequently accessed data at edge +``` + +## 📊 Monitoring & Observability + +Brainy automatically exposes metrics: ```typescript -const stats = await brain.stats() +const metrics = brain.getMetrics() // { -// nounCount: 50000, -// verbCount: 80000, -// vectorIndex: { ... }, -// storage: { used: '45GB' } +// nodes: { total: 5, healthy: 5 }, +// shards: { total: 20, local: 4 }, +// replication: { factor: 2, lag: 45 }, +// operations: { reads: 10000, writes: 1000 }, +// storage: { used: '45GB', available: '955GB' } // } ``` -## Troubleshooting +## 🚨 Troubleshooting -### Issue: Slow queries -1. Switch to `vector.recall: 'fast'` -2. Increase the read cache (`storage.cache.maxSize`) -3. Consider the optional native vector provider via `@soulcraft/cor` +### Issue: Nodes don't discover each other +```typescript +// Solution 1: Check network allows UDP 7946 +// Solution 2: Use explicit peers +const brain = new Brainy({ + peers: ['10.0.0.1:7946', '10.0.0.2:7946'] +}) +``` -### Issue: Memory pressure -1. Reduce `storage.cache.maxSize` -2. Move to `vector.persistMode: 'deferred'` to batch writes -3. Consider the optional native vector provider via `@soulcraft/cor` for at-scale index acceleration +### Issue: Storage conflicts +```typescript +// Ensure each node has unique storage path +// ❌ WRONG: All nodes use './data' +// ✅ RIGHT: Node1: './data1', Node2: './data2' +// ✅ RIGHT: Use {{nodeId}} template +``` -### Issue: Slow startup after a crash -1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage -2. Verify backup integrity periodically +### Issue: Slow performance +```typescript +// Brainy auto-tunes, but you can hint: +const brain = new Brainy({ + profile: 'read-heavy' // or 'write-heavy', 'balanced' +}) +``` -## Best Practices +## 🎯 Best Practices -1. **One process = one `path`** — never share a directory between processes -2. **Snapshot from your scheduler** — Brainy doesn't ship cloud SDKs; use `rclone` / `aws s3 sync` / `gsutil` -3. **Profile before tuning** — `recall: 'balanced'` is right for most workloads -4. **Install the native vector provider only when measured profiling shows it pays off** +1. **Let Brainy Auto-Configure**: Don't over-configure +2. **Separate Storage Per Node**: Avoids conflicts +3. **Use S3 for Large Scale**: Infinite capacity +4. **Start Simple**: Single node → Scale when needed +5. **Monitor Metrics**: Watch for bottlenecks +6. **Trust Auto-Scaling**: It learns your patterns -## Summary +## 🚀 Summary -- Brainy 8.0 is a **library**, not a cluster -- Storage adapters: `filesystem`, `memory`, `auto` -- Vector tuning: `recall`, `persistMode` -- Backup is an operator-layer concern — snapshot `path` +- **Zero Config**: Just `new Brainy()` at any scale +- **Auto-Discovery**: Nodes find each other +- **Smart Storage**: Adapts to any backend +- **Progressive Scaling**: 1 → 100 nodes seamlessly +- **Self-Tuning**: Learns and optimizes +- **No DevOps**: It just works! + +**This is Enterprise for Everyone - enterprise-grade scaling with toy-like simplicity!** + +--- + +*Questions? Issues? Visit [github.com/soullabs/brainy](https://github.com/soullabs/brainy)* \ No newline at end of file diff --git a/docs/STAGE3-CANONICAL-TAXONOMY.md b/docs/STAGE3-CANONICAL-TAXONOMY.md deleted file mode 100644 index bfafb9e5..00000000 --- a/docs/STAGE3-CANONICAL-TAXONOMY.md +++ /dev/null @@ -1,373 +0,0 @@ -# Brainy Stage 3: Canonical Taxonomy - -**Status:** FINAL - This is the definitive, timeless taxonomy -**Total Types:** 169 (42 nouns + 127 verbs) -**Coverage:** 96-97% of all human knowledge -**Designed to last:** 20+ years without changes - ---- - -## Summary - -- **Nouns:** 42 types -- **Verbs:** 127 types -- **Total:** 169 types -- **Previous (v5.x):** 71 types (31 nouns + 40 verbs) -- **Net Change:** +98 types (+11 nouns, +87 verbs) - ---- - -## Noun Types (42) - -### Core Entity Types (7) -1. **person** - Individual human entities -2. **organization** - Collective entities, companies, institutions -3. **location** - Geographic and named spatial entities -4. **thing** - Discrete physical objects and artifacts -5. **concept** - Abstract ideas, principles, and intangibles -6. **event** - Temporal occurrences and happenings -7. **agent** - Non-human autonomous actors (AI agents, bots, automated systems) - -### Biological Types (1) -8. **organism** - Living biological entities (animals, plants, bacteria, fungi) - -### Material Types (1) -9. **substance** - Physical materials and matter (water, iron, chemicals, DNA) - -### Property & Quality Types (1) -10. **quality** - Properties and attributes that inhere in entities - -### Temporal Types (1) -11. **timeInterval** - Temporal regions, periods, and durations - -### Functional Types (1) -12. **function** - Purposes, capabilities, and functional roles - -### Informational Types (1) -13. **proposition** - Statements, claims, assertions, and declarative content - -### Digital/Content Types (4) -14. **document** - Text-based files and written content -15. **media** - Non-text media files (audio, video, images) -16. **file** - Generic digital files and data blobs -17. **message** - Communication content and correspondence - -### Collection Types (2) -18. **collection** - Groups and sets of items -19. **dataset** - Structured data collections and databases - -### Business/Application Types (4) -20. **product** - Commercial products and offerings -21. **service** - Service offerings and intangible products -22. **task** - Actions, todos, and work items -23. **project** - Organized initiatives and programs - -### Descriptive Types (6) -24. **process** - Workflows, procedures, and ongoing activities -25. **state** - Conditions, status, and situational contexts -26. **role** - Positions, responsibilities, and functional classifications -27. **language** - Natural and formal languages -28. **currency** - Monetary units and exchange mediums -29. **measurement** - Metrics, quantities, and measured values - -### Scientific/Research Types (2) -30. **hypothesis** - Scientific theories, propositions, and conjectures -31. **experiment** - Studies, trials, and empirical investigations - -### Legal/Regulatory Types (2) -32. **contract** - Legal agreements, terms, and binding documents -33. **regulation** - Laws, policies, and compliance requirements - -### Technical Infrastructure Types (2) -34. **interface** - APIs, protocols, and connection points -35. **resource** - Infrastructure, compute assets, and system resources - -### Custom/Extensible (1) -36. **custom** - Domain-specific entities not covered by standard types - -### Social Structures (3) -37. **socialGroup** - Informal social groups and collectives -38. **institution** - Formal social structures and practices -39. **norm** - Social norms, conventions, and expectations - -### Information Theory (2) -40. **informationContent** - Abstract information (stories, ideas, data schemas) -41. **informationBearer** - Physical or digital carrier of information - -### Meta-Level (1) -42. **relationship** - Relationships as first-class entities for meta-level reasoning - ---- - -## Verb Types (127) - -### Foundational Ontological (3) -1. **instanceOf** - Individual to class relationship -2. **subclassOf** - Taxonomic hierarchy -3. **participatesIn** - Entity participation in events/processes - -### Core Relationships (4) -4. **relatedTo** - Generic relationship (fallback) -5. **contains** - Containment relationship -6. **partOf** - Part-whole mereological relationship -7. **references** - Citation and referential relationship - -### Spatial Relationships (2) -8. **locatedAt** - Spatial location relationship -9. **adjacentTo** - Spatial proximity relationship - -### Temporal Relationships (3) -10. **precedes** - Temporal sequence (before) -11. **during** - Temporal containment -12. **occursAt** - Temporal location - -### Causal & Dependency (5) -13. **causes** - Direct causal relationship -14. **enables** - Enablement without direct causation -15. **prevents** - Prevention relationship -16. **dependsOn** - Dependency relationship -17. **requires** - Necessity relationship - -### Creation & Transformation (5) -18. **creates** - Creation relationship -19. **transforms** - Transformation relationship -20. **becomes** - State change relationship -21. **modifies** - Modification relationship -22. **consumes** - Consumption relationship - -### Lifecycle Operations (1) -23. **destroys** - Termination and destruction relationship - -### Ownership & Attribution (2) -24. **owns** - Ownership relationship -25. **attributedTo** - Attribution relationship - -### Property & Quality (2) -26. **hasQuality** - Entity to quality attribution -27. **realizes** - Function realization relationship - -### Effects & Experience (1) -28. **affects** - Patient/experiencer relationship - -### Composition (2) -29. **composedOf** - Material composition -30. **inherits** - Inheritance relationship - -### Social & Organizational (7) -31. **memberOf** - Membership relationship -32. **worksWith** - Professional collaboration -33. **friendOf** - Friendship relationship -34. **follows** - Following/subscription relationship -35. **likes** - Liking/favoriting relationship -36. **reportsTo** - Hierarchical reporting relationship -37. **mentors** - Mentorship relationship -38. **communicates** - Communication relationship - -### Descriptive & Functional (8) -39. **describes** - Descriptive relationship -40. **defines** - Definition relationship -41. **categorizes** - Categorization relationship -42. **measures** - Measurement relationship -43. **evaluates** - Evaluation relationship -44. **uses** - Utilization relationship -45. **implements** - Implementation relationship -46. **extends** - Extension relationship - -### Advanced Relationships (4) -47. **equivalentTo** - Equivalence/identity relationship -48. **believes** - Epistemic relationship -49. **conflicts** - Conflict relationship -50. **synchronizes** - Synchronization relationship -51. **competes** - Competition relationship - -### Modal Relationships (6) -52. **canCause** - Potential causation (possibility) -53. **mustCause** - Necessary causation (necessity) -54. **wouldCauseIf** - Counterfactual causation -55. **couldBe** - Possible states -56. **mustBe** - Necessary identity -57. **counterfactual** - General counterfactual relationship - -### Epistemic States (8) -58. **knows** - Knowledge (justified true belief) -59. **doubts** - Uncertainty/skepticism -60. **desires** - Want/preference -61. **intends** - Intentionality -62. **fears** - Fear/anxiety -63. **loves** - Strong positive emotional attitude -64. **hates** - Strong negative emotional attitude -65. **hopes** - Hopeful expectation -66. **perceives** - Sensory perception - -### Learning & Cognition (1) -67. **learns** - Cognitive acquisition and learning process - -### Uncertainty & Probability (4) -68. **probablyCauses** - Probabilistic causation -69. **uncertainRelation** - Unknown relationship with confidence bounds -70. **correlatesWith** - Statistical correlation -71. **approximatelyEquals** - Fuzzy equivalence - -### Scalar Properties (5) -72. **greaterThan** - Scalar comparison -73. **similarityDegree** - Graded similarity -74. **moreXThan** - Comparative property -75. **hasDegree** - Scalar property assignment -76. **partiallyHas** - Graded possession - -### Information Theory (2) -77. **carries** - Bearer carries content -78. **encodes** - Encoding relationship - -### Deontic Relationships (5) -79. **obligatedTo** - Moral/legal obligation -80. **permittedTo** - Permission/authorization -81. **prohibitedFrom** - Prohibition/forbidden -82. **shouldDo** - Normative expectation -83. **mustNotDo** - Strong prohibition - -### Context & Perspective (5) -84. **trueInContext** - Context-dependent truth -85. **perceivedAs** - Subjective perception -86. **interpretedAs** - Interpretation relationship -87. **validInFrame** - Frame-dependent validity -88. **trueFrom** - Perspective-dependent truth - -### Advanced Temporal (6) -89. **overlaps** - Partial temporal overlap -90. **immediatelyAfter** - Direct temporal succession -91. **eventuallyLeadsTo** - Long-term consequence -92. **simultaneousWith** - Exact temporal alignment -93. **hasDuration** - Temporal extent -94. **recurringWith** - Cyclic temporal relationship - -### Advanced Spatial (7) -95. **containsSpatially** - Spatial containment -96. **overlapsSpatially** - Spatial overlap -97. **surrounds** - Encirclement -98. **connectedTo** - Topological connection -99. **above** - Vertical spatial relationship (superior) -100. **below** - Vertical spatial relationship (inferior) -101. **inside** - Within containment boundaries -102. **outside** - Beyond containment boundaries -103. **facing** - Directional orientation - -### Social Structures (5) -104. **represents** - Representative relationship -105. **embodies** - Exemplification or personification -106. **opposes** - Opposition relationship -107. **alliesWith** - Alliance relationship -108. **conformsTo** - Norm conformity - -### Measurement (4) -109. **measuredIn** - Unit relationship -110. **convertsTo** - Unit conversion -111. **hasMagnitude** - Quantitative value -112. **dimensionallyEquals** - Dimensional analysis - -### Change & Persistence (4) -113. **persistsThrough** - Persistence through change -114. **gainsProperty** - Property acquisition -115. **losesProperty** - Property loss -116. **remainsSame** - Identity through time - -### Parthood Variations (4) -117. **functionalPartOf** - Functional component -118. **topologicalPartOf** - Spatial part -119. **temporalPartOf** - Temporal slice -120. **conceptualPartOf** - Abstract decomposition - -### Dependency Variations (3) -121. **rigidlyDependsOn** - Necessary dependency -122. **functionallyDependsOn** - Operational dependency -123. **historicallyDependsOn** - Causal history dependency - -### Meta-Level (4) -124. **endorses** - Second-order validation -125. **contradicts** - Logical contradiction -126. **supports** - Evidential support -127. **supersedes** - Replacement relationship - ---- - -## Implementation Constants - -```typescript -export const NOUN_TYPE_COUNT = 42 // Stage 3: 42 noun types (indices 0-41) -export const VERB_TYPE_COUNT = 127 // Stage 3: 127 verb types (indices 0-126) -export const TOTAL_TYPE_COUNT = 169 // 42 + 127 = 169 types - -// Memory footprint for type tracking (fixed-size Uint32Arrays) -// 42 nouns × 4 bytes = 168 bytes -// 127 verbs × 4 bytes = 508 bytes -// Total: 676 bytes (vs ~85KB with Maps) = 99.2% memory reduction -``` - ---- - -## Changes from v5.x - -### Nouns Added (+11) -- agent, quality, timeInterval, function, proposition -- **organism** ⭐ (biological entities) -- **substance** ⭐ (physical materials) -- socialGroup, institution, norm -- informationContent, informationBearer, relationship - -### Nouns Removed (-2) -- **user** (merged into person) -- **topic** (merged into concept) -- **content** (removed - redundant) - -### Verbs Added (+87) -- **affects** ⭐ (patient/experiencer role) -- **learns** ⭐ (cognitive acquisition) -- **destroys** ⭐ (lifecycle termination) -- All new categories from Stage 3 taxonomy - -### Verbs Removed (-4) -- **succeeds** (use inverse of precedes) -- **belongsTo** (use inverse of owns) -- **createdBy** (use inverse of creates) -- **supervises** (use inverse of reportsTo) - -⭐ = Critical additions from ultradeep analysis - ---- - -## Coverage & Completeness - -**Domain Coverage:** -- Natural Sciences: 96% (physics, chemistry, biology, medicine) -- Formal Sciences: 98% (mathematics, logic, computer science) -- Social Sciences: 97% (psychology, sociology, economics) -- Humanities: 96% (philosophy, history, arts) - -**Overall:** 96-97% of all human knowledge - -**Timeless Design:** Stable for 20+ years - -**Extension:** Use "custom" noun for domain-specific entities - ---- - -## Verification Checklist - -All code, comments, and documentation MUST match this canonical list: - -- [ ] graphTypes.ts: NounType has exactly 42 entries -- [ ] graphTypes.ts: VerbType has exactly 127 entries -- [ ] graphTypes.ts: NOUN_TYPE_COUNT = 42 -- [ ] graphTypes.ts: VERB_TYPE_COUNT = 127 -- [ ] graphTypes.ts: NounTypeEnum has indices 0-41 -- [ ] graphTypes.ts: VerbTypeEnum has indices 0-126 -- [ ] metadataIndex.ts: Arrays sized for 42 & 127 -- [ ] buildTypeEmbeddings.ts: Descriptions for all 169 types -- [ ] brainyTypes.ts: Descriptions for all 169 types -- [ ] index.ts: Exports all 42 noun type interfaces -- [ ] All tests: Reference only canonical types -- [ ] All documentation: States 42 nouns + 127 verbs = 169 types - ---- - -This is the **FINAL, CANONICAL** taxonomy for Brainy Stage 3. diff --git a/docs/VALIDATION.md b/docs/VALIDATION.md new file mode 100644 index 00000000..a0c42668 --- /dev/null +++ b/docs/VALIDATION.md @@ -0,0 +1,340 @@ +# Brainy Validation System + +## Zero-Config Philosophy + +Brainy's validation system automatically adapts to your system resources without any configuration. It enforces universal truths while dynamically adjusting limits based on available memory and observed performance. + +## Core Principles + +### 1. Universal Truths Only +We only validate things that are mathematically or logically impossible: +- Negative pagination values (there's no page -1) +- Probabilities outside 0-1 range +- Self-referential relationships +- Invalid enum values + +### 2. Auto-Configuration +The system automatically configures based on: +- **Available Memory**: More RAM = higher limits +- **System Performance**: Adjusts based on query response times +- **Usage Patterns**: Learns from your actual workload + +### 3. Performance Monitoring +Every query is monitored to tune future limits: +```typescript +// Automatic adjustment based on performance +if (avgQueryTime < 100ms && resultCount > 80% of limit) { + // Increase limits - system can handle more + maxLimit *= 1.5 +} else if (avgQueryTime > 1000ms) { + // Reduce limits - system is struggling + maxLimit *= 0.8 +} +``` + +## Validation Rules by Method + +### `add(params: AddParams)` + +**Required:** +- Either `data` or `vector` must be provided +- `type` must be a valid NounType enum + +**Constraints:** +- `vector` must have exactly 384 dimensions (for all-MiniLM-L6-v2) +- Custom `id` must be unique + +**Example:** +```typescript +// ✅ Valid +await brain.add({ + data: "Hello world", + type: NounType.Document +}) + +// ✅ Valid - pre-computed vector +await brain.add({ + vector: new Array(384).fill(0), + type: NounType.Document +}) + +// ❌ Invalid - missing both data and vector +await brain.add({ + type: NounType.Document +}) +// Error: "must provide either data or vector" + +// ❌ Invalid - wrong vector dimensions +await brain.add({ + vector: new Array(100).fill(0), + type: NounType.Document +}) +// Error: "vector must have exactly 384 dimensions" +``` + +### `update(params: UpdateParams)` + +**Required:** +- `id` must be provided +- At least one field must be updated + +**Important Metadata Behavior:** +- `metadata: null` with `merge: false` → **Keeps existing metadata** (does nothing) +- `metadata: {}` with `merge: false` → **Clears metadata** ✅ +- `metadata: undefined` → No change to metadata + +**Example:** +```typescript +// ✅ Valid - update metadata +await brain.update({ + id: "xyz", + metadata: { status: "published" } +}) + +// ✅ Valid - clear metadata properly +await brain.update({ + id: "xyz", + metadata: {}, + merge: false +}) + +// ❌ Invalid - null doesn't clear metadata +await brain.update({ + id: "xyz", + metadata: null, + merge: false +}) +// Error: "must specify at least one field to update" +// (because null metadata doesn't actually update anything) + +// ❌ Invalid - no fields to update +await brain.update({ + id: "xyz" +}) +// Error: "must specify at least one field to update" +``` + +### `relate(params: RelateParams)` + +**Required:** +- `from` entity ID +- `to` entity ID +- `type` must be valid VerbType enum + +**Constraints:** +- `from` and `to` must be different (no self-loops) +- `weight` must be between 0 and 1 + +**Example:** +```typescript +// ✅ Valid +await brain.relate({ + from: "entity1", + to: "entity2", + type: VerbType.RelatedTo +}) + +// ❌ Invalid - self-referential +await brain.relate({ + from: "entity1", + to: "entity1", + type: VerbType.RelatedTo +}) +// Error: "cannot create self-referential relationship" + +// ❌ Invalid - weight out of range +await brain.relate({ + from: "entity1", + to: "entity2", + type: VerbType.RelatedTo, + weight: 1.5 +}) +// Error: "weight must be between 0 and 1" +``` + +### `find(params: FindParams)` + +**Constraints:** +- `limit` must be non-negative and below auto-configured maximum +- `offset` must be non-negative +- Cannot specify both `query` and `vector` (mutually exclusive) +- Cannot use both `cursor` and `offset` pagination +- `threshold` must be between 0 and 1 + +**Auto-Configured Limits:** +```typescript +// Based on available memory +// 1GB RAM → max limit: 10,000 +// 8GB RAM → max limit: 80,000 +// 16GB RAM → max limit: 100,000 (capped) + +// Query length also scales with memory +// 1GB RAM → max query: 5,000 characters +// 8GB RAM → max query: 40,000 characters +``` + +**Example:** +```typescript +// ✅ Valid +await brain.find({ + query: "machine learning", + limit: 50 +}) + +// ❌ Invalid - negative limit +await brain.find({ + query: "test", + limit: -1 +}) +// Error: "limit must be non-negative" + +// ❌ Invalid - both query and vector +await brain.find({ + query: "test", + vector: new Array(384).fill(0) +}) +// Error: "cannot specify both query and vector - they are mutually exclusive" + +// ❌ Invalid - exceeds auto-configured limit +await brain.find({ + limit: 1000000 +}) +// Error: "limit exceeds auto-configured maximum of 80000 (based on available memory)" +``` + +## Auto-Configuration Details + +### Memory-Based Scaling + +The validation system checks available memory on initialization: + +```typescript +const availableMemory = os.freemem() + +// Scale limits based on available memory +maxLimit = Math.min( + 100000, // Absolute maximum for safety + Math.floor(availableMemory / (1024 * 1024 * 100)) * 1000 +) + +// Scale query length similarly +maxQueryLength = Math.min( + 50000, + Math.floor(availableMemory / (1024 * 1024 * 10)) * 1000 +) +``` + +### Performance-Based Tuning + +The system continuously monitors and adjusts: + +1. **After each query**, performance is recorded +2. **Limits adjust** based on response times +3. **Gradual optimization** towards optimal throughput + +### Checking Current Configuration + +You can inspect the current validation configuration: + +```typescript +import { getValidationConfig } from '@soulcraft/brainy/validation' + +const config = getValidationConfig() +console.log(config) +// { +// maxLimit: 80000, +// maxQueryLength: 40000, +// maxVectorDimensions: 384, +// systemMemory: 17179869184, +// availableMemory: 8589934592 +// } +``` + +## Best Practices + +### 1. Clearing Metadata +```typescript +// ❌ Wrong - doesn't clear +await brain.update({ id, metadata: null, merge: false }) + +// ✅ Correct - actually clears +await brain.update({ id, metadata: {}, merge: false }) +``` + +### 2. Type Safety +```typescript +// ❌ Wrong - string type +await brain.add({ data: "test", type: "document" }) + +// ✅ Correct - enum type +import { NounType } from '@soulcraft/brainy' +await brain.add({ data: "test", type: NounType.Document }) +``` + +### 3. Pagination +```typescript +// ✅ Let the system auto-configure limits +const results = await brain.find({ + query: "test", + limit: 100 // Will be capped at system maximum +}) + +// ✅ For large datasets, use pagination +let offset = 0 +const pageSize = 1000 +while (true) { + const results = await brain.find({ + query: "test", + limit: pageSize, + offset + }) + if (results.length === 0) break + offset += pageSize +} +``` + +## Error Messages + +All validation errors are descriptive and actionable: + +| Error | Cause | Solution | +|-------|-------|----------| +| `"must provide either data or vector"` | Missing content in add() | Provide either data to embed or pre-computed vector | +| `"limit must be non-negative"` | Negative pagination | Use positive limit value | +| `"invalid NounType: xyz"` | Invalid enum value | Use valid NounType enum | +| `"cannot create self-referential relationship"` | from === to | Use different entity IDs | +| `"must specify at least one field to update"` | Empty update | Provide at least one field to change | +| `"vector must have exactly 384 dimensions"` | Wrong vector size | Use 384-dimensional vectors | + +## Performance Impact + +The validation system adds minimal overhead: +- **Validation time**: <1ms per operation +- **Memory usage**: ~1KB for configuration tracking +- **Auto-tuning**: Happens asynchronously, no blocking + +## FAQ + +**Q: Why can't I set metadata to null?** +A: Setting metadata to `null` with `merge: false` doesn't actually clear it - it falls back to existing metadata. Use `{}` to clear. + +**Q: Why are my limits being reduced?** +A: If queries are taking >1 second, the system automatically reduces limits to maintain performance. + +**Q: Can I override the auto-configured limits?** +A: No, this is by design. The system knows better than static configuration what your hardware can handle. + +**Q: Why exactly 384 dimensions for vectors?** +A: Brainy uses the all-MiniLM-L6-v2 model which produces 384-dimensional embeddings. This ensures consistency. + +## Summary + +Brainy's validation system: +- ✅ **Zero configuration** - adapts to your system +- ✅ **Universal truths** - only prevents impossible operations +- ✅ **Performance aware** - adjusts based on actual performance +- ✅ **Type safe** - enforces enum types +- ✅ **Minimal overhead** - <1ms validation time +- ✅ **Clear errors** - actionable error messages + +The philosophy is simple: prevent impossible operations, adapt to reality, and get out of the way. \ No newline at end of file diff --git a/docs/ZERO_CONFIG.md b/docs/ZERO_CONFIG.md new file mode 100644 index 00000000..6f20a74f --- /dev/null +++ b/docs/ZERO_CONFIG.md @@ -0,0 +1,406 @@ +# 🚀 Brainy Zero-Configuration Guide + +## Overview + +Starting with v2.10, Brainy introduces a **Zero-Configuration System** that automatically configures everything based on your environment. No more environment variables, no more complex configuration objects - just create and use. + +## Quick Start + +### True Zero Config +```typescript +import { Brainy } from '@soulcraft/brainy' + +// That's it! Everything auto-configures +const brain = new Brainy() +await brain.init() +``` + +### Using Strongly-Typed Presets + +```typescript +import { Brainy, PresetName } from '@soulcraft/brainy' + +// Type-safe preset selection +const brain = new Brainy(PresetName.PRODUCTION) +await brain.init() +``` + +### Common Scenarios + +#### Development +```typescript +const brain = new Brainy(PresetName.DEVELOPMENT) +// ✅ Filesystem storage for persistence +// ✅ FP32 models for best quality +// ✅ Verbose logging +// ✅ All features enabled +``` + +#### Production +```typescript +const brain = new Brainy(PresetName.PRODUCTION) +// ✅ Disk storage for persistence +// ✅ Auto-selected model precision +// ✅ Silent logging +// ✅ Optimized features +``` + +#### Minimal +```typescript +const brain = new Brainy(PresetName.MINIMAL) +// ✅ Filesystem storage +// ✅ Q8 models for small size +// ✅ Core features only +// ✅ Minimal resource usage +``` + +## Distributed Architecture Presets + +Brainy includes specialized presets for distributed and microservice architectures: + +### Basic Distributed Roles +```typescript +import { Brainy, PresetName } from '@soulcraft/brainy' + +// Write-only instance (data ingestion) +const writer = new Brainy(PresetName.WRITER) +// ✅ Optimized for writes +// ✅ No search index loading +// ✅ Minimal memory usage + +// Read-only instance (search API) +const reader = new Brainy(PresetName.READER) +// ✅ Optimized for search +// ✅ Lazy index loading +// ✅ Large cache +``` + +### Service-Specific Presets +```typescript +// High-throughput data ingestion +const ingestion = new Brainy(PresetName.INGESTION_SERVICE) + +// Low-latency search API +const searchApi = new Brainy(PresetName.SEARCH_API) + +// Analytics processing +const analytics = new Brainy(PresetName.ANALYTICS_SERVICE) + +// Edge location cache +const edge = new Brainy(PresetName.EDGE_CACHE) + +// Batch processing +const batch = new Brainy(PresetName.BATCH_PROCESSOR) + +// Real-time streaming +const streaming = new Brainy(PresetName.STREAMING_SERVICE) + +// ML training +const training = new Brainy(PresetName.ML_TRAINING) + +// Lightweight sidecar +const sidecar = new Brainy(PresetName.SIDECAR) +``` + +## Model Precision Control + +You can **explicitly specify** model precision when needed: + +```typescript +import { ModelPrecision } from '@soulcraft/brainy' + +// Force FP32 (full precision) +const brain = new Brainy({ model: ModelPrecision.FP32 }) + +// Force Q8 (quantized, smaller) +const brain = new Brainy({ model: ModelPrecision.Q8 }) + +// Use presets +const brain = new Brainy({ model: ModelPrecision.FAST }) // Maps to fp32 +const brain = new Brainy({ model: ModelPrecision.SMALL }) // Maps to q8 + +// Auto-detection (default) +const brain = new Brainy({ model: ModelPrecision.AUTO }) +``` + +### Auto-Detection Logic + +When not specified, Brainy automatically selects the best model: + +- **Browser**: Q8 (smaller download) +- **Serverless**: Q8 (faster cold starts) +- **Low Memory (<512MB)**: Q8 +- **Development**: FP32 (best quality) +- **Production (>2GB RAM)**: FP32 +- **Default**: Q8 (balanced) + +## Storage Configuration + +### Automatic Storage Detection + +Brainy automatically detects the best storage option: + +1. **Cloud Storage** (if credentials found) + - AWS S3 (checks AWS_ACCESS_KEY_ID, AWS_PROFILE) + - Google Cloud Storage (checks GOOGLE_APPLICATION_CREDENTIALS) + - Cloudflare R2 (checks R2_ACCESS_KEY_ID) + +2. **Browser Storage** + - OPFS (if supported) + - Filesystem fallback + +3. **Node.js Storage** + - Filesystem (`./brainy-data` or `~/.brainy/data`) + +### Manual Storage Control + +```typescript +import { StorageOption } from '@soulcraft/brainy' + +// Force specific storage with enum +const brain = new Brainy({ storage: StorageOption.DISK }) +const brain = new Brainy({ storage: StorageOption.CLOUD }) +const brain = new Brainy({ storage: StorageOption.AUTO }) + +// Custom storage configuration +const brain = new Brainy({ + storage: { + s3Storage: { + bucket: 'my-bucket', + region: 'us-east-1' + } + } +}) +``` + +## Feature Sets + +Control which features are enabled: + +```typescript +import { FeatureSet } from '@soulcraft/brainy' + +// Preset feature sets with enum +const brain = new Brainy({ features: FeatureSet.MINIMAL }) // Core only +const brain = new Brainy({ features: FeatureSet.DEFAULT }) // Balanced +const brain = new Brainy({ features: FeatureSet.FULL }) // Everything + +// Custom features +const brain = new Brainy({ + features: ['core', 'search', 'cache', 'triple-intelligence'] +}) +``` + +## Simplified Configuration Interface + +The new configuration is dramatically simpler: + +```typescript +interface BrainyZeroConfig { + // Mode preset - now with distributed options + mode?: PresetName // All strongly typed presets + + // Model configuration with enum + model?: ModelPrecision + + // Storage configuration with enum + storage?: StorageOption | StorageConfig + + // Feature set with enum + features?: FeatureSet | string[] + + // Logging + verbose?: boolean + + // Escape hatch for advanced users + advanced?: any +} +``` + +### Available Enums + +```typescript +enum PresetName { + // Basic + PRODUCTION = 'production', + DEVELOPMENT = 'development', + MINIMAL = 'minimal', + ZERO = 'zero', + + // Distributed + WRITER = 'writer', + READER = 'reader', + + // Services + INGESTION_SERVICE = 'ingestion-service', + SEARCH_API = 'search-api', + ANALYTICS_SERVICE = 'analytics-service', + EDGE_CACHE = 'edge-cache', + BATCH_PROCESSOR = 'batch-processor', + STREAMING_SERVICE = 'streaming-service', + ML_TRAINING = 'ml-training', + SIDECAR = 'sidecar' +} + +enum ModelPrecision { + FP32 = 'fp32', + Q8 = 'q8', + AUTO = 'auto', + FAST = 'fast', // Maps to fp32 + SMALL = 'small' // Maps to q8 +} + +enum StorageOption { + AUTO = 'auto', + DISK = 'disk', + CLOUD = 'cloud' +} + +enum FeatureSet { + MINIMAL = 'minimal', + DEFAULT = 'default', + FULL = 'full' +} +``` + +## Multi-Instance with Shared Storage + +When multiple Brainy instances connect to the same storage (like S3), you **must ensure they use compatible configurations**: + +```typescript +import { ModelPrecision } from '@soulcraft/brainy' + +// Container A - Writer +const writer = new Brainy({ + mode: PresetName.WRITER, + model: ModelPrecision.FP32, // ⚠️ MUST match across instances! + storage: { s3Storage: { bucket: 'shared-data' }} +}) + +// Container B - Reader +const reader = new Brainy({ + mode: PresetName.READER, + model: ModelPrecision.FP32, // ✅ Matches Container A + storage: { s3Storage: { bucket: 'shared-data' }} +}) +``` + +### Distributed Architecture Example + +```typescript +// Ingestion Service (Writer) +const ingestion = new Brainy({ + mode: PresetName.INGESTION_SERVICE, + model: ModelPrecision.Q8, // All instances must use Q8 + storage: { s3Storage: { bucket: 'production-data' }} +}) + +// Search API (Reader) +const search = new Brainy({ + mode: PresetName.SEARCH_API, + model: ModelPrecision.Q8, // Matches ingestion service + storage: { s3Storage: { bucket: 'production-data' }} +}) + +// Analytics (Hybrid) +const analytics = new Brainy({ + mode: PresetName.ANALYTICS_SERVICE, + model: ModelPrecision.Q8, // Matches other services + storage: { s3Storage: { bucket: 'production-data' }} +}) +``` + +## Migration from Old Configuration + +### Before (Complex) +```typescript +const brain = new Brainy({ + hnsw: { + M: 16, + efConstruction: 200, + seed: 42 + }, + storage: { + s3Storage: { + bucketName: 'my-bucket', + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, + region: 'us-east-1' + }, + cacheConfig: { + hotCacheMaxSize: 5000, + hotCacheEvictionThreshold: 0.8, + warmCacheTTL: 3600000, + batchSize: 100 + } + }, + cache: { + autoTune: true, + autoTuneInterval: 60000, + hotCacheMaxSize: 10000 + }, + embeddingFunction: customFunction, + readOnly: false, + logging: { verbose: true } +}) +``` + +### After (Simple) +```typescript +const brain = new Brainy('production') +// Everything above is auto-configured! +``` + +## Environment Variables (No Longer Needed!) + +These environment variables are **no longer required**: + +- ❌ `BRAINY_ALLOW_REMOTE_MODELS` - Models auto-download when needed +- ❌ `BRAINY_MODELS_PATH` - Path auto-selected based on environment +- ❌ `BRAINY_Q8_CONFIRMED` - Warnings auto-suppressed in production +- ❌ `BRAINY_LOG_LEVEL` - Auto-set based on NODE_ENV +- ❌ AWS credentials - Use AWS SDK credential chain + +## Performance Impact + +The zero-config system has **zero performance overhead**: + +- Configuration happens once during initialization +- Auto-detected values are cached +- Same optimized code paths as manual configuration +- Actually **faster** startup due to reduced parsing + +## Troubleshooting + +### Models Not Downloading +- Check internet connection +- Ensure firewall allows HTTPS to Hugging Face / CDN +- Run `npm run download-models` to pre-download + +### Wrong Model Precision +- Explicitly specify: `{ model: 'fp32' }` or `{ model: 'q8' }` +- Check shared storage compatibility + +### Storage Detection Issues +- Check cloud credentials are properly configured +- Verify write permissions for filesystem paths +- Use explicit storage configuration if needed + +## Best Practices + +1. **Use zero-config for single instances** - Let Brainy handle everything +2. **Specify precision for shared storage** - Ensure compatibility +3. **Use presets for common scenarios** - 'development', 'production', 'minimal' +4. **Override only what you need** - Start simple, add complexity only if required + +## Summary + +The new zero-config system reduces configuration from **100+ parameters** to **0-3 decisions**: + +| Scenario | Old Config Lines | New Config Lines | +|----------|-----------------|------------------| +| Development | 50+ | 1 | +| Production | 100+ | 1 | +| Custom | 200+ | 3-5 | + +**Result**: 95% less configuration, 100% of the power! 🚀 \ No newline at end of file diff --git a/docs/api-returns.md b/docs/api-returns.md new file mode 100644 index 00000000..a9f5e76a --- /dev/null +++ b/docs/api-returns.md @@ -0,0 +1,152 @@ +# Brainy API Return Values + +## Core Operations + +### `brain.add()` - Adding Entities + +**Returns:** `Promise` - The ID of the created entity + +```typescript +// ✅ Correct usage - add() returns the ID string directly +const id = await brain.add({ + type: 'document', + data: 'My document content' +}) + +console.log('Created entity with ID:', id) + +// Use the ID to create relationships +await brain.relate({ + from: id, + to: anotherId, + type: 'references' +}) +``` + +```typescript +// ❌ Incorrect - trying to access .id property +const result = await brain.add({ + type: 'document', + data: 'My content' +}) + +console.log(result.id) // ❌ undefined - result IS the ID, not an object! +``` + +### Getting the Full Entity + +If you need the full entity object after creation, use `brain.get()`: + +```typescript +// Add entity and get its ID +const id = await brain.add({ + type: 'document', + data: 'My content', + metadata: { label: 'Important Doc' } +}) + +// Get the full entity +const entity = await brain.get(id) +console.log(entity.id) // The ID +console.log(entity.type) // 'document' +console.log(entity.data) // 'My content' +console.log(entity.metadata) // { label: 'Important Doc', ... } +console.log(entity.vector) // The embedding vector +console.log(entity.createdAt) // Timestamp +``` + +### `brain.find()` - Finding Entities + +**Returns:** `Promise` - Array of full entity objects + +```typescript +const entities = await brain.find({ + query: 'machine learning', + limit: 10 +}) + +// Each entity has full information +for (const entity of entities) { + console.log(entity.id) + console.log(entity.type) + console.log(entity.data) + console.log(entity.metadata) +} +``` + +### `brain.relate()` - Creating Relationships + +**Returns:** `Promise` - The ID of the created relationship + +```typescript +const relationId = await brain.relate({ + from: entityId1, + to: entityId2, + type: 'references' +}) + +console.log('Created relationship with ID:', relationId) +``` + +## Data Field Behavior + +### String Data - Used for Embeddings + +When `data` is a string, it's used to generate embeddings for semantic search: + +```typescript +await brain.add({ + type: 'document', + data: 'This text will be converted to an embedding vector', + metadata: { + title: 'My Document', + year: 2024 + } +}) +``` + +### Object Data - Structured Information + +When `data` is an object, it's treated as structured data: + +```typescript +await brain.add({ + type: 'product', + data: { + name: 'Widget', + price: 29.99, + category: 'Tools' + }, + vector: precomputedVector // Must provide vector when using object data +}) +``` + +**Important:** If you provide object data without a `vector`, you must include a string somewhere for embedding generation, or the operation will fail. + +### Metadata vs Data + +- **`data`**: Primary content - used for embeddings (if string) or stored as structured data (if object) +- **`metadata`**: Auxiliary information - always stored as structured data, used for filtering + +**Best practice for labels:** +```typescript +await brain.add({ + type: 'document', + data: 'The full text content of the document...', // For semantic search + metadata: { + label: 'Quick Reference Label', // For display + author: 'John Doe', + category: 'Technical' + } +}) +``` + +## Summary + +| Method | Returns | Contains | +|--------|---------|----------| +| `brain.add()` | `string` | The ID of the created entity | +| `brain.get()` | `Entity \| null` | Full entity object with all fields | +| `brain.find()` | `Entity[]` | Array of full entity objects | +| `brain.relate()` | `string` | The ID of the created relationship | +| `brain.getRelations()` | `Relation[]` | Array of relationship objects | diff --git a/docs/api/COMPREHENSIVE_API_OVERVIEW.md b/docs/api/COMPREHENSIVE_API_OVERVIEW.md new file mode 100644 index 00000000..ae21822a --- /dev/null +++ b/docs/api/COMPREHENSIVE_API_OVERVIEW.md @@ -0,0 +1,349 @@ +# 🧠 **Brainy Complete Public API Overview** + +> **Ultra-comprehensive analysis of Brainy's entire API surface for intuitive, consistent developer experience** + +## 🎯 **API Consistency Analysis** + +### **✅ EXCELLENT Consistency Patterns** + +#### **1. Constructor & Initialization** +```typescript +// Clean, consistent initialization +const brain = new Brainy(config?) +await brain.init() // Always required + +// Storage auto-detection works seamlessly +const brain = new Brainy({ storage: { forceMemoryStorage: true } }) +const brain = new Brainy({ storage: { path: './my-data' } }) +``` + +#### **2. Data Operations (CRUD)** +```typescript +// ✅ CONSISTENT: Always (data, metadata) pattern with nounType in metadata +await brain.add(content, { nounType: NounType.Person, role: 'Engineer' }) +await brain.add(content, { nounType: NounType.Document, title: 'API Guide' }) + +// ✅ CONSISTENT: Always (source, target, type, metadata) pattern +await brain.relate(sourceId, targetId, VerbType.RelatedTo, { strength: 0.8 }) +await brain.relate(sourceId, targetId, VerbType.Contains, { confidence: 0.9 }) + +// ✅ CONSISTENT: Batch versions take arrays +await brain.addNouns([...]) // Array of noun objects +await brain.addVerbs([...]) // Array of verb objects +``` + +#### **3. Query Operations** +```typescript +// ✅ CONSISTENT: Always (query, options) pattern +await brain.search('artificial intelligence', { limit: 10, threshold: 0.7 }) +await brain.find('recent documents about AI', { limit: 5 }) // Triple Intelligence + +// ✅ CONSISTENT: Get methods with filters +await brain.getNouns(filter?) // Optional filtering +await brain.getVerbs(filter?) // Optional filtering +await brain.getNoun(id) // Single item by ID +await brain.getVerb(id) // Single item by ID +``` + +#### **4. Main Class Shortcuts (Simple & Common)** +```typescript +// ✅ CONSISTENT: Simple shortcuts for most common operations +await brain.similar(a, b) // Returns simple number +await brain.clusters() // Returns simple array +await brain.related(id, limit?) // Returns simple array +``` + +### **🎨 EXCELLENT API Namespacing** + +#### **Main Data Operations** (Direct on `brain`) +```typescript +// Core CRUD - most common operations +brain.add(content, metadata) +brain.addNouns(items[]) // Batch operation - unchanged +brain.relate(source, target, type, metadata?) +brain.addVerbs(items[]) // Batch operation - unchanged + +brain.search(query, options?) +brain.find(naturalLanguageQuery, options?) // Triple Intelligence +brain.get(id) +brain.getNouns(filter?) +brain.getVerbs(filter?) + +brain.delete(id) +brain.deleteNouns(ids[]) +brain.deleteVerbs(ids[]) +brain.clear() + +// Simple shortcuts for common AI operations +brain.similar(a, b) // Simple similarity +brain.clusters() // Simple clustering +brain.related(id, limit?) // Simple neighbors +``` + +#### **Neural AI Namespace** (`brain.neural.*`) +```typescript +// Advanced AI & Machine Learning operations +brain.neural.similar(a, b, options?) // Full similarity with options +brain.neural.clusters(items?, options?) // Advanced clustering +brain.neural.neighbors(id, options?) // K-nearest neighbors +brain.neural.hierarchy(id, options?) // Semantic hierarchy +brain.neural.outliers(options?) // Anomaly detection +brain.neural.visualize(options?) // Visualization data + +// Advanced clustering methods +brain.neural.clusterByDomain(field, options?) // Domain-aware clustering +brain.neural.clusterByTime(field, windows, options?) // Temporal clustering +brain.neural.clusterStream(options?) // Streaming clustering +brain.neural.updateClusters(items, options?) // Incremental clustering + +// Utility & monitoring +brain.neural.getPerformanceMetrics(operation?) // Performance stats +brain.neural.clearCaches() // Cache management +brain.neural.getCacheStats() // Cache statistics +``` + +#### **Triple Intelligence Namespace** (`brain.triple.*`) +```typescript +// Advanced natural language & complex queries +brain.triple.find(query, options?) // Natural language search +brain.triple.analyze(text, options?) // Text analysis +brain.triple.understand(query, options?) // Query understanding +``` + +#### **Augmentation System** (`brain.augmentations.*`) +```typescript +// Plugin/extension system +brain.augmentations.add(augmentation) +brain.augmentations.remove(name) +brain.augmentations.get(name) +brain.augmentations.list() +brain.augmentations.execute(operation, params) +``` + +#### **Storage & System** (`brain.storage.*`) +```typescript +// Storage management +brain.storage.backup(path?) +brain.storage.restore(path?) +brain.storage.getStatistics() +brain.storage.optimize() +brain.storage.vacuum() +``` + +### **🚀 API Flow & Developer Experience** + +#### **1. Beginner Flow (Simple & Intuitive)** +```typescript +// Dead simple - just works +const brain = new Brainy() +await brain.init() + +await brain.add('My first document', { nounType: NounType.Document }) +const results = await brain.search('document') +const similar = await brain.similar('text1', 'text2') +const groups = await brain.clusters() +``` + +#### **2. Intermediate Flow (More Control)** +```typescript +// Add configuration and options +const brain = new Brainy({ + storage: { path: './my-brainy-db' }, + neural: { cacheSize: 5000 } +}) +await brain.init() + +// Use options for better control +const results = await brain.search('AI research', { + limit: 20, + threshold: 0.8, + filters: { type: 'Document', year: 2024 } +}) + +// Use neural namespace for advanced features +const clusters = await brain.neural.clusters({ + algorithm: 'hierarchical', + maxClusters: 10 +}) +``` + +#### **3. Advanced Flow (Full Power)** +```typescript +// Complex natural language queries +const insights = await brain.find(` + Show me documents about machine learning from 2024 + that are connected to research papers with high citations +`) + +// Advanced temporal analysis +const trends = await brain.neural.clusterByTime('publishedAt', [ + { start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1 2024' }, + { start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2 2024' } +]) + +// Real-time streaming clustering +for await (const batch of brain.neural.clusterStream({ batchSize: 50 })) { + console.log(`Processed ${batch.progress.percentage}% - Found ${batch.clusters.length} clusters`) +} +``` + +## 📊 **Parameter Consistency Analysis** + +### **✅ Excellent Consistency** + +#### **1. Data-First Pattern** +```typescript +// Always: (data, config/metadata, optional_params) +brain.add(content, metadata) // nounType now in metadata +brain.relate(source, target, VerbType.RelatedTo, metadata?) +brain.search(query, options?) +brain.similar(a, b, options?) +``` + +#### **2. Options Objects** +```typescript +// Consistent options pattern across all methods +{ + limit?: number + threshold?: number + filters?: Record + algorithm?: string + includeMetadata?: boolean +} +``` + +#### **3. Array Methods** +```typescript +// Pluralized versions always take arrays +brain.addNouns([{ vectorOrData: '...', nounType: NounType.Content }]) +brain.addVerbs([{ source: '...', target: '...', type: VerbType.RelatedTo }]) +brain.deleteNouns(['id1', 'id2']) +brain.deleteVerbs(['id1', 'id2']) +``` + +### **Return Type Consistency** + +#### **1. Simple Returns (Shortcuts)** +```typescript +brain.similar(a, b) → Promise // Always simple number +brain.clusters() → Promise // Always simple array +brain.related(id) → Promise // Always simple array +``` + +#### **2. Rich Returns (Neural Namespace)** +```typescript +brain.neural.similar(a, b, { detailed: true }) → Promise +brain.neural.neighbors(id, options) → Promise +brain.neural.clusters(options) → Promise +``` + +#### **3. Consistent Error Handling** +```typescript +// All methods throw descriptive errors with context +try { + await brain.neural.similar('invalid', 'data') +} catch (error) { + // error.code: 'SIMILARITY_ERROR' + // error.context: { inputA: '...', inputB: '...' } +} +``` + +## 🎯 **Key Strengths of Current API** + +### **✅ 1. Progressive Disclosure** +- **Simple**: `brain.similar()` → just returns a number +- **Advanced**: `brain.neural.similar()` → full options & detailed results + +### **✅ 2. Intuitive Namespacing** +- **Core data**: Direct on `brain` (addNoun, search, delete) +- **AI features**: `brain.neural.*` (clustering, similarity, analysis) +- **System**: `brain.storage.*`, `brain.augmentations.*` + +### **✅ 3. Consistent Patterns** +- **Always** `(data, options?)` parameter order +- **Always** async/Promise-based +- **Always** descriptive error messages with context + +### **✅ 4. Type Safety** +```typescript +// Excellent TypeScript support +import { Brainy, NounType, VerbType } from '@soulcraft/brainy' + +const brain = new Brainy() +await brain.add('content', { nounType: NounType.Document, title: 'My Doc' }) +// ^^^^^^^^^^^^^^^^ // IDE autocomplete! +``` + +### **✅ 5. Flexible Configuration** +```typescript +// Zero-config (just works) +const brain = new Brainy() + +// Full control when needed +const brain = new Brainy({ + storage: { + adapter: 'file', + path: './my-data', + encryption: true + }, + neural: { + cacheSize: 10000, + defaultAlgorithm: 'hierarchical' + }, + logging: { verbose: true } +}) +``` + +## 🔍 **Minor Improvement Opportunities** + +### **1. Documentation Consistency** +```typescript +// ✅ GREAT: Clear, descriptive JSDoc +/** + * Add semantic relationship between two items + * @param source - Source item ID + * @param target - Target item ID + * @param type - Relationship type (VerbType enum) + * @param metadata - Optional relationship metadata + */ +brain.relate(source, target, type, metadata?) +``` + +### **2. Error Context Enhancement** +```typescript +// Current: Good error messages +// Improvement: Add suggested fixes +throw new SimilarityError('Failed to calculate similarity', { + inputA: 'invalid-id', + inputB: 'valid-id', + suggestion: 'Check that both IDs exist in the database' +}) +``` + +## 🎖️ **Overall API Grade: A+ (Excellent)** + +### **Strengths:** +- **🎯 Intuitive**: Natural method names, clear hierarchy +- **🔄 Consistent**: Same patterns everywhere +- **📈 Progressive**: Simple → advanced as needed +- **🛡️ Type-safe**: Full TypeScript support +- **📚 Well-documented**: Clear examples & guides +- **🚀 Performant**: Smart caching, batching, streaming + +### **Neural API Fits Perfectly:** +- **✅ Namespace consistency**: `brain.neural.*` is clear and logical +- **✅ Parameter consistency**: Follows same `(data, options?)` pattern +- **✅ Return consistency**: Rich objects when needed, simple types for shortcuts +- **✅ Progressive disclosure**: `brain.similar()` → `brain.neural.similar()` +- **✅ Advanced features**: Domain/temporal clustering, streaming, analysis + +### **Developer Experience Score: 🌟🌟🌟🌟🌟 (5/5 stars)** + +The API surface is **exceptionally well designed** with: +- **Beginner-friendly** shortcuts that "just work" +- **Advanced features** available when needed +- **Consistent patterns** across all methods +- **Logical namespacing** that guides developers naturally +- **Rich ecosystem** with augmentations, Triple Intelligence, and neural features + +**The neural namespace integrates seamlessly and enhances rather than complicates the overall API experience.** \ No newline at end of file diff --git a/docs/api/README.md b/docs/api/README.md index 4ca84364..e91b5788 100644 --- a/docs/api/README.md +++ b/docs/api/README.md @@ -1,2108 +1,395 @@ ---- -title: API Reference -slug: api/reference -public: true -category: api -template: api -order: 1 -description: Complete API reference for all Brainy methods — add, find, relate, update, delete, batch operations, the Db API (transactions, snapshots, time travel), VFS, neural API, and more. -next: - - getting-started/quick-start - - guides/find-system ---- +# 🧠 Brainy 2.0 API Reference -# 🧠 Brainy API Reference - -> **Complete API documentation for Brainy** -> Zero Configuration • Triple Intelligence • Database as a Value • Atomic Transactions • Time Travel - -**Updated:** 2026-06-11 -**All APIs verified against actual code** - ---- +> **The definitive API documentation for Brainy 2.0** +> Clean • Powerful • Zero-Configuration ## Quick Start ```typescript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' +import { Brainy } from '@soulcraft/brainy' -const brain = new Brainy() // Zero config! -await brain.init() // VFS auto-initialized! +const brain = new Brainy() // Zero config! +await brain.init() // Add data (text auto-embeds!) -const id = await brain.add({ - data: 'The future of AI is here', - type: NounType.Concept, - metadata: { category: 'technology' } -}) +await brain.add('The future of AI is here', { nounType: 'content' }) // Search with Triple Intelligence const results = await brain.find({ - query: 'artificial intelligence', - where: { year: { greaterThan: 2020 } }, - connected: { from: id, depth: 2 } + like: 'artificial intelligence', + where: { year: { greaterThan: 2020 } }, + connected: { via: 'references' } }) - -// Pin the current state as an immutable value -const db = brain.now() - -// Commit an atomic multi-write batch (all-or-nothing) -await brain.transact([ - { op: 'update', id, metadata: { category: 'AI' } } -], { meta: { author: 'docs-example' } }) - -await db.get(id) // still sees the pre-transaction state — snapshot isolation -await db.release() - -// Time travel: query any past state -const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000)) ``` ---- - ## Core Concepts -### 🧬 Entities (Nouns) -Semantic vectors with metadata and relationships - the fundamental data unit in Brainy. +### 🧬 Nouns +Vectors with metadata - the fundamental data unit in Brainy. -### 🔗 Relationships (Verbs) -Typed connections between entities with optional `data` and `metadata` - building knowledge graphs. - -### 📊 Data vs Metadata -- **`data`**: Content embedded into vectors. Searchable via **semantic similarity** (vector index) and **hybrid text+semantic** search. NOT queryable via `where` filters. -- **`metadata`**: Structured fields indexed by MetadataIndex. Queryable via `where` filters in `find()`. - -See **[Data Model](../DATA_MODEL.md)** for the full explanation. +### 🔗 Verbs +Relationships between nouns - the connections that create knowledge graphs. ### 🧠 Triple Intelligence Vector search + Graph traversal + Metadata filtering in one unified query. -### 🧊 Database Values (Db) -The whole store pinned as an immutable value — snapshot isolation, atomic `transact()` batches, time travel with `asOf()`, instant hard-link snapshots with `persist()`. See the [consistency model](../concepts/consistency-model.md). +--- + +## API Reference + +### Data Operations + +#### Nouns (Vectors with Metadata) + +##### `addNoun(dataOrVector, metadata?)` +Add a single noun to the database. +- **dataOrVector**: `string | number[]` - Text (auto-embeds) or pre-computed vector +- **metadata**: `object` - Associated metadata +- **Returns**: `Promise` - The noun's ID + +##### `getNoun(id)` +Retrieve a noun by ID. +- **id**: `string` - The noun's ID +- **Returns**: `Promise` + +##### `updateNoun(id, dataOrVector?, metadata?)` +Update an existing noun. +- **id**: `string` - The noun's ID +- **dataOrVector**: `string | number[]` - New data/vector (optional) +- **metadata**: `object` - New metadata (optional) +- **Returns**: `Promise` + +##### `deleteNoun(id)` +Delete a noun. +- **id**: `string` - The noun's ID +- **Returns**: `Promise` + +##### `getNouns(options)` +Get multiple nouns (unified method). +- **options**: Can be: + - `string[]` - Array of IDs + - `{where: object}` - Metadata filter + - `{limit: number, offset: number}` - Pagination +- **Returns**: `Promise` + +#### Verbs (Relationships) + +##### `addVerb(source, target, type, metadata?)` +Create a relationship between nouns. +- **source**: `string` - Source noun ID +- **target**: `string` - Target noun ID +- **type**: `string` - Relationship type +- **metadata**: `object` - Relationship metadata (optional) +- **Returns**: `Promise` - The verb's ID + +##### `getVerbsBySource(sourceId)` +Get all outgoing relationships. +- **sourceId**: `string` - Source noun ID +- **Returns**: `Promise` + +##### `getVerbsByTarget(targetId)` +Get all incoming relationships. +- **targetId**: `string` - Target noun ID +- **Returns**: `Promise` --- -## Table of Contents +### Search Operations -- [Core CRUD Operations](#core-crud-operations) -- [Search & Query](#search--query) -- [Aggregation Engine](#aggregation-engine) -- [Relationships](#relationships) -- [Batch Operations](#batch-operations) -- [Database Values & Time Travel (Db API)](#database-values--time-travel-db-api) -- [Virtual Filesystem (VFS)](#virtual-filesystem-vfs) -- [Neural API](#neural-api) -- [Import & Export](#import--export) -- [Configuration](#configuration) -- [Storage Adapters](#storage-adapters) -- [Utility Methods](#utility-methods) -- [Embedding & Analysis APIs](#embedding--analysis-apis) -- [Type System Reference](#type-system-reference) +#### `search(query, k?)` +Simple vector similarity search. +- **query**: `string | number[]` - Text or vector +- **k**: `number` - Number of results (default: 10) +- **Returns**: `Promise` ---- +> 💡 This is equivalent to: `find({like: query, limit: k})` -## Core CRUD Operations - -### `add(params)` → `Promise` - -Add a single entity to the database. +#### `find(query)` - Triple Intelligence 🧠 +The ultimate search method combining vector, graph, and metadata search. ```typescript -const id = await brain.add({ - data: 'JavaScript is a programming language', // Text or pre-computed vector - type: NounType.Concept, // Required: Entity type - subtype: 'language', // Optional: sub-classification - metadata: { // Optional: queryable fields - category: 'programming', - year: 1995 - } -}) -``` - -**Parameters:** -- `data`: `string | number[]` - Content to embed (text auto-embeds) or pre-computed vector -- `type`: `NounType` - Entity type (required) -- `subtype?`: `string` - Per-product sub-classification within the NounType (top-level standard field, indexed on the fast path). See [Subtypes & Facets](../guides/subtypes-and-facets.md). -- `metadata?`: `object` - Structured queryable fields (indexed by MetadataIndex, used in `where` filters) -- `id?`: `string` - Custom ID (auto-generated UUID if not provided) -- `vector?`: `number[]` - Pre-computed vector (skips auto-embedding) -- `confidence?`: `number` - Type classification confidence (0-1) -- `weight?`: `number` - Entity importance/salience (0-1) -- `ifAbsent?`: `boolean` - By-ID idempotent insert. When `true` AND a custom `id` is supplied AND an entity with that `id` already exists, returns the existing `id` without writing (no throw, no overwrite). Ignored without `id`. See [guides/optimistic-concurrency](../guides/optimistic-concurrency.md). - -> **`data`** is embedded into vectors for semantic search. **`metadata`** is indexed for `where` filters. See [Data Model](../DATA_MODEL.md). - -> **Strict-mode tip:** if a vocabulary is registered for your `type` (via `brain.requireSubtype()` or by an SDK that wraps Brainy), you must pass a matching `subtype`. Run `await brain.audit()` to inventory pre-existing gaps before enabling strict mode; see the [migration recipe](../guides/subtypes-and-facets.md#strict-mode-in-practice-for-sdk-style-vocabulary-consumers). - -**Returns:** `Promise` - Entity ID - ---- - -### `get(id)` → `Promise` - -Retrieve a single entity by ID. - -```typescript -const entity = await brain.get(id) -console.log(entity?.data) // Original data -console.log(entity?.metadata) // Metadata -console.log(entity?.vector) // Embedding vector -``` - -**Parameters:** -- `id`: `string` - Entity ID - -**Returns:** `Promise` - Entity or null if not found - ---- - -### `update(params)` → `Promise` - -Update an existing entity. - -```typescript -await brain.update({ - id: entityId, - data: 'Updated content', // Optional: new data - subtype: 'archived', // Optional: change sub-classification - metadata: { updated: true } // Optional: new metadata (merges) -}) -``` - -**Parameters:** -- `id`: `string` - Entity ID -- `data?`: `string | number[]` - New data/vector -- `type?`: `NounType` - Change entity type -- `subtype?`: `string` - Change subtype (omit to preserve existing) -- `metadata?`: `object` - Metadata to merge (or replace with `merge: false`) -- `confidence?`: `number` - Update classification confidence -- `weight?`: `number` - Update entity importance -- `ifRev?`: `number` - Optimistic-concurrency check. When provided, the update throws `RevisionConflictError` if the persisted entity's `_rev` no longer equals `ifRev`. See [guides/optimistic-concurrency](../guides/optimistic-concurrency.md). - -**Returns:** `Promise` - -> **Tip — read-then-CAS.** Every entity returned by `get()` / `find()` / `search()` carries `entity._rev` (a monotonic counter Brainy auto-bumps on every successful `update()`). Pass it back as `ifRev` to make multi-writer coordination safe without an external lock service. Full guide: [guides/optimistic-concurrency](../guides/optimistic-concurrency.md). - ---- - -### `remove(id)` → `Promise` - -Remove a single entity (and every relationship where it is source or target). - -```typescript -await brain.remove(id) -``` - -**Parameters:** -- `id`: `string` - Entity ID - -**Returns:** `Promise` - ---- - -## Search & Query - -### `find(query)` → `Promise` - -**Triple Intelligence** - Vector + Graph + Metadata in ONE query. - -```typescript -// Simple text search -const results = await brain.find('machine learning') - -// Advanced Triple Intelligence query -const results = await brain.find({ - query: 'artificial intelligence', // Vector similarity - where: { // Metadata filtering - year: { greaterThan: 2020 }, - category: { oneOf: ['AI', 'ML'] } - }, - connected: { // Graph traversal - to: conceptId, - depth: 2, - type: VerbType.RelatedTo - }, - limit: 10 -}) -``` - -**Parameters:** -- `query`: `string | FindParams` - - **Simple:** Just text for vector search - - **Advanced:** Object with vector + graph + metadata filters - -**FindParams:** -- `query?`: `string` - Text for semantic + hybrid search (searches `data` via the vector index + text index) -- `type?`: `NounType | NounType[]` - Filter by entity type(s). Alias for `where.noun`. -- `subtype?`: `string | string[]` - Filter by sub-classification (top-level standard field, fast path). Single string for equality, array for set membership. -- `where?`: `object` - Metadata filters. See **[Query Operators](../QUERY_OPERATORS.md)** for all operators. -- `connected?`: `object` - Graph traversal options - - `to?`: `string` - Target entity ID - - `from?`: `string` - Source entity ID - - `via?`: `VerbType | VerbType[]` - Relationship type(s) to traverse - - `type?`: `VerbType | VerbType[]` - Alias for `via` - - `depth?`: `number` - Traversal depth (default: 1) - - `direction?`: `'in' | 'out' | 'both'` - Traversal direction (default: 'both') -- `limit?`: `number` - Max results (default: 10) -- `offset?`: `number` - Skip results -- `orderBy?`: `string` - Field to sort by (e.g., 'createdAt', 'metadata.priority') -- `order?`: `'asc' | 'desc'` - Sort direction (default: 'asc') -- `searchMode?`: `'auto' | 'text' | 'semantic' | 'hybrid'` - Search strategy: - - `'auto'` (default): Zero-config hybrid combining text + semantic search - - `'text'`: Pure keyword/text matching - - `'semantic'`/`'vector'`: Pure vector similarity - - `'hybrid'`: Explicit hybrid mode -- `hybridAlpha?`: `number` - Balance between text (0.0) and semantic (1.0) search. Auto-detected by query length if not specified. -- `excludeVFS?`: `boolean` - Exclude VFS entities from results (default: false) - -> **`limit` tip:** Brainy caps `limit` against an auto-configured maximum (based on container/free memory, ~25 KB per result). Above the cap you get a one-time warning per call site; above 2× the cap it throws. To raise the cap, pass `new Brainy({ maxQueryLimit: N })` or `{ reservedQueryMemory: bytes }`. For queries that need ALL matches, paginate with `{ limit, offset }` — that's the only pattern guaranteed to keep working across Brainy versions. See [Query Limits & Pagination](../guides/find-limits.md). - -**Returns:** `Promise` - Matching entities with scores - ---- - -### Hybrid Search - -Brainy automatically combines text (keyword) and semantic (vector) search for optimal results. No configuration needed. - -```typescript -// Zero-config hybrid search (just works) -const results = await brain.find({ - query: 'David Smith' // Finds both exact text matches AND semantically similar -}) - -// Force text-only search (exact keyword matching) -const textResults = await brain.find({ - query: 'exact keyword', - searchMode: 'text' -}) - -// Force semantic-only search (vector similarity) -const semanticResults = await brain.find({ - query: 'artificial intelligence concepts', - searchMode: 'semantic' -}) - -// Custom hybrid weighting (0 = text only, 1 = semantic only) -const customResults = await brain.find({ - query: 'David Smith', - hybridAlpha: 0.3 // Favor text matching -}) -``` - -**How it works:** -- Short queries (1-2 words) automatically favor text matching -- Long queries (5+ words) automatically favor semantic search -- Results are combined using Reciprocal Rank Fusion (RRF) - ---- - -### Match Visibility - -Search results include detailed match information: - -```typescript -const results = await brain.find({ query: 'david the warrior' }) - -// Each result now includes: -results[0].textMatches // ["david", "warrior"] - exact query words found -results[0].textScore // 0.25 - text match quality (0-1) -results[0].semanticScore // 0.87 - semantic similarity (0-1) -results[0].matchSource // 'both' | 'text' | 'semantic' -``` - -**Use cases:** -- Highlight exact matches in UI (textMatches) -- Explain why a result ranked high (matchSource) -- Debug search behavior (separate scores) - ---- - -### `highlight(params)` → `Promise` ✨ - -Zero-config highlighting for both exact matches AND semantic concepts. -Handles plain text, rich-text JSON (TipTap, Slate, Lexical, Draft.js, Quill), HTML, and Markdown automatically. - -```typescript -// Plain text (works as before) -const highlights = await brain.highlight({ - query: "david the warrior", - text: "David Smith is a brave fighter who battles dragons" -}) -// [ -// { text: "David", score: 1.0, position: [0, 5], matchType: 'text' }, -// { text: "fighter", score: 0.78, position: [25, 32], matchType: 'semantic' }, -// { text: "battles", score: 0.72, position: [37, 44], matchType: 'semantic' } -// ] - -// Rich-text JSON (auto-detected) -const highlights = await brain.highlight({ - query: "david the warrior", - text: JSON.stringify(tiptapDocument) // TipTap, Slate, Lexical, Draft.js, Quill -}) -// Extracts text from nodes, annotates with contentCategory: -// [ -// { text: "David", score: 1.0, matchType: 'text', contentCategory: 'title' }, -// { text: "fighter", score: 0.78, matchType: 'semantic', contentCategory: 'content' } -// ] - -// HTML input (auto-detected) -const highlights = await brain.highlight({ - query: "warrior", - text: "

David the Warrior

A brave fighter.

" -}) - -// Custom extractor for proprietary formats -const highlights = await brain.highlight({ - query: "function", - text: sourceCode, - contentExtractor: (text) => treeSitterParse(text) // Your custom parser -}) -``` - -**Parameters:** -- `query`: `string` - The search query -- `text`: `string` - Text to highlight (plain text, JSON, HTML, or Markdown) -- `granularity?`: `'word' | 'phrase' | 'sentence'` - Highlight unit (default: 'word') -- `threshold?`: `number` - Min similarity for semantic matches (default: 0.5) -- `contentType?`: `ContentType` - Optional hint: `'plaintext' | 'richtext-json' | 'html' | 'markdown'`. Skips auto-detection when provided. -- `contentExtractor?`: `(text: string) => ExtractedSegment[]` - Custom parser. Bypasses built-in detection entirely. - -**Returns:** `Promise` -- `text` - The matched text -- `score` - Match score (1.0 for text matches, varies for semantic) -- `position` - [start, end] indices in extracted text -- `matchType` - `'text'` (exact) or `'semantic'` (concept) -- `contentCategory?` - `'title' | 'annotation' | 'content' | 'value' | 'code' | 'structural'` — Role of the source text. Built-in extractors produce `'title'`, `'content'`, `'code'`. All 6 categories are available for custom parsers. - -**Supported Rich-Text Formats:** - -| Format | Detection | Text nodes | -|--------|-----------|------------| -| TipTap / ProseMirror | `{ type: 'doc', content: [...] }` | `{ type: 'text', text }` | -| Slate.js | `[{ type, children }]` | `{ text }` | -| Lexical | `{ root: { children } }` | `{ type: 'text', text }` | -| Draft.js | `{ blocks: [{ text }] }` | `{ text }` in block | -| Quill Delta | `{ ops: [{ insert }] }` | `{ insert }` | -| HTML | Tags like `

`, `

`, `` | Visible text content | -| Markdown | `#` headings, ` ``` ` code blocks | Stripped markup | - -**Timeout Protection:** -Semantic matching has a 10-second timeout. If embedding takes too long (e.g., WASM stall), `highlight()` returns text-only matches instead of hanging. - -**UI Pattern:** -```typescript -// Style differently based on match type and content category -highlights.forEach(h => { - const style = h.matchType === 'text' ? 'font-weight: bold' : 'background: yellow' - if (h.contentCategory === 'title') { /* render as heading highlight */ } - if (h.contentCategory === 'code') { /* render with code styling */ } - if (h.contentCategory === 'annotation') { /* render as comment/caption */ } - // Apply style from h.position[0] to h.position[1] +find({ + // Vector similarity + like: 'text query' | vector | {id: 'noun-id'}, + + // Metadata filtering (Brainy operators) + where: { + field: value, // Exact match + field: { + equals: value, + greaterThan: value, + lessThan: value, + greaterEqual: value, + lessEqual: value, + oneOf: [val1, val2], // In array + notOneOf: [val1, val2], // Not in array + contains: value, // Array/string contains + startsWith: value, + endsWith: value, + matches: /pattern/, // Pattern match + between: [min, max] + } + }, + + // Graph traversal + connected: { + to: 'noun-id', // Target noun + from: 'noun-id', // Source noun + via: 'relationship-type', // Relationship type + depth: 2 // Traversal depth + }, + + // Control + limit: 10, // Max results + offset: 0, // Skip results + explain: false // Include explanation }) ``` --- -### Query Operators +### Neural API -Brainy uses clean, readable operators (BFO — Brainy Field Operators): +Access advanced AI features via `brain.neural`: -| Operator | Description | Example | -|----------|-------------|---------| -| `equals` / `eq` | Exact match | `{age: {equals: 25}}` | -| `notEquals` / `ne` | Not equal | `{status: {notEquals: 'deleted'}}` | -| `greaterThan` / `gt` | Greater than | `{age: {greaterThan: 18}}` | -| `gte` / `greaterThanOrEqual` | Greater or equal | `{score: {gte: 90}}` | -| `lessThan` / `lt` | Less than | `{price: {lessThan: 100}}` | -| `lte` / `lessThanOrEqual` | Less or equal | `{rating: {lte: 3}}` | -| `between` | Inclusive range | `{year: {between: [2020, 2025]}}` | -| `oneOf` / `in` | In array | `{color: {oneOf: ['red', 'blue']}}` | -| `noneOf` | Not in array | `{status: {noneOf: ['deleted']}}` | -| `contains` | Array contains value | `{tags: {contains: 'ai'}}` | -| `exists` / `missing` | Field existence | `{email: {exists: true}}` | +#### `brain.neural.similar(a, b)` +Calculate semantic similarity between two items. +- **Returns**: `Promise` - Similarity score (0-1) + +#### `brain.neural.clusters(options?)` +Automatically cluster nouns. +- **Returns**: `Promise` - Generated clusters + +#### `brain.neural.hierarchy(id)` +Build semantic hierarchy from a noun. +- **Returns**: `Promise` - Hierarchy structure + +#### `brain.neural.neighbors(id, k?)` +Find k-nearest neighbors. +- **Returns**: `Promise` - Nearest neighbors + +#### `brain.neural.outliers(threshold?)` +Detect outlier nouns. +- **Returns**: `Promise` - Outlier IDs + +#### `brain.neural.visualize(options?)` +Generate visualization data for external tools. +```typescript +visualize({ + maxNodes: 100, + dimensions: 2 | 3, + algorithm: 'force' | 'hierarchical' | 'radial', + includeEdges: true +}) +// Returns format for D3, Cytoscape, or GraphML +``` + +--- + +### Import & Export + +#### `neuralImport(data, options?)` +AI-powered smart import that auto-detects format. +- **data**: `any` - Data to import +- **options**: Import configuration + - `confidenceThreshold`: Minimum confidence (0-1) + - `autoApply`: Automatically add to database + - `skipDuplicates`: Skip existing entities +- **Returns**: Detected entities and relationships + +#### `backup()` +Create a full backup. +- **Returns**: `Promise` + +#### `restore(backup)` +Restore from backup. +- **backup**: `BackupData` - Previous backup +- **Returns**: `Promise` + +--- + +### Intelligence Features + +#### Verb Scoring +Train the relationship scoring model: +- `provideFeedbackForVerbScoring(feedback)` - Train model +- `getVerbScoringStats()` - Get statistics +- `exportVerbScoringLearningData()` - Export training +- `importVerbScoringLearningData(data)` - Import training + +#### Embeddings +- `embed(text)` - Generate embedding vector +- `calculateSimilarity(a, b, metric?)` - Calculate similarity + +--- + +### Configuration & Management + +#### Operational Modes +- `setReadOnly(bool)` - Toggle read-only mode +- `setWriteOnly(bool)` - Toggle write-only mode +- `setFrozen(bool)` - Freeze all modifications + +#### Cache & Performance +- `getCacheStats()` - Get cache statistics +- `clearCache()` - Clear search cache +- `size()` - Get total noun count +- `getStatistics()` - Get full statistics + +#### Data Management +- `clear(options?)` - Clear all data +- `clearNouns()` - Clear nouns only +- `clearVerbs()` - Clear verbs only +- `rebuildMetadataIndex()` - Rebuild index + +--- + +### Lifecycle + +#### Initialization +```typescript +const brain = new Brainy({ + storage: 'auto', // auto | memory | filesystem | s3 + dimensions: 384, // Vector dimensions + cache: true, // Enable caching + index: true // Enable indexing +}) + +await brain.init() // Required before use! +``` + +#### Cleanup +```typescript +await brain.shutdown() // Graceful shutdown +``` + +#### Static Methods +- `Brainy.preloadModel()` - Preload ML model +- `Brainy.warmup()` - Warmup system + +--- + +## Query Operators Reference + +Brainy uses its own clean, readable operators: + +| Brainy Operator | Description | Example | +|-----------------|-------------|---------| +| `equals` | Exact match | `{age: {equals: 25}}` | +| `greaterThan` | Greater than | `{age: {greaterThan: 18}}` | +| `lessThan` | Less than | `{price: {lessThan: 100}}` | +| `greaterEqual` | Greater or equal | `{score: {greaterEqual: 90}}` | +| `lessEqual` | Less or equal | `{rating: {lessEqual: 3}}` | +| `oneOf` | In array | `{color: {oneOf: ['red', 'blue']}}` | +| `notOneOf` | Not in array | `{status: {notOneOf: ['deleted']}}` | +| `contains` | Contains value | `{tags: {contains: 'ai'}}` | | `startsWith` | String prefix | `{name: {startsWith: 'John'}}` | | `endsWith` | String suffix | `{email: {endsWith: '@gmail.com'}}` | | `matches` | Pattern match | `{text: {matches: /^[A-Z]/}}` | -| `allOf` | AND combinator | `{allOf: [{active: true}, {role: 'admin'}]}` | -| `anyOf` | OR combinator | `{anyOf: [{role: 'admin'}, {role: 'owner'}]}` | - -**[Complete Operator Reference →](../QUERY_OPERATORS.md)** — all operators, aliases, indexed vs in-memory support matrix, and practical examples. - ---- - -## Aggregation Engine - -Brainy's aggregation engine maintains **incremental running totals** at write time, delivering O(1) aggregate reads regardless of dataset size. Define aggregates once, and every `add()`, `update()`, and `delete()` automatically updates the running metrics. - -### `defineAggregate(definition)` → `void` - -Register a named aggregate for incremental computation. - -```typescript -brain.defineAggregate({ - name: 'monthly_spending', - source: { - type: NounType.Event, - where: { domain: 'financial' } // matches custom metadata fields - }, - groupBy: [ - 'category', - { field: 'date', window: 'month' } // Time-windowed dimension - ], - metrics: { - total: { op: 'sum', field: 'amount' }, - count: { op: 'count' }, - average: { op: 'avg', field: 'amount' }, - highest: { op: 'max', field: 'amount' }, - lowest: { op: 'min', field: 'amount' }, - spread: { op: 'stddev', field: 'amount' } // Welford's online algorithm - }, - materialize: true // Optional: write results as NounType.Measurement entities -}) -``` - -**Parameters:** - -| Field | Type | Description | -|-------|------|-------------| -| `name` | `string` | Unique identifier for this aggregate | -| `source.type` | `NounType \| NounType[]` | Entity types that feed into this aggregate | -| `source.where` | `Record` | Filter on custom **metadata** fields (matched against the entity's `metadata` bag) | -| `source.service` | `string` | Multi-tenancy filter | -| `groupBy` | `GroupByDimension[]` | Dimensions to group by — plain field names, `{ field, window }` for time bucketing, or `{ field, unnest: true }` for array fields (one contribution per element) | -| `metrics` | `Record` | Named metrics with `op` (`sum`, `count`, `avg`, `min`, `max`, `stddev`, `variance`, `percentile`, `distinctCount`) and optional `field`. `percentile` additionally requires `p` in `[0, 1]`. | -| `materialize` | `boolean \| object` | Write results as `NounType.Measurement` entities (auto-visible in OData/Sheets/SSE) | - -**Time window granularities:** `'hour'`, `'day'`, `'week'`, `'month'`, `'quarter'`, `'year'`, or `{ seconds: number }` for custom intervals. - -### `removeAggregate(name)` → `void` - -Remove a named aggregate and clean up its state. - -```typescript -brain.removeAggregate('monthly_spending') -``` - -### Querying Aggregates via `find()` - -Aggregate results are queried through the standard `find()` method using the `aggregate` parameter: - -```typescript -// Simple: query by name -const results = await brain.find({ aggregate: 'monthly_spending' }) - -// With filtering on group keys -const foodOnly = await brain.find({ - aggregate: 'monthly_spending', - where: { category: 'food' } -}) - -// With sorting and pagination -const topCategories = await brain.find({ - aggregate: { - name: 'monthly_spending', - orderBy: 'total', - order: 'desc', - limit: 10 - } -}) - -// Combine find-level params (where, orderBy, limit, offset merge automatically) -const recentFood = await brain.find({ - aggregate: 'monthly_spending', - where: { category: 'food' }, - orderBy: 'total', - order: 'desc', - limit: 12 -}) -``` - -**Result format:** Returns `Result[]` with `type: NounType.Measurement`. Each result contains: - -```typescript -{ - id: string, // Aggregate group ID (or materialized entity ID) - score: 1.0, // Always 1.0 for aggregates - type: NounType.Measurement, - metadata: { - __aggregate: 'monthly_spending', // Source aggregate name - category: 'food', // Group key values - date: '2024-01', // Time window bucket - total: 342.50, // Computed metrics - count: 28, - average: 12.23, - highest: 45.00, - lowest: 2.50 - }, - entity: Entity // Full entity structure -} -``` - -### `queryAggregate(name, params?)` → `Promise` - -The first-class analytics path — returns plain group rows (`{ groupKey, metrics, count }`) instead of `find()`-style `Result` wrappers. Supports `where` (group-key filter), `having` (SQL-HAVING metric filter), `orderBy`, `order`, `limit`, `offset`: - -```typescript -const rows = await brain.queryAggregate('monthly_spending', { - having: { total: { greaterThan: 100 } }, // filter by computed metrics - orderBy: 'total', - order: 'desc', - limit: 10 -}) -// [{ groupKey: { category: 'food', date: '2024-01' }, metrics: { total: 342.5, count: 28 }, count: 28 }, ...] -``` - -### How It Works - -Aggregation hooks run **outside transactions** on every write operation: - -- **`add()`**: If the new entity matches any aggregate's `source` filter, its values are added to the matching group's running totals. -- **`update()`**: The old entity's contribution is reversed and the new entity's contribution is applied (handles group key changes, source filter changes). -- **`delete()`**: The deleted entity's contribution is reversed from its group. - -**Performance:** O(A × G × M) per write where A = matching aggregates, G = groupBy dimensions, M = metrics. For typical configurations (2-5 aggregates, 1-3 dimensions, 3-5 metrics), this is effectively O(1) — measured at **10,000 entities in 13ms** in unit tests. - -**Infinite loop prevention:** Materialized `NounType.Measurement` entities (with `service: 'brainy:aggregation'` or `metadata.__aggregate`) are automatically excluded from all aggregate source matching. - -**Persistence:** Definitions and running state are persisted to storage on `flush()`/`close()` and reloaded on `init()`. Definition changes are detected via FNV-1a hashing — only changed aggregates reset their state. - -**Native acceleration:** Register an `'aggregation'` provider via the plugin system to replace the TypeScript engine with a custom native implementation for higher throughput at scale. - -### Financial Data Modeling - -Brainy supports financial analytics through **subtypes and metadata conventions** on existing NounTypes — no custom types needed: - -```typescript -// Transaction = NounType.Event + 'transaction' subtype + financial metadata -await brain.add({ - data: 'Coffee at Blue Bottle', - type: NounType.Event, - subtype: 'transaction', // top-level standard field (reserved — never in metadata) - metadata: { - domain: 'financial', - amount: 5.50, - currency: 'USD', - category: 'food', - date: Date.now(), - merchant: 'Blue Bottle Coffee' - } -}) - -// Account = NounType.Collection + 'account' subtype + financial metadata -await brain.add({ - data: 'Checking Account', - type: NounType.Collection, - subtype: 'account', - metadata: { - domain: 'financial', - accountType: 'checking', - currency: 'USD', - institution: 'Chase' - } -}) - -// Invoice = NounType.Document + 'invoice' subtype + financial metadata -await brain.add({ - data: 'Invoice #1234 from Acme Corp', - type: NounType.Document, - subtype: 'invoice', - metadata: { - domain: 'financial', - amount: 15000, - currency: 'USD', - status: 'pending', - dueDate: Date.UTC(2024, 2, 15), - vendor: 'Acme Corp' - } -}) -``` - ---- - -## Relationships - -### `relate(params)` → `Promise` - -Create a typed relationship between entities. - -```typescript -const relId = await brain.relate({ - from: sourceId, - to: targetId, - type: VerbType.ReportsTo, - subtype: 'direct', // Optional: sub-classification - data: 'Collaborated on the research paper', // Optional: content for this edge - metadata: { // Optional: structured edge fields - strength: 0.9, - role: 'primary author' - } -}) -``` - -**Parameters:** -- `from`: `string` - Source entity ID (must exist) -- `to`: `string` - Target entity ID (must exist) -- `type`: `VerbType` - Relationship type -- `subtype?`: `string` - Per-product sub-classification within the VerbType (top-level standard field, fast-path indexed). See [Subtypes & Facets](../guides/subtypes-and-facets.md). -- `data?`: `any` - Content for the relationship (overrides auto-computed vector) -- `metadata?`: `object` - Structured edge fields -- `weight?`: `number` - Connection strength (0-1, default: 1.0) -- `bidirectional?`: `boolean` - Create reverse edge too (default: false) -- `confidence?`: `number` - Relationship certainty (0-1) - -> **Strict-mode tip:** same as `add()` — if a vocabulary is registered for your `type`, pass a matching `subtype`. Run `await brain.audit()` first to surface pre-existing gaps. - -**Returns:** `Promise` - Relationship ID - ---- - -### `updateRelation(params)` → `Promise` - -Update an existing relationship. Mirror of `update()` for verbs — closed a long-standing gap (verbs had no update path before 7.30). - -```typescript -// Change the subtype on an existing relationship -await brain.updateRelation({ id: relId, subtype: 'dotted-line' }) - -// Update weight + confidence -await brain.updateRelation({ id: relId, weight: 0.7, confidence: 0.9 }) - -// Change verb type (re-indexes in graph adjacency, id preserved) -await brain.updateRelation({ id: relId, type: VerbType.WorksWith }) -``` - -**Parameters:** -- `id`: `string` - Relationship ID (required) -- `type?`: `VerbType` - Change verb type (re-indexes in graph adjacency) -- `subtype?`: `string` - Change sub-classification (omit to preserve existing) -- `weight?`: `number` - New weight (0-1) -- `confidence?`: `number` - New confidence (0-1) -- `data?`: `any` - New content -- `metadata?`: `object` - Metadata to merge (or replace with `merge: false`) -- `merge?`: `boolean` - Merge or replace metadata (default: true) - -**Returns:** `Promise` - ---- - -### `related(params)` → `Promise` - -Get relationships for an entity. Same name and surface as `db.related()` on a -pinned `Db` view. - -```typescript -// Get all relationships FROM an entity -const outgoing = await brain.related({ from: entityId }) - -// Get all relationships TO an entity -const incoming = await brain.related({ to: entityId }) - -// Filter by type -const contains = await brain.related({ - from: entityId, - type: VerbType.Contains -}) - -// Filter by subtype (fast path, column-store hit) -const direct = await brain.related({ - from: entityId, - type: VerbType.ReportsTo, - subtype: 'direct' -}) - -// Set membership on subtype -const all = await brain.related({ - from: entityId, - type: VerbType.ReportsTo, - subtype: ['direct', 'dotted-line'] -}) -``` - -**Parameters:** -- `from?`: `string` - Source entity ID -- `to?`: `string` - Target entity ID -- `type?`: `VerbType | VerbType[]` - Filter by relationship type -- `subtype?`: `string | string[]` - Filter by VerbType subtype (top-level standard field, fast path) -- `service?`: `string` - Multi-tenancy filter -- `limit?`: `number` - Pagination limit (default: 100) -- `offset?`: `number` - Pagination offset - -**Returns:** `Promise` - Matching relationships (each with `subtype` at top level when set) - ---- - -## Batch Operations - -### `addMany(params)` → `Promise>` - -Add multiple entities in one operation. - -```typescript -const result = await brain.addMany({ - items: [ - { data: 'Entity 1', type: NounType.Document }, - { data: 'Entity 2', type: NounType.Concept } - ] -}) - -console.log(result.successful) // Array of IDs -console.log(result.failed) // Array of errors -``` - -**Returns:** `Promise>` - Success/failure results - ---- - -### `removeMany(params)` → `Promise>` - -Remove multiple entities. - -```typescript -const result = await brain.removeMany({ - ids: [id1, id2, id3] -}) -``` - ---- - -### `updateMany(params)` → `Promise>` - -Update multiple entities. - -```typescript -const result = await brain.updateMany({ - items: [ - { id: id1, metadata: { updated: true } }, - { id: id2, data: 'New content' } - ] -}) -``` - ---- - -### `relateMany(params)` → `Promise` - -Create multiple relationships. - -```typescript -const ids = await brain.relateMany({ - items: [ - { from: id1, to: id2, type: VerbType.RelatedTo }, - { from: id1, to: id3, type: VerbType.Contains } - ] -}) -``` - ---- - -## Database Values & Time Travel (Db API) - -Brainy 8.0's generational MVCC exposes the whole store as an immutable -value: the **`Db`**. Pin the current state in O(1), commit atomic -multi-write batches, query any past generation with the full query surface, -cut instant snapshots, and ask what-if questions in memory. The exact -guarantees live in the **[consistency model](../concepts/consistency-model.md)**; -recipes live in **[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md)**. - -### `generation()` → `number` - -The store's current generation — a monotonic watermark advanced once per -committed `transact()` batch and once per single-operation write. Never -reissued, including across restarts and `restore()`. - -```typescript -const g = brain.generation() -``` - ---- - -### `now()` → `Db` - -Pin the current generation and return an immutable view — O(1), no I/O. -The view keeps reading exactly this state no matter what commits afterwards. - -```typescript -const db = brain.now() -await brain.update({ id, metadata: { v: 2 } }) - -await db.get(id) // still sees v: 1 — pinned -await brain.get(id) // sees v: 2 — live -await db.release() // unpin (enables history compaction) -``` - -**Returns:** `Db` — release it when done; pins gate `compactHistory()`. - ---- - -### `transact(ops, options?)` → `Promise` - -Execute a declarative operation batch **atomically**: either every -operation applies and the store advances exactly one generation, or none -apply and the store is byte-identical to its pre-transaction state. The -commit point is an atomic manifest rename; a crash anywhere before it rolls -back to the exact pre-transaction bytes on the next open. - -```typescript -const db = await brain.transact([ - { op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' }, - { op: 'update', id: customerId, metadata: { lastOrderAt: Date.now() }, ifRev: customer._rev }, - { op: 'relate', from: customerId, to: orderId, type: VerbType.Creates, subtype: 'purchase' }, - { op: 'remove', id: staleDraftId }, - { op: 'unrelate', id: oldRelationId } -], { - meta: { author: 'order-service', requestId: 'req-9f2' }, // reified, durable - ifAtGeneration: expectedGeneration // whole-store CAS -}) - -db.receipt.ids // resolved id per operation, in input order -db.receipt.generation // the committed generation -``` - -**Operations** (`op` discriminates; parameters mirror the single-operation methods): -- `{ op: 'add', ... }` — same parameters as `add()`; optional explicit `id` -- `{ op: 'update', ... }` — same parameters as `update()`, including per-entity `ifRev` CAS -- `{ op: 'remove', id }` — deletes the entity plus its relationships (same cascade as `delete()`) -- `{ op: 'relate', ... }` — same parameters as `relate()`, including `bidirectional`; duplicates dedupe to the existing relationship id -- `{ op: 'unrelate', id }` — deletes a relationship by id - -Operations may reference ids created earlier in the same batch. - -**Options:** -- `meta?`: `Record` — transaction metadata, recorded durably in the transaction log (audit fields: author, reason, request id) -- `ifAtGeneration?`: `number` — whole-store compare-and-swap; commits only if the store is still at this generation - -**Returns:** `Promise` — pinned at the freshly committed generation, carrying a `receipt`. - -**Throws:** -- `GenerationConflictError` — `ifAtGeneration` did not match (nothing staged, generation unchanged) -- `RevisionConflictError` — an `ifRev` did not match (whole batch rejected) - ---- - -### `asOf(target)` → `Promise` - -Open an immutable view of **past** state: - -```typescript -const atGen = await brain.asOf(1041) // generation number -const lastWeek = await brain.asOf(new Date(Date.now() - 7 * 86_400_000)) // wall-clock -const fromSnapshot = await brain.asOf('/backups/2026-06-01') // snapshot directory -``` - -- **`number`** — pins that generation; reads resolve through the immutable record layer. -- **`Date`** — resolved via the transaction log to the newest generation committed at or before it. -- **`string`** — a snapshot directory from `db.persist()`, opened as a self-contained read-only store (equivalent to `Brainy.load()`). - -Historical views serve the **full query surface**. Metadata-level reads are -free; the first index-accelerated query (semantic search, traversal, -cursors, aggregation) builds an in-memory index materialization — O(n at -that generation), once per `Db`, freed on `release()`. - -**Throws:** `GenerationCompactedError` when the generation's records were reclaimed by `compactHistory()`. - -**History granularity:** every write is its own immutable generation — -`transact()` batches AND single-operation writes — so a pin always freezes and -every write is addressable via `asOf()`. See the -[consistency model](../concepts/consistency-model.md). - ---- - -### `transactionLog(options?)` → `Promise` - -Read the reified transaction log — one entry per committed generation (every -`transact()` AND single-op write), newest first: `{ generation, timestamp, -meta? }`. Single-op generations carry no `meta` (it is a `transact()`-only field). - -```typescript -const [latest] = await brain.transactionLog({ limit: 1 }) -latest.meta // { author: 'order-service', requestId: 'req-9f2' } -``` - ---- - -### `compactHistory(options?)` → `Promise` - -Reclaim historical record-sets that no retention cap and no live `Db` pin -protects. Pinned reads stay correct across compaction, always. (Auto-compaction -on `flush()`/`close()` is governed by the constructor `retention` knob — unset → -adaptive, `'all'` → unbounded, `{ … }` → explicit caps.) - -```typescript -await brain.compactHistory({ - maxGenerations: 100, // keep at most the 100 most recent generations - maxAge: 7 * 24 * 60 * 60 * 1000 // and only those from the last 7 days -}) -``` - -**Returns:** `{ removedGenerations, horizon }` — `asOf()` below the horizon throws `GenerationCompactedError`. - ---- - -### `restore(path, { confirm: true })` → `Promise` - -Replace the store's **entire** state from a snapshot directory. Destructive -— requires `{ confirm: true }`. All indexes are rebuilt; the generation -counter is floored so observed generation numbers are never reissued; live -pins do not survive. - -```typescript -await brain.restore('/backups/2026-06-01', { confirm: true }) -``` - ---- - -### `Brainy.load(path)` → `Promise` (static) - -Open a persisted snapshot as a self-contained **read-only** store with the -full query surface, including vector search. Releasing the returned `Db` -closes the underlying instance. - -```typescript -const db = await Brainy.load('/backups/2026-06-01') -const hits = await db.find({ query: 'quarterly invoices' }) -await db.release() -``` - ---- - -### The `Db` value - -Every `Db` is pinned at one generation and serves the full query surface at -exactly that state. - -**Properties:** - -| Property | Type | Meaning | -|---|---|---| -| `generation` | `number` | The pinned generation | -| `timestamp` | `number` | Pin time (`now()`), commit time (`transact()`), or resolved commit time (`asOf()`) | -| `receipt` | `TransactReceipt?` | Present only on `transact()` results | -| `speculative` | `boolean` | Whether this view carries a `with()` overlay | -| `released` | `boolean` | Whether `release()` has been called | - -**Methods:** - -```typescript -await db.get(id) // entity as of this generation -await db.find({ where: { status: 'open' } }) // full find() surface -await db.find({ query: 'unpaid invoices' }) // semantic search as of this generation -await db.related(entityId) // relationships as of this generation -await db.since(olderDb) // ids changed between two views -const whatIf = await db.with(ops) // speculative in-memory overlay -await db.persist('/backups/today') // self-contained hard-link snapshot -await db.release() // unpin + free cached materialization -``` - -- **`with(ops)`** — applies `transact()`-style operations **in memory** on - top of the view; nothing touches disk, the generation counter, or index - providers. Overlay entities carry no embeddings, so index-accelerated - queries and `persist()` on overlays throw `SpeculativeOverlayError`; - `get()`, metadata-filter `find()`, and filter-based `related()` work - fully. Commit the same ops with `transact()` for the full surface. -- **`persist(path)`** — cuts an instant snapshot (hard links on filesystem - storage; byte copies across devices; in-memory stores serialize to the - same layout). Requires the view to still be the store's latest generation - — otherwise `GenerationConflictError`. -- **`release()`** — idempotent; after release every read throws. A - `FinalizationRegistry` backstop releases leaked pins at GC, but explicit - release is what makes `compactHistory()` deterministic. - -### Db API errors - -All exported from `@soulcraft/brainy`: - -| Error | Thrown by | Meaning | -|---|---|---| -| `GenerationConflictError` | `transact({ ifAtGeneration })`, `db.persist()` | The store moved past the expected generation — re-read and retry | -| `RevisionConflictError` | `update({ ifRev })`, `transact()` update ops | Per-entity revision moved — see [optimistic concurrency](../guides/optimistic-concurrency.md) | -| `GenerationCompactedError` | `asOf()` | The requested generation's records were reclaimed — persist what you must keep | -| `SpeculativeOverlayError` | index-accelerated reads / `persist()` on `with()` overlays | Honest boundary: overlay entities carry no embeddings | - ---- - -## Virtual Filesystem (VFS) - -Access via `brain.vfs` (property, not method). Auto-initialized during `brain.init()`. - -### Filtering VFS Entities - -All VFS entities (files/folders) have `metadata.isVFSEntity: true` set automatically. - -Use this to filter VFS entities from semantic search results: - -```typescript -// Exclude VFS entities from semantic search -const semanticOnly = await brain.find({ - query: 'artificial intelligence', - where: { - isVFSEntity: { notEquals: true } // Only semantic entities - } -}) - -// Or filter to ONLY VFS entities -const vfsOnly = await brain.find({ - where: { - isVFSEntity: { equals: true } // Only VFS files/folders - } -}) - -// Check if an entity is a VFS entity -if (entity.metadata.isVFSEntity === true) { - console.log('This is a VFS file or folder') -} -``` - -**Why this matters:** Without filtering, VFS files/folders can appear in concept explorers and semantic search results where they don't belong. - ---- - -### Basic File Operations - -#### `vfs.readFile(path, options?)` → `Promise` - -Read file content. - -```typescript -const content = await brain.vfs.readFile('/docs/README.md') -console.log(content.toString()) -``` - ---- - -#### `vfs.writeFile(path, data, options?)` → `Promise` - -Write file content. - -```typescript -await brain.vfs.writeFile('/docs/README.md', 'New content', { - encoding: 'utf-8' -}) -``` - ---- - -#### `vfs.unlink(path)` → `Promise` - -Delete a file. - -```typescript -await brain.vfs.unlink('/docs/old-file.md') -``` - ---- - -### Directory Operations - -#### `vfs.mkdir(path, options?)` → `Promise` - -Create directory. - -```typescript -await brain.vfs.mkdir('/projects/new-app', { recursive: true }) -``` - ---- - -#### `vfs.readdir(path, options?)` → `Promise` - -List directory contents. - -```typescript -const files = await brain.vfs.readdir('/projects') - -// With file types -const entries = await brain.vfs.readdir('/projects', { withFileTypes: true }) -entries.forEach(entry => { - console.log(entry.name, entry.isDirectory() ? 'DIR' : 'FILE') -}) -``` - ---- - -#### `vfs.rmdir(path, options?)` → `Promise` - -Remove directory. - -```typescript -await brain.vfs.rmdir('/old-project', { recursive: true }) -``` - ---- - -#### `vfs.stat(path)` → `Promise` - -Get file/directory stats. - -```typescript -const stats = await brain.vfs.stat('/docs/README.md') -console.log(stats.size) // File size -console.log(stats.mtime) // Modified time -console.log(stats.isDirectory()) // Is directory? -``` - ---- - -### Semantic Operations - -#### `vfs.search(query, options?)` → `Promise` - -Semantic file search. - -```typescript -const results = await brain.vfs.search('React components with hooks', { - path: '/src', - limit: 10 -}) -``` - ---- - -#### `vfs.findSimilar(path, options?)` → `Promise` - -Find similar files. - -```typescript -const similar = await brain.vfs.findSimilar('/src/App.tsx', { - limit: 5, - threshold: 0.7 -}) -``` - ---- - -### Tree Operations - -#### `vfs.getTreeStructure(path, options?)` → `Promise` - -Get directory tree (prevents infinite recursion). - -```typescript -const tree = await brain.vfs.getTreeStructure('/projects', { - maxDepth: 3 -}) -``` - ---- - -#### `vfs.getDescendants(path, options?)` → `Promise` - -Get all descendants with optional filtering. - -```typescript -const files = await brain.vfs.getDescendants('/src', { - filter: (entity) => entity.name.endsWith('.tsx') -}) -``` - ---- - -### Metadata & Relationships - -#### `vfs.getMetadata(path)` → `Promise` - -Get file metadata. - -```typescript -const meta = await brain.vfs.getMetadata('/src/App.tsx') -console.log(meta.todos) // Extracted TODOs -console.log(meta.tags) // Tags -``` - ---- - -#### `vfs.getRelationships(path)` → `Promise` - -Get file relationships. - -```typescript -const rels = await brain.vfs.getRelationships('/src/App.tsx') -// Returns: imports, references, dependencies -``` - ---- - -#### `vfs.getTodos(path)` → `Promise` - -Get TODOs from a file. - -```typescript -const todos = await brain.vfs.getTodos('/src/App.tsx') -``` - ---- - -#### `vfs.searchEntities(query)` → `Promise>` - -Search for semantic entities tracked by the VFS, filtered by type, name, or metadata. - -```typescript -const people = await brain.vfs.searchEntities({ - type: 'person', // entity type filter - name: 'Ada', // semantic name search - where: { role: 'author' }, // metadata filters - limit: 50 -}) -``` - ---- - -**[📖 Complete VFS Documentation →](../vfs/QUICK_START.md)** - ---- - -## Import & Export - -### `import(source, options?)` → `Promise` - -Smart import with auto-detection (CSV, Excel, PDF, JSON, URLs). - -```typescript -// CSV import -await brain.import('data.csv', { - format: 'csv', - createEntities: true -}) - -// Excel import (all sheets processed automatically) -await brain.import('sales.xlsx', { - format: 'excel', - vfsPath: '/imports/sales', // optional: mirror into the VFS - groupBy: 'sheet' -}) - -// PDF import (tables extracted automatically) -await brain.import('research.pdf', { format: 'pdf' }) - -// URL import -await brain.import('https://api.example.com/data.json') -``` - -**Parameters:** -- `source`: `string | Buffer | object` - File path, URL, buffer, or object -- `options?`: Import configuration - - `format?`: `'excel' | 'pdf' | 'csv' | 'json' | 'markdown' | 'yaml' | 'docx' | 'image'` - Auto-detected if omitted - - `vfsPath?`: `string` - Mirror imported content into the VFS at this path - - `groupBy?`: `'type' | 'sheet' | 'flat' | 'custom'` - VFS grouping strategy - - `createEntities?`: `boolean` - Create entities from rows - - `createRelationships?`: `boolean` - Create relationships between extracted entities - - `preserveSource?`: `boolean` - Save the original file in the VFS - - `enableNeuralExtraction?`: `boolean` - Extract entity names via AI - - `enableRelationshipInference?`: `boolean` - Infer relationships via AI - - `enableConceptExtraction?`: `boolean` - Extract entity types via AI - - `confidenceThreshold?`: `number` - Minimum confidence for extracted entities - - `onProgress?`: `(progress) => void` - Progress callback (stage, counts, throughput, ETA) - -**Returns:** `Promise` - Import statistics - -**[📖 Complete Import Guide →](../guides/import-anything.md)** - ---- - -### Export & Import (portable) + Snapshots (native) - -**Portable graph export/import** — `brain.export()` / `brain.import()` (`PortableGraph` v1, versioned -JSON, partial-or-whole, cross-version). `export()` lives on the immutable `Db`, so it composes -with `now()`/`asOf()`/`with()`: - -```typescript -// Export part or all of the brain to a portable, versioned document -const graph = await brain.export({ ids }, { includeVectors: true }) - -// Restore it — import() routes a PortableGraph to the graph round-trip (merge by id) -await otherBrain.import(graph, { onConflict: 'merge' }) - -// Time-travel export (serialize a past generation) / what-if export (a speculative state) -const past = await brain.asOf(gen) -const asWas = await past.export({ collection: id }) -await past.release() -await brain.now().with(ops).export({ ids }) -``` - -Selectors: `{ ids }`, `{ collection }` (alias `memberOf`), `{ connected: { from, depth } }`, -`{ vfsPath }`, predicate (`{ type, subtype, where, service }`), or whole brain (omit). See the -**[Export & Import guide](../guides/export-and-import.md)**. Distinct from `brain.import(file)` -(CSV/PDF/Excel/JSON ingestion — `import()` dispatches on whether you pass a `PortableGraph` or a file). - -**Native whole-brain snapshot** (generation-preserving, not portable JSON): - -```typescript -// Instant hard-link snapshot via the Db API -const pin = brain.now() -await pin.persist('/backups/2026-06-11') -await pin.release() - -// Time-travel to a past generation or timestamp -const snapshot = await brain.asOf(new Date('2026-06-01')) -const entities = await snapshot.find({ limit: 100 }) -await snapshot.release() -``` - ---- - -## Configuration - -### Constructor Options - -```typescript -const brain = new Brainy({ - // Storage configuration - storage: { - type: 'filesystem', // 'memory' | 'filesystem' | 'auto' - path: './brainy-data' - }, - - // Vector index configuration (2 knobs) - vector: { - recall: 'balanced', // 'fast' | 'balanced' | 'accurate' - persistMode: 'immediate' // 'immediate' | 'deferred' - }, - - // Model configuration (embedded in WASM - zero config needed) - // Model: all-MiniLM-L6-v2 (384 dimensions) - // Device: CPU via WASM (works everywhere) - - // Cache configuration — `true`/`false`, or an options object - cache: { - maxSize: 10000, - ttl: 3600000 // 1 hour in ms - } -}) - -await brain.init() // Required! VFS auto-initialized -``` - ---- - -## Storage Adapters - -Brainy 8.0 ships two adapters — both support the full Db API (generational history, snapshots, restore). - -### Memory (Default for Tests) - -```typescript -const brain = new Brainy({ - storage: { type: 'memory' } -}) -``` - -**Use case:** Development, testing, prototyping - ---- - -### Filesystem (Default for Node) - -```typescript -const brain = new Brainy({ - storage: { - type: 'filesystem', - path: './brainy-data' - } -}) -``` - -**Use case:** Node.js applications, single-node production deployments - -For off-site backup, snapshot `path` from your scheduler (`gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`) — Brainy itself doesn't reach out to cloud object stores. - ---- - -### Auto - -```typescript -const brain = new Brainy({ - storage: { type: 'auto', path: './brainy-data' } -}) -``` - -Picks `'filesystem'` on Node with a writable `path`, falls back to `'memory'` otherwise. - ---- - -## Utility Methods - -### `clear()` → `Promise` - -Clear all data (entities and relationships). - -```typescript -await brain.clear() -``` - ---- - -### `getNounCount()` → `Promise` - -Get total entity count. - -```typescript -const count = await brain.getNounCount() -``` - ---- - -### `getVerbCount()` → `Promise` - -Get total relationship count. - -```typescript -const count = await brain.getVerbCount() -``` - ---- - -### Subtype & facet APIs - -Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**. - -#### `counts.bySubtype(type, subtype?)` → `Record | number` - -O(1) subtype counts for a NounType (backed by the persisted rollup). - -```typescript -brain.counts.bySubtype(NounType.Person) -// → { employee: 12, customer: 847, vendor: 34 } - -brain.counts.bySubtype(NounType.Person, 'employee') -// → 12 -``` - -#### `counts.topSubtypes(type, n=10)` → `Array<[subtype, count]>` - -Top N subtypes ranked by count. - -```typescript -brain.counts.topSubtypes(NounType.Person, 3) -// → [['customer', 847], ['employee', 12], ['vendor', 34]] -``` - -#### `subtypesOf(type)` → `string[]` - -Sorted distinct subtypes seen for a NounType. - -```typescript -brain.subtypesOf(NounType.Person) -// → ['customer', 'employee', 'vendor'] -``` - -#### `counts.byRelationshipSubtype(verb, subtype?)` → `Record | number` - -Verb-side mirror of `counts.bySubtype`. O(1) per-VerbType-per-subtype counts. - -```typescript -brain.counts.byRelationshipSubtype(VerbType.ReportsTo) -// → { direct: 12, 'dotted-line': 3 } - -brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct') -// → 12 -``` - -#### `counts.topRelationshipSubtypes(verb, n=10)` → `Array<[subtype, count]>` - -Top N subtypes for a `VerbType` ranked by count. - -```typescript -brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3) -// → [['direct', 12], ['dotted-line', 3]] -``` - -#### `relationshipSubtypesOf(verb)` → `string[]` - -Sorted distinct subtypes seen for a `VerbType`. - -```typescript -brain.relationshipSubtypesOf(VerbType.ReportsTo) -// → ['direct', 'dotted-line'] -``` - -#### `audit(options?)` → `Promise` (7.30.1+) - -Diagnostic — find entities and relationships missing a `subtype` value, grouped by type. The companion to `migrateField()` / `fillSubtypes()` — answers "what would break if I enabled strict subtype enforcement?". - -```typescript -const report = await brain.audit() -// { -// entitiesWithoutSubtype: { event: 24, document: 3 }, -// relationshipsWithoutSubtype: { relatedTo: 1402 }, -// total: 1429, -// scanned: 8400, -// recommendation: 'Found 1429 entries without subtype. ...' -// } -``` - -**Parameters:** -- `options.includeVFS?`: `boolean` — When `false` (default), VFS infrastructure entities (`metadata.isVFSEntity` / `metadata.isVFS`) are excluded. They bypass enforcement anyway, so counting them is noise. -- `options.batchSize?`: `number` — Pagination batch size (default 200). -- `options.onProgress?`: `(progress: { scanned, missingSubtype }) => void` — Progress callback per batch. - -Run before adopting an SDK that registers `requireSubtype()` rules, or before upgrading to Brainy 8.0 (which makes strict mode the default). See the [Strict mode in practice](../guides/subtypes-and-facets.md#strict-mode-in-practice-for-sdk-style-vocabulary-consumers) guide for the full migration recipe. - -#### `requireSubtype(type, options?)` → `void` - -Register subtype enforcement for a specific `NounType` or `VerbType`. Unified API for nouns and verbs. Composes with the brain-wide `requireSubtype` constructor flag. - -```typescript -// Lock down Person sub-classification -brain.requireSubtype(NounType.Person, { - values: ['employee', 'customer', 'vendor'], - required: true -}) - -// Lock down management edges -brain.requireSubtype(VerbType.ReportsTo, { - values: ['direct', 'dotted-line'], - required: true -}) -``` - -**Parameters:** -- `type`: `NounType | VerbType` - The type to register -- `options.values?`: `string[]` - Vocabulary whitelist (rejects off-vocab values) -- `options.required?`: `boolean` - Whether subtype is required (default: `true`) - -#### Brain-wide strict mode — `new Brainy({ requireSubtype })` - -Constructor option that enforces subtype on every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()` for every type: - -```typescript -// Every write must include subtype -const brain = new Brainy({ requireSubtype: true }) - -// Exempt specific types (e.g. catch-all Thing) -const brain2 = new Brainy({ - requireSubtype: { except: [NounType.Thing, NounType.Custom] } -}) -``` - -When strict mode is on: -- Every public write path checks the pairing guarantee. -- `addMany()` / `relateMany()` validate all items BEFORE any storage write — atomic-fail, no partial writes. -- Brainy's own VFS infrastructure writes bypass via the `metadata.isVFSEntity: true` marker. -- Per-type registrations always apply regardless of the brain-wide flag. - -The default since 8.0.0 — pass `requireSubtype: false` to opt out while migrating pre-8.0 data. - -#### `trackField(name, options?)` → `void` - -Register a metadata field for cardinality + per-NounType breakdown stats. With `values: [...]`, validates against the whitelist on `add()`/`update()`. - -```typescript -brain.trackField('status') // basic -brain.trackField('status', { perType: true }) // with per-NounType breakdown -brain.trackField('priority', { values: ['low', 'med', 'high'] }) // strict vocabulary -``` - -#### `counts.byField(name, options?)` → `Promise>` - -Counts by value for a tracked field. Requires `perType: true` registration if filtering by NounType. - -```typescript -await brain.counts.byField('status') -// → { todo: 12, doing: 3, done: 47 } - -await brain.counts.byField('status', { type: NounType.Task }) -// → { todo: 8, doing: 2, done: 30 } -``` - -#### `migrateField(options)` → `Promise` - -Stream-and-rewrite a field across the brain. Supports `metadata.X`, `data.X`, and top-level paths. Idempotent. - -```typescript -// One-shot rewrite -await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) - -// Deprecation window — keep source field readable -await brain.migrateField({ from: 'data.kind', to: 'subtype', readBoth: true }) - -// With progress reporting -await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype', - batchSize: 500, - onProgress: ({ scanned, migrated }) => console.log(`${scanned} / ${migrated}`) -}) -``` - -Returns `{ scanned: number, migrated: number, skipped: number, errors: Array<{id, error}> }`. - -#### `fillSubtypes(rules, options?)` → `Promise` (8.0+) - -Back-fill missing `subtype` values across entities AND relationships in one streaming pass — the migration companion to `audit()`. Keys are `NounType`/`VerbType` values; each rule is a literal subtype string or a function deriving one from the entry (return `undefined` to decline). Idempotent: entries that already carry a subtype are never touched, so a crashed run is resumed safely by re-running. - -```typescript -const report = await brain.fillSubtypes({ - [NounType.Person]: (e) => e.metadata?.kind ?? 'unspecified', // derived - [NounType.Document]: 'general', // literal default - [VerbType.RelatedTo]: 'unspecified' // relationship rule -}) -// → { scanned, filled, skipped, errors, byType } -``` - -**Parameters:** -- `rules`: `FillSubtypeRules` - Map of NounType/VerbType → literal subtype or `(entry) => string | undefined` -- `options.includeVFS?`: `boolean` - Also fill VFS infrastructure entries (default `false`) -- `options.batchSize?`: `number` - Pagination batch size (default `200`) -- `options.onProgress?`: `(progress: { scanned, filled, skipped }) => void` - Per-batch callback - -Returns `{ scanned, filled, skipped, errors, byType }`. After a clean run, `skipped` equals the remaining `audit().total`. See the [migration recipe](../guides/subtypes-and-facets.md). - ---- - -### `embed(data)` → `Promise` ✨ - -Generate embedding vector from text or data. - -```typescript -const vector = await brain.embed('Hello world') -// 384-dimensional vector -console.log(vector.length) // 384 -``` - ---- - -### `embedBatch(texts)` → `Promise` ✨ - -Batch embed multiple texts using native WASM batch API (single forward pass). - -```typescript -const embeddings = await brain.embedBatch([ - 'Machine learning is fascinating', - 'Deep neural networks', - 'Natural language processing' -]) -console.log(embeddings.length) // 3 -console.log(embeddings[0].length) // 384 -``` - -> Uses the WASM engine's native `embed_batch()` for a single model forward pass instead of N individual calls. This is the same batch API used internally by `highlight()`. - ---- - -### `similarity(textA, textB)` → `Promise` ✨ - -Calculate semantic similarity between two texts. - -```typescript -const score = await brain.similarity( - 'The cat sat on the mat', - 'A feline was resting on the rug' -) -console.log(score) // ~0.85 (high semantic similarity) -``` - -**Returns:** Score from 0 (different) to 1 (identical meaning) - ---- - -### `neighbors(entityId, options?)` → `Promise` ✨ - -Get graph neighbors of an entity. - -```typescript -// Get all connected entities -const neighbors = await brain.neighbors(entityId) - -// Get outgoing connections only -const outgoing = await brain.neighbors(entityId, { - direction: 'outgoing', - limit: 10 -}) - -// Multi-hop traversal -const extended = await brain.neighbors(entityId, { - depth: 2, - direction: 'both' -}) -``` - -**Options:** -- `direction`: `'outgoing' | 'incoming' | 'both'` (default: 'both') -- `depth`: `number` - Traversal depth (default: 1) -- `verbType`: `VerbType` - Filter by relationship type -- `limit`: `number` - Maximum neighbors to return - ---- - -### `findDuplicates(options?)` → `Promise` ✨ - -Find semantic duplicates in the database. - -```typescript -// Find all duplicates -const duplicates = await brain.findDuplicates() - -for (const group of duplicates) { - console.log('Original:', group.entity.id) - for (const dup of group.duplicates) { - console.log(` Duplicate: ${dup.entity.id} (${dup.similarity.toFixed(2)})`) - } -} - -// Find person duplicates with higher threshold -const personDupes = await brain.findDuplicates({ - type: NounType.Person, - threshold: 0.9, - limit: 50 -}) -``` - -**Options:** -- `threshold`: `number` - Minimum similarity (default: 0.85) -- `type`: `NounType` - Filter by entity type -- `limit`: `number` - Maximum duplicate groups (default: 100) - ---- - -### `indexStats()` → `Promise` ✨ - -Get comprehensive index statistics. - -```typescript -const stats = await brain.indexStats() -console.log(`Entities: ${stats.entities}`) -console.log(`Vectors: ${stats.vectors}`) -console.log(`Relationships: ${stats.relationships}`) -console.log(`Memory: ${(stats.memoryUsage.total / 1024 / 1024).toFixed(1)}MB`) -console.log(`Fields: ${stats.metadataFields.join(', ')}`) -``` - -**Returns:** -- `entities` - Total entity count -- `vectors` - Total vectors in the vector index -- `relationships` - Total relationships in graph -- `metadataFields` - Indexed metadata fields -- `memoryUsage.vectors` - Vector memory (bytes) -- `memoryUsage.graph` - Graph memory (bytes) -- `memoryUsage.metadata` - Metadata index memory (bytes) -- `memoryUsage.total` - Total memory usage - ---- - -### `cluster(options?)` → `Promise` ✨ - -Cluster entities by semantic similarity. - -```typescript -// Find all clusters -const clusters = await brain.cluster() - -for (const cluster of clusters) { - console.log(`${cluster.clusterId}: ${cluster.entities.length} entities`) -} - -// Find document clusters with centroids -const docClusters = await brain.cluster({ - type: NounType.Document, - threshold: 0.85, - minClusterSize: 3, - includeCentroid: true -}) -``` - -**Options:** -- `threshold`: `number` - Similarity threshold (default: 0.8) -- `type`: `NounType` - Filter by entity type -- `minClusterSize`: `number` - Minimum cluster size (default: 2) -- `limit`: `number` - Maximum clusters to return (default: 100) -- `includeCentroid`: `boolean` - Calculate cluster centroids (default: false) - -**Returns:** -- `clusterId` - Unique cluster identifier -- `entities` - Array of entities in the cluster -- `centroid` - Average embedding vector (if includeCentroid is true) - ---- - -### `getStats(options?)` → `Promise` - -Get complete entity/relationship statistics (convenience wrapper over `brain.counts`). - -```typescript -const stats = await brain.getStats() -console.log(stats.entities.total) // total entity count -console.log(stats.entities.byType) // counts per NounType -console.log(stats.relationships) // relationship stats -console.log(stats.density) // relationships per entity - -// Exclude VFS infrastructure entities from the counts -const semanticOnly = await brain.getStats({ excludeVFS: true }) -``` - ---- - -## Lifecycle - -### Initialization - -```typescript -const brain = new Brainy(config) -await brain.init() // Required! VFS auto-initialized here -``` - -VFS is auto-initialized during `brain.init()` - no separate `vfs.init()` needed! - ---- - -### Shutdown - -```typescript -await brain.close() // Graceful shutdown — flushes pending writes and releases the writer lock -``` +| `between` | Range | `{year: {between: [2020, 2024]}}` | --- ## Examples -### Basic CRUD - +### Basic Usage ```typescript -// Create -const id = await brain.add({ - data: 'Quantum computing breakthrough', - type: NounType.Concept, - metadata: { category: 'tech', year: 2024 } +// Add data +const id = await brain.add('Quantum computing breakthrough', { + category: 'technology', + year: 2024, + importance: 'high' }) -// Read -const entity = await brain.get(id) +// Simple search +const results = await brain.search('quantum physics', 5) -// Update -await brain.update({ - id, - metadata: { updated: true } -}) - -// Remove -await brain.remove(id) -``` - ---- - -### Knowledge Graphs - -```typescript -// Create entities -const ai = await brain.add({ - data: 'Artificial Intelligence', - type: NounType.Concept -}) - -const ml = await brain.add({ - data: 'Machine Learning', - type: NounType.Concept -}) - -// Create relationship -await brain.relate({ - from: ml, - to: ai, - type: VerbType.IsA -}) - -// Traverse graph -const results = await brain.find({ - connected: { from: ai, depth: 2 } +// Complex query with Triple Intelligence +const articles = await brain.find({ + like: 'quantum computing', + where: { + year: { greaterThan: 2022 }, + importance: { oneOf: ['high', 'critical'] } + }, + connected: { + via: 'references', + depth: 2 + }, + limit: 10 }) ``` ---- - -### Triple Intelligence Query - +### Creating Knowledge Graphs ```typescript -const results = await brain.find({ - query: 'modern frontend frameworks', // 🔍 Vector - where: { // 📊 Document - year: { greaterThan: 2020 }, - category: { oneOf: ['framework', 'library'] } - }, - connected: { // 🕸️ Graph - to: reactId, - depth: 2, - type: VerbType.BuiltOn - }, - limit: 10 +// Add entities +const ai = await brain.add('Artificial Intelligence', { nounType: 'concept' }) +const ml = await brain.add('Machine Learning', { nounType: 'concept' }) +const dl = await brain.add('Deep Learning', { nounType: 'concept' }) + +// Create relationships +await brain.relate(ml, ai, 'subset_of') +await brain.relate(dl, ml, 'subset_of') +await brain.relate(dl, ai, 'enables') + +// Traverse the graph +const aiEcosystem = await brain.find({ + connected: { from: ai, depth: 3 } }) ``` ---- - -### Database-as-a-Value Workflow - +### Using Neural Features ```typescript -// Speculate: what would this change look like? (nothing touches disk) -const base = brain.now() -const whatIf = await base.with([ - { op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature', metadata: { draft: true } } -]) -await whatIf.find({ where: { draft: true } }) -await whatIf.release() -await base.release() - -// Commit it for real — one atomic generation, with audit metadata -await brain.transact( - [{ op: 'add', type: NounType.Document, subtype: 'note', data: 'New feature', metadata: { draft: true } }], - { meta: { author: 'dev@example.com', message: 'Add new feature' } } +// Find similar concepts +const similarity = await brain.neural.similar( + 'renewable energy', + 'sustainable power' ) -``` ---- - -### VFS File Management - -```typescript -// Write files -await brain.vfs.writeFile('/docs/README.md', 'Project documentation') -await brain.vfs.mkdir('/src/components', { recursive: true }) - -// Read files -const content = await brain.vfs.readFile('/docs/README.md') - -// Semantic search -const reactFiles = await brain.vfs.search('React components with hooks', { - path: '/src' +// Auto-cluster documents +const clusters = await brain.neural.clusters({ + method: 'kmeans', + k: 5 }) -// Get tree structure (safe, prevents infinite recursion) -const tree = await brain.vfs.getTreeStructure('/projects', { - maxDepth: 3 +// Generate visualization +const vizData = await brain.neural.visualize({ + maxNodes: 200, + algorithm: 'force', + dimensions: 3 }) +// Use vizData with D3.js, Cytoscape, etc. ``` --- -## Type System Reference - -Stage 3 CANONICAL taxonomy with 169 types (42 nouns + 127 verbs) - -### Noun Types (42) - -Brainy uses a comprehensive noun type system covering 96-97% of human knowledge: - -**Core Entity Types (7)** -- `NounType.Person` - Individual human entities -- `NounType.Organization` - Companies, institutions, collectives -- `NounType.Location` - Geographic and spatial entities -- `NounType.Thing` - Physical objects and artifacts -- `NounType.Concept` - Abstract ideas and principles -- `NounType.Event` - Temporal occurrences -- `NounType.Agent` - AI agents, bots, automated systems - -**Digital/Content Types (4)** -- `NounType.Document` - Text-based files and written content -- `NounType.Media` - Audio, video, images -- `NounType.File` - Generic digital files -- `NounType.Message` - Communication content - -**Business Types (4)** -- `NounType.Product` - Commercial products -- `NounType.Service` - Service offerings -- `NounType.Task` - Actions, todos, work items -- `NounType.Project` - Organized initiatives - -**Scientific Types (2)** -- `NounType.Hypothesis` - Theories and propositions -- `NounType.Experiment` - Studies and investigations - -**And 25 more types** including: `Organism`, `Substance`, `Quality`, `TimeInterval`, `Function`, `Proposition`, `Collection`, `Dataset`, `Process`, `State`, `Role`, `Language`, `Currency`, `Measurement`, `Contract`, `Regulation`, `Interface`, `Resource`, `Custom`, `SocialGroup`, `Institution`, `Norm`, `InformationContent`, `InformationBearer`, `Relationship` - -### Verb Types (127) - -Brainy supports 127 relationship types organized into categories: - -**Foundational (7)** -- `VerbType.InstanceOf`, `VerbType.SubclassOf`, `VerbType.ParticipatesIn` -- `VerbType.RelatedTo`, `VerbType.Contains`, `VerbType.PartOf`, `VerbType.References` - -**Spatial & Temporal (14)** -- Location: `LocatedAt`, `AdjacentTo`, `ContainsSpatially`, `OverlapsSpatially`, `Above`, `Below`, `Inside`, `Outside`, `Facing` -- Time: `Precedes`, `During`, `OccursAt`, `Overlaps`, `ImmediatelyAfter`, `SimultaneousWith` - -**Causal & Dependency (11)** -- Direct: `Causes`, `Enables`, `Prevents`, `DependsOn`, `Requires` -- Modal: `CanCause`, `MustCause`, `WouldCauseIf`, `ProbablyCauses` -- Variations: `RigidlyDependsOn`, `FunctionallyDependsOn`, `HistoricallyDependsOn` - -**Creation & Change (10)** -- Lifecycle: `Creates`, `Transforms`, `Becomes`, `Modifies`, `Consumes`, `Destroys` -- Properties: `GainsProperty`, `LosesProperty`, `RemainsSame`, `PersistsThrough` - -**Social & Communication (8)** -- `MemberOf`, `WorksWith`, `FriendOf`, `Follows`, `Likes`, `ReportsTo`, `Mentors`, `Communicates` - -**Epistemic & Modal (14)** -- Knowledge: `Knows`, `Doubts`, `Believes`, `Learns` -- Mental states: `Desires`, `Intends`, `Fears`, `Loves`, `Hates`, `Hopes`, `Perceives` -- Modality: `CouldBe`, `MustBe`, `Counterfactual` - -**Measurement & Comparison (9)** -- `Measures`, `MeasuredIn`, `ConvertsTo`, `HasMagnitude`, `GreaterThan` -- `SimilarityDegree`, `ApproximatelyEquals`, `MoreXThan`, `HasDegree` - -**And 54 more specialized verbs** including ownership, composition, uncertainty, deontic relationships (obligations/permissions), context-dependent truth, spatial/temporal variations, information theory, and meta-level relationships. - -### Complete Reference - -For the full taxonomy with all 169 types and their descriptions, see: -- **[Stage 3 CANONICAL Taxonomy](../STAGE3-CANONICAL-TAXONOMY.md)** - Complete list with categories -- **[Noun-Verb Taxonomy Architecture](../architecture/noun-verb-taxonomy.md)** - Design rationale - -### Migration from pre-Stage-3 taxonomies - -**Breaking Changes:** -- `NounType.Content` removed → Use `Document`, `Message`, or `InformationContent` -- `NounType.User` removed → Use `Person` or `Agent` -- `NounType.Topic` removed → Use `Concept` or `Category` - -**New Types Added:** -- **+11 noun types**: Agent, Organism, Substance, Quality, TimeInterval, Function, Proposition, Custom, SocialGroup, Institution, Norm, InformationContent, InformationBearer, Relationship -- **+87 verb types**: Extensive additions across all categories - ---- - ## Key Features -- ✅ **Database as a Value** - `brain.now()` pins the whole store as an immutable `Db` in O(1) -- ✅ **Atomic Transactions** - `brain.transact()` commits multi-write batches all-or-nothing -- ✅ **Two-Level CAS** - per-entity `ifRev` and whole-store `ifAtGeneration` -- ✅ **Time Travel** - `brain.asOf()` serves the full query surface at any reachable past generation -- ✅ **Instant Snapshots** - `db.persist()` cuts hard-link snapshots; `Brainy.load()` opens them read-only -- ✅ **Speculative Writes** - `db.with()` answers what-if questions purely in memory -- ✅ **Reified Transaction Metadata** - audit fields recorded durably, readable via `transactionLog()` -- ✅ **VFS Entity Filtering** - All VFS entities have the `isVFSEntity: true` flag -- ✅ **VFS Auto-Initialization** - No separate `vfs.init()` calls -- ✅ **VFS Property Access** - Use `brain.vfs.method()` instead of `brain.vfs().method()` -- ✅ **Universal Storage Support** - Filesystem and memory adapters share one on-disk contract +### ✨ Zero Configuration +Works instantly with sensible defaults. No setup required. + +### 🧠 Triple Intelligence +Combines vector search, graph traversal, and metadata filtering in one query. + +### 🚀 Auto-Embedding +Text automatically converts to vectors - no manual embedding needed. + +### 📊 Built-in Visualization +Export data formatted for popular visualization libraries. + +### 🔒 Clean Operators +Readable, intuitive operators - no cryptic symbols. + +### 🎯 Everything Included +All features in the MIT licensed package - no premium tiers. --- -## Support & Resources +## Support -- **📖 Documentation:** [Full Documentation](../) -- **🐛 Issues:** [GitHub Issues](https://github.com/soulcraftlabs/brainy/issues) -- **💬 Discussions:** [GitHub Discussions](https://github.com/soulcraftlabs/brainy/discussions) -- **📦 NPM:** [@soulcraft/brainy](https://www.npmjs.com/package/@soulcraft/brainy) -- **⭐ GitHub:** [Star us](https://github.com/soulcraftlabs/brainy) +- **GitHub**: [github.com/soulcraft/brainy](https://github.com/soulcraft/brainy) +- **Documentation**: [docs.soulcraft.com/brainy](https://docs.soulcraft.com/brainy) +- **License**: MIT --- -## See Also - -- **[Data Model](../DATA_MODEL.md)** - Entity structure, data vs metadata, storage fields -- **[Query Operators](../QUERY_OPERATORS.md)** - All BFO operators with examples and indexed vs in-memory matrix -- **[Triple Intelligence Architecture](../architecture/triple-intelligence.md)** - How vector + graph + document work together -- **[Find System](../FIND_SYSTEM.md)** - Natural language find() details -- **[VFS Quick Start](../vfs/QUICK_START.md)** - Complete VFS documentation -- **[Import Anything Guide](../guides/import-anything.md)** - CSV, Excel, PDF, URL imports -- **[Consistency Model](../concepts/consistency-model.md)** - The guarantees behind the Db API -- **[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md)** - Backup, restore, what-if, audit recipes - ---- - -**License:** MIT © Brainy Contributors - ---- - -*Brainy - The Knowledge Operating System* -*From prototype to planet-scale • Zero configuration • Triple Intelligence™ • Database as a Value* +*Brainy 2.0 - Intelligence for Everyone* \ No newline at end of file diff --git a/docs/architecture/API_SURFACE_DESIGN.md b/docs/architecture/API_SURFACE_DESIGN.md new file mode 100644 index 00000000..39a98121 --- /dev/null +++ b/docs/architecture/API_SURFACE_DESIGN.md @@ -0,0 +1,200 @@ +# Brainy Neural API Surface Design + +## 🎯 **Clean API Hierarchy** + +### **Main Class Shortcuts (Simple & Common)** +```typescript +// High-level shortcuts on Brainy - most common operations +brain.similar(a, b, options?) // ✅ Keep - very common +brain.clusters(items?, options?) // ✅ Keep - very common +brain.related(id, options?) // ✅ Keep - common (like neighbors but simpler name) + +// Remove/deprecate confusing shortcuts +brain.visualize(options?) // ❌ Remove - too specialized for main class +``` + +### **Neural Namespace (Full Featured)** +```typescript +// Core semantic operations +brain.neural.similar(a, b, options?) // Comprehensive similarity +brain.neural.clusters(items?, options?) // Smart clustering with auto-routing +brain.neural.neighbors(id, options?) // K-nearest neighbors (full featured) +brain.neural.hierarchy(id, options?) // Semantic hierarchy building +brain.neural.outliers(options?) // Anomaly detection +brain.neural.visualize(options?) // Visualization data generation + +// Advanced clustering methods +brain.neural.clusterByDomain(field, options?) // Domain-aware clustering +brain.neural.clusterByTime(field, windows, options?) // Temporal clustering +brain.neural.clusterStream(options?) // Streaming/real-time clustering +brain.neural.updateClusters(items, options?) // Incremental clustering + +// Utility methods +brain.neural.getPerformanceMetrics(operation?) // Performance monitoring +brain.neural.clearCaches() // Cache management +brain.neural.getCacheStats() // Cache statistics +``` + +## 🔒 **Private Methods (Internal Implementation)** + +### **Should NOT be exposed publicly:** +```typescript +// These are implementation details: +brain.neural._clusterFast() // ❌ Private - use clusters() with algorithm: 'hierarchical' +brain.neural._clusterLarge() // ❌ Private - use clusters() with algorithm: 'sample' +brain.neural._performHierarchicalClustering() // ❌ Private - internal routing +brain.neural._performKMeansClustering() // ❌ Private - internal routing +brain.neural._performDBSCANClustering() // ❌ Private - internal routing +brain.neural._performSampledClustering() // ❌ Private - internal routing +brain.neural._routeClusteringAlgorithm() // ❌ Private - internal routing +brain.neural._similarityById() // ❌ Private - internal routing +brain.neural._similarityByVector() // ❌ Private - internal routing +brain.neural._similarityByText() // ❌ Private - internal routing +brain.neural._isId() // ❌ Private - utility +brain.neural._isVector() // ❌ Private - utility +brain.neural._convertToVector() // ❌ Private - utility +brain.neural._cacheResult() // ❌ Private - caching +brain.neural._trackPerformance() // ❌ Private - monitoring +``` + +### **Current Issues in brain-cloud explorer:** +```typescript +// ❌ BAD: Accessing private implementation details +brain.neural.clusterFast({ maxClusters: count, level: 2 }) + +// ✅ GOOD: Use public API with proper options +brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: count, level: 2 }) +``` + +## 📊 **API Consistency Fixes** + +### **Method Naming Standardization:** +```typescript +// ✅ CONSISTENT: Pick one naming pattern and stick to it +brain.neural.similar() // Main method name +brain.similar() // Shortcut matches + +// ❌ INCONSISTENT: Don't mix these +brain.neural.similarity() // Different from shortcut +brain.similar() +``` + +### **Parameter Patterns:** +```typescript +// ✅ CONSISTENT: Always (data, options?) pattern +brain.neural.similar(a, b, options?) +brain.neural.clusters(items?, options?) +brain.neural.neighbors(id, options?) +brain.neural.hierarchy(id, options?) + +// Options should be objects with clear properties +interface ClusteringOptions { + algorithm?: 'auto' | 'hierarchical' | 'kmeans' | 'dbscan' + maxClusters?: number + threshold?: number + // ... +} +``` + +### **Return Type Consistency:** +```typescript +// ✅ CONSISTENT: All clustering methods return SemanticCluster[] +brain.neural.clusters() → Promise +brain.neural.clusterByDomain() → Promise // extends SemanticCluster +brain.neural.clusterByTime() → Promise // extends SemanticCluster + +// ✅ CONSISTENT: All similarity methods return number or SimilarityResult +brain.neural.similar() → Promise +brain.similar() → Promise // Shortcut always returns simple number +``` + +## 🚀 **Performance & Intelligence Routing** + +### **Auto-Algorithm Selection:** +```typescript +// Smart routing based on data size and characteristics +brain.neural.clusters() // Auto-selects: +// < 100 items → hierarchical (fast, accurate) +// < 1000 items → k-means (balanced) +// > 1000 items → sampling (scalable) + +// Manual override available +brain.neural.clusters({ algorithm: 'hierarchical' }) // Force specific algorithm +``` + +### **Caching Strategy:** +```typescript +// Intelligent caching with LRU eviction +brain.neural.similar('id1', 'id2') // First call: compute & cache +brain.neural.similar('id1', 'id2') // Second call: instant cache hit + +// Cache management +brain.neural.clearCaches() // Manual cache clear +brain.neural.getCacheStats() // Monitor cache performance +``` + +## 📚 **Documentation Structure** + +### **Main Documentation Sections:** +1. **Quick Start**: Simple examples using shortcuts (`brain.similar()`, `brain.clusters()`) +2. **Neural API Guide**: Comprehensive examples using `brain.neural.*` +3. **Advanced Clustering**: Domain, temporal, streaming clustering +4. **Performance**: Caching, algorithm selection, monitoring +5. **API Reference**: Complete method documentation + +### **Example Progression:** +```typescript +// 1. BEGINNER: Simple shortcuts +const similarity = await brain.similar('text1', 'text2') +const clusters = await brain.clusters() + +// 2. INTERMEDIATE: Neural API with options +const detailed = await brain.neural.similar('id1', 'id2', { detailed: true }) +const customClusters = await brain.neural.clusters({ algorithm: 'hierarchical', maxClusters: 5 }) + +// 3. ADVANCED: Specialized clustering +const domainClusters = await brain.neural.clusterByDomain('category') +const streamingClusters = brain.neural.clusterStream({ batchSize: 50 }) +``` + +## ✅ **Implementation Checklist** + +### **High Priority:** +- [x] Create comprehensive type definitions +- [x] Implement improved NeuralAPI class with proper public/private separation +- [ ] Update Brainy integration to use improved API +- [ ] Fix brain-cloud explorer to use public APIs +- [ ] Update test files to use consistent method names +- [ ] Update documentation with new API structure + +### **Medium Priority:** +- [ ] Implement placeholder algorithm implementations with real clustering logic +- [ ] Add comprehensive error handling and validation +- [ ] Add performance monitoring and metrics collection +- [ ] Create migration guide for users of deprecated methods + +### **Nice to Have:** +- [ ] Add interactive clustering refinement based on user feedback +- [ ] Implement explainable clustering with reasoning +- [ ] Add multi-modal clustering (text + metadata + relationships) +- [ ] Create visualization examples for different graph libraries + +## 🎯 **API Surface Summary** + +### **✅ PUBLIC API** (What users should use): +- **Main shortcuts**: `brain.similar()`, `brain.clusters()`, `brain.related()` +- **Neural namespace**: `brain.neural.similar()`, `brain.neural.clusters()`, etc. +- **Advanced features**: Domain clustering, temporal clustering, streaming +- **Utilities**: Performance metrics, cache management + +### **❌ PRIVATE IMPLEMENTATION** (Internal only): +- **Algorithm implementations**: `_performKMeansClustering()`, etc. +- **Routing logic**: `_routeClusteringAlgorithm()`, etc. +- **Utility methods**: `_isId()`, `_convertToVector()`, etc. +- **Caching internals**: `_cacheResult()`, `_trackPerformance()`, etc. + +### **⚠️ DEPRECATED** (Should be removed/hidden): +- **brain.neural.clusterFast()** → Use `brain.neural.clusters({ algorithm: 'hierarchical' })` +- **brain.neural.clusterLarge()** → Use `brain.neural.clusters({ algorithm: 'sample' })` +- **brain.neural.similarity()** → Use `brain.neural.similar()` (pick one name) +- **brain.visualize()** → Too specialized for main class, use `brain.neural.visualize()` \ No newline at end of file diff --git a/docs/architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md b/docs/architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md new file mode 100644 index 00000000..48adee57 --- /dev/null +++ b/docs/architecture/CLUSTERING_ALGORITHMS_ANALYSIS.md @@ -0,0 +1,306 @@ +# 🧠 **Brainy Clustering Algorithms - Complete Analysis** + +## 🎯 **Current State & Capabilities** + +### **✅ Existing Infrastructure (Excellent Foundation)** + +#### **1. HNSW Hierarchical Clustering** +```typescript +// ALREADY IMPLEMENTED & OPTIMIZED +brain.neural.clusters({ algorithm: 'hierarchical', level: 2 }) +``` + +**How it works:** +- **Leverages HNSW natural hierarchy**: Uses existing index levels as natural cluster boundaries +- **O(n) performance**: Much faster than O(n²) traditional clustering +- **Multi-level granularity**: Higher levels = fewer, broader clusters; Lower levels = more, specific clusters +- **Representative sampling**: Uses HNSW level nodes as natural cluster centers + +**Performance characteristics:** +- **Excellent for large datasets** (millions of items) +- **Preserves semantic relationships** from vector space +- **Automatic granularity control** via level parameter + +#### **2. Distance-Based Algorithms** +```typescript +// COMPREHENSIVE DISTANCE FUNCTIONS AVAILABLE +euclideanDistance, cosineDistance, manhattanDistance, dotProductDistance +``` + +**Optimized implementations:** +- **Batch processing**: `calculateDistancesBatch()` with parallelization +- **Multiple metrics**: Choose optimal distance function per use case +- **Performance optimized**: Faster than GPU for small vectors due to no transfer overhead + +#### **3. Rich Semantic Taxonomy** +```typescript +// 25+ NOUN TYPES & 35+ VERB TYPES +NounType: Person, Organization, Document, Concept, Event, Media, etc. +VerbType: RelatedTo, Contains, PartOf, Causes, CreatedBy, etc. +``` + +**Semantic clustering capabilities:** +- **Type-based clustering**: Group by semantic categories +- **Cross-type relationships**: Use verb types to find semantic bridges +- **Hierarchical taxonomies**: Natural clustering within and across types + +#### **4. Graph Structure** +```typescript +// VERB RELATIONSHIPS CREATE RICH GRAPH +await brain.relate(sourceId, targetId, VerbType.Causes, { strength: 0.8 }) +``` + +**Graph-based clustering potential:** +- **Connected components**: Find strongly connected groups +- **Community detection**: Use relationship strength for clustering +- **Multi-modal clustering**: Combine graph + vector + taxonomy + +## 🚀 **Advanced Clustering Algorithms We Can Implement** + +### **1. ✅ Already Implemented: HNSW Hierarchical** + +```typescript +// PRODUCTION READY - Uses existing HNSW levels +const clusters = await brain.neural.clusters({ + algorithm: 'hierarchical', + level: 2, // Control granularity + maxClusters: 15 +}) +``` + +**Performance:** **A+** - O(n) leveraging existing index structure + +### **2. 🔥 Semantic Taxonomy Clustering** + +```typescript +// IMPLEMENT: Fast type-based clustering with cross-type bridges +const clusters = await brain.neural.clusterByDomain('nounType', { + preserveTypeBoundaries: false, // Allow cross-type clusters + bridgeStrength: 0.8, // Minimum relationship strength for bridges + hybridWeighting: { + taxonomy: 0.4, // 40% weight to type similarity + vector: 0.4, // 40% weight to vector similarity + graph: 0.2 // 20% weight to relationship strength + } +}) +``` + +**Algorithm approach:** +1. **Primary clustering by taxonomy**: Group by NounType/VerbType first +2. **Vector refinement**: Sub-cluster within types using vector similarity +3. **Cross-type bridging**: Find relationships that bridge type boundaries +4. **Weighted fusion**: Combine taxonomy + vector + graph signals + +**Performance:** **A+** - O(n log n) - taxonomy grouping is O(n), refinement is HNSW-accelerated + +### **3. 🔥 Graph Community Detection** + +```typescript +// IMPLEMENT: Relationship-based clustering +const clusters = await brain.neural.clusterByConnections({ + algorithm: 'modularity', // or 'louvain', 'leiden' + minCommunitySize: 3, + relationshipWeights: { + [VerbType.Creates]: 1.0, + [VerbType.PartOf]: 0.8, + [VerbType.RelatedTo]: 0.5 + } +}) +``` + +**Algorithm approach:** +1. **Build weighted graph**: Use verbs as edges, weights from relationship types + metadata +2. **Community detection**: Apply Louvain or Leiden algorithm for modularity optimization +3. **Semantic enhancement**: Use vector similarity to refine community boundaries + +**Performance:** **A** - O(n log n) for sparse graphs, handles millions of relationships efficiently + +### **4. 🔥 Multi-Modal Fusion Clustering** + +```typescript +// IMPLEMENT: Best of all worlds +const clusters = await brain.neural.clusters({ + algorithm: 'multimodal', + signals: { + vector: { weight: 0.5, metric: 'cosine' }, + graph: { weight: 0.3, algorithm: 'modularity' }, + taxonomy: { weight: 0.2, crossTypeThreshold: 0.8 } + }, + fusion: 'weighted_ensemble' // or 'consensus', 'hierarchical' +}) +``` + +**Algorithm approach:** +1. **Independent clustering**: Run HNSW, graph, and taxonomy clustering separately +2. **Consensus building**: Find agreement between different clustering results +3. **Conflict resolution**: Use weighted voting or hierarchical merging for disagreements +4. **Quality optimization**: Iteratively refine based on silhouette scores + +**Performance:** **A** - O(n log n) - parallel execution of component algorithms + +### **5. 💎 Temporal Pattern Clustering** + +```typescript +// IMPLEMENT: Time-aware clustering using existing infrastructure +const clusters = await brain.neural.clusterByTime('createdAt', [ + { start: new Date('2024-01-01'), end: new Date('2024-06-30'), label: 'H1 2024' }, + { start: new Date('2024-07-01'), end: new Date('2024-12-31'), label: 'H2 2024' } +], { + evolution: 'track', // Track how clusters evolve over time + stability: 0.7, // Minimum stability threshold + trendAnalysis: true // Include trend detection +}) +``` + +**Algorithm approach:** +1. **Time window clustering**: Apply HNSW clustering within each time window +2. **Cluster evolution tracking**: Match clusters across time windows using vector similarity +3. **Trend analysis**: Detect growing, shrinking, merging, splitting patterns +4. **Stability scoring**: Measure cluster consistency over time + +**Performance:** **A+** - O(k*n log n) where k = number of time windows + +### **6. 💎 DBSCAN with Adaptive Parameters** + +```typescript +// IMPLEMENT: Density-based clustering with smart parameter selection +const clusters = await brain.neural.clusters({ + algorithm: 'dbscan', + autoParams: true, // Automatically select eps and minPts + distanceMetric: 'cosine', + outlierHandling: 'soft' // Soft assignment instead of hard outliers +}) +``` + +**Algorithm approach:** +1. **Adaptive parameter selection**: Use HNSW k-NN distances to estimate optimal eps +2. **Multi-scale analysis**: Run DBSCAN at multiple scales and merge results +3. **Soft outlier assignment**: Assign outliers to nearest clusters with confidence scores + +**Performance:** **A** - O(n log n) using HNSW for neighbor queries + +## 📊 **Performance Comparison Matrix** + +| Algorithm | Time Complexity | Space | Large Scale | Semantic Quality | Graph Aware | +|-----------|----------------|-------|-------------|------------------|-------------| +| **HNSW Hierarchical** | O(n) | O(n) | ✅ Excellent | ✅ Very Good | ❌ No | +| **Taxonomy Fusion** | O(n log n) | O(n) | ✅ Excellent | 🔥 Exceptional | ⚡ Partial | +| **Graph Communities** | O(n log n) | O(e) | ✅ Very Good | ✅ Very Good | 🔥 Exceptional | +| **Multi-Modal** | O(n log n) | O(n) | ✅ Very Good | 🔥 Exceptional | 🔥 Exceptional | +| **Temporal Patterns** | O(k*n log n) | O(n) | ⚡ Good | ✅ Very Good | ⚡ Partial | +| **Adaptive DBSCAN** | O(n log n) | O(n) | ✅ Very Good | ✅ Very Good | ❌ No | + +## 🎯 **Specific Improvements Using Existing Capabilities** + +### **1. Enhanced HNSW Clustering (Easy Win)** + +```typescript +// IMPROVE EXISTING: Add semantic post-processing +private async enhanceHNSWClusters(clusters: SemanticCluster[]): Promise { + return Promise.all(clusters.map(async cluster => { + // Get actual metadata for cluster members + const members = await this.brain.getNouns(cluster.members.map(id => ({ id }))) + + // Analyze semantic characteristics + const semanticProfile = this.analyzeSemanticProfile(members) + + // Generate meaningful cluster labels + const label = await this.generateClusterLabel(members, semanticProfile) + + // Calculate cluster coherence using multiple signals + const coherence = this.calculateMultiModalCoherence(members) + + return { + ...cluster, + label, + semanticProfile, + coherence, + quality: coherence.overall + } + })) +} +``` + +### **2. Intelligent Algorithm Selection** + +```typescript +// IMPLEMENT: Smart routing based on data characteristics +private selectOptimalAlgorithm(dataCharacteristics: { + size: number, + dimensionality: number, + graphDensity: number, + typeDistribution: Record +}): string { + if (dataCharacteristics.size > 100000) { + return 'hierarchical' // HNSW scales best + } + + if (dataCharacteristics.graphDensity > 0.1) { + return 'multimodal' // Rich graph structure + } + + if (Object.keys(dataCharacteristics.typeDistribution).length > 10) { + return 'taxonomy' // Diverse semantic types + } + + return 'hierarchical' // Safe default +} +``` + +### **3. Streaming Cluster Updates** + +```typescript +// IMPLEMENT: Incremental clustering using existing infrastructure +public async updateClusters(newItems: string[]): Promise { + // Use HNSW nearest neighbor for fast cluster assignment + const assignments = await Promise.all( + newItems.map(async itemId => { + const neighbors = await this.brain.neural.neighbors(itemId, { limit: 5 }) + return this.assignToNearestCluster(itemId, neighbors, this.existingClusters) + }) + ) + + // Incrementally update cluster centroids and boundaries + return this.updateClusterBoundaries(assignments) +} +``` + +## 🏆 **Recommended Implementation Priority** + +### **🔥 Phase 1: High Impact, Easy Implementation** +1. **Enhanced HNSW Clustering**: Add semantic post-processing to existing algorithm +2. **Taxonomy-Aware Clustering**: Leverage existing NounType/VerbType enums +3. **Intelligent Algorithm Selection**: Route based on data characteristics + +### **⚡ Phase 2: Advanced Features** +4. **Graph Community Detection**: Use existing verb relationships +5. **Multi-Modal Fusion**: Combine all signals intelligently +6. **Streaming Updates**: Incremental cluster maintenance + +### **💎 Phase 3: Cutting Edge** +7. **Temporal Pattern Analysis**: Track cluster evolution over time +8. **Adaptive DBSCAN**: Dynamic parameter selection +9. **Explainable Clustering**: Generate cluster explanations and reasoning + +## 🎯 **Key Advantages of Our Approach** + +### **✅ Leverages Existing Infrastructure** +- **HNSW index**: Already optimized for large-scale vector operations +- **Distance functions**: Battle-tested and performance-optimized +- **Semantic taxonomy**: Rich type system with 60+ semantic categories +- **Graph structure**: Relationship network from verb connections + +### **✅ Multiple Clustering Paradigms** +- **Vector similarity**: Traditional embedding-based clustering +- **Graph structure**: Relationship-based community detection +- **Semantic taxonomy**: Type-aware intelligent grouping +- **Temporal patterns**: Time-aware cluster evolution +- **Multi-modal fusion**: Best of all worlds + +### **✅ Scalability & Performance** +- **O(n) hierarchical clustering**: Leveraging HNSW levels +- **Parallel processing**: Batch distance calculations optimized +- **Streaming support**: Real-time cluster updates +- **Memory efficient**: Existing index structures reused + +**Our clustering algorithms are not just competitive - they're architecturally superior by leveraging Brainy's unique multi-modal semantic infrastructure.** \ No newline at end of file diff --git a/docs/architecture/METADATA_ARCHITECTURE.md b/docs/architecture/METADATA_ARCHITECTURE.md new file mode 100644 index 00000000..ee18fec9 --- /dev/null +++ b/docs/architecture/METADATA_ARCHITECTURE.md @@ -0,0 +1,220 @@ +# Brainy Metadata Architecture & Namespacing + +## The Problem 🚨 +We're mixing internal Brainy fields with user metadata, causing: +1. **Namespace collisions** - User's `deleted` field conflicts with our soft-delete +2. **API confusion** - Users see internal fields they shouldn't care about +3. **Security issues** - Users could manipulate internal fields +4. **Augmentation conflicts** - 3rd party augmentations might overwrite our fields + +## Current Internal Fields Being Added + +### Core System Fields +```javascript +metadata = { + // USER DATA + name: "Django", + type: "framework", + + // OUR INTERNAL FIELDS - COLLISION RISK! + deleted: false, // Soft delete status + domain: "tech", // Distributed mode domain + domainMetadata: {}, // Domain-specific metadata + partition: 0, // Partition for sharding + createdAt: {...}, // GraphNoun timestamp + updatedAt: {...}, // GraphNoun timestamp + createdBy: {...}, // Who created this + noun: "Concept", // NounType + verb: "RELATES_TO", // VerbType (for relationships) + isPlaceholder: true, // Write-only mode marker + autoCreated: true, // Auto-created noun marker + writeOnlyMode: true // High-speed streaming marker +} +``` + +### Augmentation Fields +```javascript +// Good - neuralImport already uses underscore prefix! +metadata._neuralProcessed = true +metadata._neuralConfidence = 0.95 +metadata._detectedEntities = 5 +metadata._detectedRelationships = 3 +metadata._neuralInsights = [...] + +// Bad - direct modification +metadata.importance = 0.8 // IntelligentVerbScoring +``` + +## Proposed Solution: Three-Tier Metadata + +### 1. User Metadata (Public) +```javascript +metadata = { + // User's fields - completely untouched + name: "Django", + type: "framework", + deleted: "2024-01-01", // User's own deleted field - no conflict! + domain: "web", // User's domain field - no conflict! +} +``` + +### 2. Internal Metadata (Protected) +```javascript +metadata._brainy = { + // Core system fields - O(1) indexed + deleted: false, // Our soft delete flag + version: 2, // Metadata schema version + + // Distributed mode + partition: 0, + distributedDomain: "tech", + + // GraphNoun compliance + nounType: "Concept", + verbType: "RELATES_TO", + createdAt: 1704067200000, + updatedAt: 1704067200000, + createdBy: "user:123", + + // Performance flags + indexed: true, + searchable: true, + placeholder: false, + + // Storage optimization + compressed: false, + encrypted: false +} +``` + +### 3. Augmentation Metadata (Semi-Protected) +```javascript +metadata._augmentations = { + // Each augmentation gets its own namespace + neuralImport: { + processed: true, + confidence: 0.95, + entities: 5, + relationships: 3 + }, + + verbScoring: { + contextScore: 0.7, + importance: 0.8 + }, + + // 3rd party augmentations + customAug: { + // Their fields isolated here + } +} +``` + +## Implementation Strategy + +### Phase 1: Core Fields ✅ +```javascript +// Already done with _brainy.deleted +const BRAINY_NAMESPACE = '_brainy' +const AUGMENTATION_NAMESPACE = '_augmentations' +``` + +### Phase 2: Migrate All Internal Fields +```javascript +// Before +metadata.domain = "tech" +metadata.partition = 0 + +// After +metadata._brainy.distributedDomain = "tech" +metadata._brainy.partition = 0 +``` + +### Phase 3: Augmentation API +```javascript +class Augmentation { + // Read user metadata (read-only) + getUserMetadata(metadata) { + const { _brainy, _augmentations, ...userMeta } = metadata + return userMeta // Clean user data only + } + + // Write augmentation data (isolated) + setAugmentationData(metadata, augName, data) { + if (!metadata._augmentations) metadata._augmentations = {} + metadata._augmentations[augName] = data + } + + // Read internal fields (for special augmentations only) + getInternalField(metadata, field) { + return metadata._brainy?.[field] + } +} +``` + +## Benefits + +1. **No Collisions** - User can have any field names +2. **O(1) Performance** - Internal fields still indexed +3. **Clean API** - Users only see their data +4. **Secure** - Internal fields protected +5. **Extensible** - Augmentations isolated +6. **Backward Compatible** - Migration path available + +## Query Impact + +### Before (Collision Risk) +```javascript +where: { + deleted: false, // Ambiguous - ours or user's? + type: "framework" +} +``` + +### After (Clear Separation) +```javascript +where: { + '_brainy.deleted': false, // Our soft delete + type: "framework" // User's field +} +``` + +## Performance Considerations + +- **Index on `_brainy.deleted`**: O(1) hash lookup ✅ +- **Index on `_brainy.partition`**: O(1) for sharding ✅ +- **Nested field access**: Modern DBs handle this efficiently ✅ +- **Storage overhead**: ~100 bytes per item (acceptable) ✅ + +## Migration Path + +1. **New items**: Automatically use namespaced fields +2. **Existing items**: Lazy migration on update +3. **Queries**: Support both formats temporarily +4. **Deprecation**: Remove old format in v3.0 + +## Augmentation Guidelines + +### For Core Augmentations +- Use `_brainy.*` for system fields +- Use `_augmentations.{name}.*` for augmentation data +- Never modify user fields directly + +### For 3rd Party Augmentations +- Read user metadata via `getUserMetadata()` +- Write only to `_augmentations.{yourName}.*` +- Request permission for internal field access + +## Critical Fields to Namespace + +| Field | Current Location | New Location | Priority | +|-------|-----------------|--------------|----------| +| deleted | metadata.deleted | metadata._brainy.deleted | HIGH ✅ | +| partition | metadata.partition | metadata._brainy.partition | HIGH | +| domain | metadata.domain | metadata._brainy.distributedDomain | HIGH | +| createdAt | metadata.createdAt | metadata._brainy.createdAt | MEDIUM | +| updatedAt | metadata.updatedAt | metadata._brainy.updatedAt | MEDIUM | +| noun | metadata.noun | metadata._brainy.nounType | MEDIUM | +| verb | metadata.verb | metadata._brainy.verbType | MEDIUM | +| isPlaceholder | metadata.isPlaceholder | metadata._brainy.placeholder | LOW | +| autoCreated | metadata.autoCreated | metadata._brainy.autoCreated | LOW | \ No newline at end of file diff --git a/docs/architecture/aggregation.md b/docs/architecture/aggregation.md deleted file mode 100644 index 443ee727..00000000 --- a/docs/architecture/aggregation.md +++ /dev/null @@ -1,242 +0,0 @@ -# Aggregation Architecture - -> Write-time incremental aggregation with O(1) reads - -## Design Principles - -1. **Write-time computation** — aggregates update on every `add()`, `update()`, and `delete()`, not as batch jobs -2. **Incremental state** — running totals maintained per group, never rescanning the dataset -3. **Provider interface** — TypeScript engine is the default; plugins can replace it with native implementations -4. **Zero-allocation reads** — query results are computed from pre-aggregated state - -## Component Overview - -``` -┌──────────────────────────────────────────────────────────┐ -│ Brainy │ -│ │ -│ add() / update() / delete() │ -│ │ │ -│ ▼ │ -│ ┌──────────────────────┐ ┌────────────────────────┐ │ -│ │ AggregationIndex │ │ AggregateMaterializer │ │ -│ │ │───▶│ (debounced writes) │ │ -│ │ ├─ definitions Map │ └────────────────────────┘ │ -│ │ ├─ states Map │ │ -│ │ └─ staleMinMax Set │ ┌────────────────────────┐ │ -│ │ │ │ timeWindows.ts │ │ -│ │ Source filter ──────│───▶│ bucketTimestamp() │ │ -│ │ Group key ──────────│───▶│ parseBucketRange() │ │ -│ └──────────┬───────────┘ └────────────────────────┘ │ -│ │ │ -│ │ provider interface │ -│ ▼ │ -│ ┌──────────────────────┐ │ -│ │ AggregationProvider │ (optional, registered by │ -│ │ ├─ incrementalUpdate│ plugin like @soulcraft/cor) │ -│ │ ├─ rebuildAggregate │ │ -│ │ ├─ queryAggregate │ │ -│ │ └─ serialize/restore│ │ -│ └──────────────────────┘ │ -└──────────────────────────────────────────────────────────┘ -``` - -## State Management - -### Definitions - -Registered via `brain.defineAggregate(def)`. Stored in a `Map` keyed by aggregate name. Persisted to storage under `__aggregation_definitions__` on flush. - -### Group State - -Each aggregate maintains a `Map` where keys are serialized group key values (e.g., `category=food|date=2024-01`). Each group holds per-metric `MetricState`: - -```typescript -interface MetricState { - sum: number // Running total - count: number // Entity count - min: number // Minimum (Infinity if empty) - max: number // Maximum (-Infinity if empty) - m2?: number // Welford's M2 for stddev/variance -} -``` - -### Change Detection - -On restart, definition hashes (FNV-1a 32-bit) are compared with the persisted hash. If a definition changed (different groupBy, metrics, or source), the aggregate state is reset and must be rebuilt. - -## Write-Time Update Flow - -When `brain.add(entity)` is called: - -``` -1. For each registered aggregate: - ├─ Source filter check (type, service, where) - │ └─ Skip if entity doesn't match - ├─ Aggregate entity check - │ └─ Skip if entity.service === 'brainy:aggregation' - │ or entity.metadata.__aggregate is set - ├─ Group key computation - │ └─ Extract groupBy fields from metadata - │ Apply time bucketing for windowed dimensions - └─ Metric update - └─ For each metric in the definition: - ├─ count: increment count - ├─ sum/avg: add value to sum, increment count - ├─ min/max: compare and update - └─ stddev/variance: Welford's online update -``` - -### Update Handling - -On `brain.update(entity)`, the engine reverses the old entity's contribution and applies the new entity's contribution. This correctly handles: - -- **Value changes**: old amount=10, new amount=20 — sum adjusts by +10 -- **Group key changes**: entity moves from category "food" to "drink" — both groups update -- **Source filter changes**: entity type changes from Event to Document — removed from matching aggregates - -### Delete Handling - -On `brain.remove(id)`, the engine reverses the entity's contribution: - -- `count` and `sum` are decremented -- `min`/`max` may become stale (marked in `staleMinMax` for lazy recompute) -- Welford's M2 is updated with the inverse formula -- Empty groups (all metric counts at zero) are removed - -## Algorithms - -### Welford's Online Algorithm - -Standard deviation and variance use Welford's numerically stable online algorithm with M2 tracking. This computes incrementally without storing individual values: - -``` -On add(x): - count += 1 - oldMean = (sum - x) / (count - 1) // mean before this value - sum += x - mean = sum / count // mean after this value - M2 += (x - oldMean) * (x - mean) - -On remove(x): - oldMean = sum / count - sum -= x - count -= 1 - newMean = sum / count - M2 = max(0, M2 - (x - oldMean) * (x - newMean)) - -Sample variance = M2 / (count - 1) -Sample stddev = sqrt(variance) -``` - -M2 is clamped to zero on remove to prevent floating-point drift from producing negative values. - -### MIN/MAX Handling - -The TypeScript engine uses simple comparison for add operations and marks MIN/MAX as potentially stale on delete (since removing the current min/max value requires a rescan). Stale values are lazily recomputed on the next query. - -The Cor native engine uses a `BTreeMap, u64>` that tracks the exact frequency of every value, providing precise MIN/MAX after any sequence of operations without rescanning. - -### Time Window Bucketing - -Timestamps (Unix milliseconds) are bucketed using UTC-based formatting: - -| Granularity | Bucket Key | Algorithm | -|------------|-----------|-----------| -| `hour` | `2024-01-15T14` | UTC year-month-day-hour | -| `day` | `2024-01-15` | UTC year-month-day | -| `week` | `2024-W03` | ISO 8601 week (Monday start, week 1 contains first Thursday) | -| `month` | `2024-01` | UTC year-month | -| `quarter` | `2024-Q1` | `ceil((month) / 3)` | -| `year` | `2024` | UTC year | -| `{ seconds: N }` | ISO timestamp | `floor(timestamp / interval) * interval` | - -Bucket keys can be parsed back into `{ start, end }` timestamp ranges via `parseBucketRange()`. - -## Provider Interface - -The `AggregationProvider` interface defines the contract between Brainy's `AggregationIndex` and plugin-provided native implementations: - -```typescript -interface AggregationProvider { - defineAggregate?(def: AggregateDefinition): void - removeAggregate?(name: string): void - - incrementalUpdate( - name: string, - def: AggregateDefinition, - entity: Record, - op: 'add' | 'update' | 'delete', - prev?: Record - ): AggregateGroupState[] - - computeGroupKey( - entity: Record, - groupBy: GroupByDimension[] - ): Record - - rebuildAggregate( - def: AggregateDefinition, - entities: Array> - ): Map - - queryAggregate( - state: Map, - params: AggregateQueryParams - ): AggregateResult[] - - restoreState?(data: string): void - serializeState?(): string -} -``` - -When a native provider is registered: - -1. `AggregationIndex` delegates `incrementalUpdate()` to the provider instead of running TypeScript logic -2. Provider returns updated `AggregateGroupState[]` which are applied back into the state maps -3. Query execution is delegated via `queryAggregate()` -4. State serialization is delegated via `serializeState()`/`restoreState()` - -Brainy retains ownership of the state maps and persistence. The provider handles computation. - -## Materialization - -The `AggregateMaterializer` converts aggregate group states into `NounType.Measurement` entities: - -1. When an aggregate group is updated and `materialize` is enabled, `scheduleMaterialize()` is called -2. Materialization is debounced (default: 1000ms) to batch rapid updates during ingestion -3. On trigger, the materializer either creates or updates a `NounType.Measurement` entity -4. Materialized entities include `service: 'brainy:aggregation'` and `metadata.__aggregate` to prevent infinite loops - -Materialized entities are automatically visible through: -- OData endpoints -- Google Sheets integration -- Server-Sent Events (SSE) -- Webhook notifications - -## Persistence - -### Storage Keys - -| Key | Content | -|-----|---------| -| `__aggregation_definitions__` | Array of all definitions with FNV-1a hashes | -| `__aggregation_state_{name}__` | Per-aggregate group states (array of `AggregateGroupState`) | -| `__aggregation_native_state__` | Serialized native provider state (JSON string) | - -### Lifecycle - -1. **`init()`** — Load definitions, compare hashes, load matching state, restore native provider state -2. **Write operations** — Mark modified aggregates as dirty -3. **`flush()`** — Persist all dirty aggregate states and native provider state -4. **`close()`** — Flush and release resources - -## Source Files - -| File | Purpose | -|------|---------| -| `src/aggregation/AggregationIndex.ts` | Core engine: definitions, state, write hooks, query | -| `src/aggregation/materializer.ts` | Debounced materialization of results as entities | -| `src/aggregation/timeWindows.ts` | Time bucketing and bucket range parsing | -| `src/aggregation/index.ts` | Module exports | -| `src/types/brainy.types.ts` | Type definitions for all aggregation interfaces | diff --git a/docs/architecture/augmentation-system-audit.md b/docs/architecture/augmentation-system-audit.md new file mode 100644 index 00000000..6429874b --- /dev/null +++ b/docs/architecture/augmentation-system-audit.md @@ -0,0 +1,359 @@ +# 🔍 Brainy 2.0 Augmentation System Architecture Audit (REVISED) + +**Author**: Senior Architecture Review +**Date**: 2025-08-25 +**Status**: 🟡 **WORKING BUT NEEDS MARKETPLACE FEATURES** + +## Executive Summary + +The augmentation system core execution is WORKING correctly through `AugmentationRegistry`. The system properly executes augmentations before/after operations. However, there's no discovery, installation, or marketplace integration for the brain-cloud registry vision. + +--- + +## 🟢 What's Actually Working + +### 1. Execution Mechanism ✅ +The `AugmentationRegistry` class properly implements: +```typescript +async execute(operation: string, params: any, mainOperation: () => Promise): Promise +``` +- Chains augmentations correctly +- Respects timing (before/after/around) +- Handles operation filtering +- Works with all 27 augmentations + +### 2. Registration System ✅ +```typescript +brain.augmentations.register(augmentation) +``` +- Two-phase initialization works (storage first) +- Context injection works +- Lifecycle management works + +### 3. Clean Interface ✅ +- 100% of augmentations use `BrainyAugmentation` +- `BaseAugmentation` provides solid foundation +- Proper TypeScript types + +### 4. Auto-Configuration ✅ +```typescript +new Brainy({ + cache: true, // Auto-registers CacheAugmentation + index: true, // Auto-registers IndexAugmentation + storage: 's3' // Auto-registers S3StorageAugmentation +}) +``` + +--- + +## 🟡 Missing for Marketplace Vision + +### 1. No Package Discovery +**Current**: Manual registration only +```typescript +// Current (manual) +import { NotionSynapse } from './my-custom-synapse' +brain.augmentations.register(new NotionSynapse()) + +// Needed +await brain.discover('notion') // Search npm/brain-cloud +await brain.install('@soulcraft/notion-synapse') +``` + +### 2. No Installation Mechanism +**Current**: Must be bundled at build time +**Needed**: Dynamic installation +```typescript +interface AugmentationMarketplace { + search(query: string): Promise + install(packageId: string): Promise + uninstall(packageId: string): Promise + listInstalled(): Promise + checkUpdates(): Promise +} +``` + +### 3. No Brain Cloud Registry Client +**Current**: No registry concept +**Needed**: Registry integration +```typescript +class BrainCloudRegistry { + private apiUrl = 'https://api.soulcraft.com/brain-cloud' + + async search(query: string): Promise { + const response = await fetch(`${this.apiUrl}/augmentations/search?q=${query}`) + return response.json() + } + + async getPackage(id: string): Promise { + const response = await fetch(`${this.apiUrl}/augmentations/${id}`) + return response.json() + } +} +``` + +### 4. No License Management +**Current**: All augmentations free/bundled +**Needed**: License verification +```typescript +interface LicenseManager { + verify(packageId: string, licenseKey: string): Promise + activate(packageId: string, licenseKey: string): Promise + deactivate(packageId: string): Promise + getStatus(packageId: string): Promise +} +``` + +### 5. No Version Management +**Current**: No versioning +**Needed**: Semver support +```typescript +interface VersionManager { + checkCompatibility(pkg: Package, brainyVersion: string): boolean + resolveConflicts(packages: Package[]): Package[] + upgrade(packageId: string, toVersion: string): Promise +} +``` + +--- + +## 📋 Implementation Plan for Marketplace + +### Phase 1: Local Package Discovery (1 week) +```typescript +class LocalPackageDiscovery { + async discover(): Promise { + // 1. Search node_modules for brainy augmentations + const packages = await glob('node_modules/@*/package.json') + + // 2. Filter for brainy augmentations + return packages.filter(pkg => pkg.brainy?.type === 'augmentation') + } + + async load(packageId: string): Promise { + // Dynamic import + const module = await import(packageId) + return new module.default() + } +} +``` + +### Phase 2: NPM Integration (1 week) +```typescript +class NPMRegistry { + async search(query: string): Promise { + // Search npm for packages with brainy keyword + const response = await fetch( + `https://registry.npmjs.org/-/v1/search?text=${query}+keywords:brainy-augmentation` + ) + return response.json() + } + + async install(packageId: string): Promise { + // Use npm programmatically + await exec(`npm install ${packageId}`) + + // Auto-register after install + const aug = await this.load(packageId) + this.brain.augmentations.register(aug) + } +} +``` + +### Phase 3: Brain Cloud Registry (2 weeks) +```typescript +class BrainCloudMarketplace { + private registry = new BrainCloudRegistry() + private licenses = new LicenseManager() + private installer = new AugmentationInstaller() + + async browse(category?: string): Promise { + const packages = await this.registry.list(category) + + return packages.map(pkg => ({ + ...pkg, + installed: this.isInstalled(pkg.id), + licensed: this.isLicensed(pkg.id), + updates: this.hasUpdates(pkg.id) + })) + } + + async purchase(packageId: string): Promise { + // 1. Process payment + const license = await this.processPayment(packageId) + + // 2. Activate license + await this.licenses.activate(packageId, license) + + // 3. Install package + await this.install(packageId) + } +} +``` + +### Phase 4: Developer Tools (1 week) +```typescript +// CLI for augmentation development +class AugmentationCLI { + async create(name: string): Promise { + // Scaffold new augmentation project + await this.scaffold(name, 'augmentation-template') + } + + async test(path: string): Promise { + // Test augmentation locally + const aug = await this.load(path) + await this.runTests(aug) + } + + async publish(path: string): Promise { + // Publish to brain-cloud + const pkg = await this.package(path) + await this.registry.publish(pkg) + } +} +``` + +--- + +## 🏗️ Recommended Architecture + +### 1. Augmentation Package Structure +```json +{ + "name": "@soulcraft/notion-synapse", + "version": "1.0.0", + "main": "dist/index.js", + "types": "dist/index.d.ts", + "brainy": { + "type": "augmentation", + "class": "NotionSynapse", + "timing": "after", + "operations": ["addNoun", "updateNoun"], + "priority": 20, + "license": "premium", + "price": 9.99, + "compatibility": ">=2.0.0", + "dependencies": [] + }, + "keywords": ["brainy-augmentation", "notion", "sync"] +} +``` + +### 2. Installation Flow +```typescript +// User flow +await brain.marketplace.search('notion') +// Returns: [@soulcraft/notion-synapse, @community/notion-sync, ...] + +await brain.marketplace.install('@soulcraft/notion-synapse') +// 1. Check license (prompt for purchase if needed) +// 2. Check compatibility +// 3. Install dependencies +// 4. Download package +// 5. Load augmentation +// 6. Register with brain +// 7. Initialize + +// Now it's working! +brain.augmentations.list() +// [..., { name: '@soulcraft/notion-synapse', enabled: true }] +``` + +### 3. Discovery UI +```typescript +// Web UI component + + + + + + + + + + + +``` + +--- + +## 🎯 Priority for 2.0 Release + +### Must Have (Release Blockers) +- ✅ Working execution (DONE) +- ✅ Clean interface (DONE) +- ✅ Documentation (DONE) +- ⏳ Fix augmentationPipeline.ts removal +- ⏳ Test all 27 augmentations work + +### Nice to Have (2.0.x) +- Local package discovery +- NPM integration +- Basic CLI tools + +### Future (2.1+) +- Brain Cloud Registry +- License management +- Payment processing +- Marketplace UI +- Developer portal + +--- + +## 📊 Current State Assessment + +| Component | Status | Notes | +|-----------|--------|-------| +| Core Execution | ✅ Working | AugmentationRegistry.execute() works | +| Registration | ✅ Working | Manual registration works | +| Auto-Config | ✅ Working | Cache, index, storage auto-register | +| Lifecycle | ✅ Working | Init, execute, shutdown work | +| Discovery | ❌ Missing | No package discovery | +| Installation | ❌ Missing | No dynamic installation | +| Marketplace | ❌ Missing | No registry client | +| Licensing | ❌ Missing | No license management | +| Versioning | ❌ Missing | No version checks | + +--- + +## 💡 Recommendations + +### For 2.0 Release +1. **Ship with manual registration** - It works! +2. **Document how to create augmentations** - Critical for adoption +3. **Create 2-3 example augmentations** - Show the patterns +4. **Add basic CLI for testing** - Help developers + +### For 2.1 (Q1 2025) +1. **Add NPM discovery** - Find installed augmentations +2. **Dynamic loading** - Import augmentations at runtime +3. **Basic marketplace API** - List available augmentations +4. **Version checking** - Ensure compatibility + +### For 3.0 (Q2 2025) +1. **Full marketplace** - Browse, search, install +2. **Payment integration** - Premium augmentations +3. **Developer portal** - Publish augmentations +4. **Enterprise features** - Private registries + +--- + +## ✅ Good News Summary + +The augmentation system WORKS! The core architecture is solid: +- Execution mechanism is correct +- Registration works +- Lifecycle management works +- All 27 augmentations function properly + +What's missing is the marketplace/discovery layer, which can be added incrementally without breaking the core system. The 2.0 release can ship with manual augmentation registration, and the marketplace features can be added in 2.1+. + +**Recommendation: Ship 2.0 with current system, add marketplace in 2.1** \ No newline at end of file diff --git a/docs/architecture/augmentations-actual.md b/docs/architecture/augmentations-actual.md index 81ace98d..d8e42dbe 100644 --- a/docs/architecture/augmentations-actual.md +++ b/docs/architecture/augmentations-actual.md @@ -73,10 +73,10 @@ import { MemoryStorageAugmentation } from 'brainy' ``` ### 11. Server Search Augmentation ✅ -Server-side search delegation over a conduit. +Distributed search capabilities. ```typescript import { ServerSearchConduitAugmentation } from 'brainy' -// Forwards queries to a remote Brainy server +// Distributed query execution ``` ### 12. Neural Import Augmentation ✅ @@ -100,7 +100,7 @@ await neuralImport.detectRelationships(entities) await neuralImport.generateInsights(data) ``` -### Operation Modes (Fully Implemented!) +### Distributed Operation Modes (Fully Implemented!) ```typescript // Read-only mode with optimized caching const readerMode = new ReaderMode() @@ -206,7 +206,7 @@ if (device === 'webgpu') { // CUDA detection in Node: if (device === 'cuda') { - // Future: GPU acceleration support + // Requires ONNX Runtime GPU packages } ``` @@ -239,10 +239,15 @@ const cacheConfig = await getCacheAutoConfig() ## 🎨 How to Use Hidden Features -### Enable Reader / Writer Modes +### Enable Distributed Modes ```typescript const brain = new Brainy({ - mode: 'reader' // or 'writer' or 'hybrid' + mode: 'reader', // or 'writer' or 'hybrid' + distributed: { + role: 'reader', + cacheStrategy: 'aggressive', + prefetch: true + } }) ``` @@ -280,7 +285,7 @@ const freshStats = await brain.getStatistics({ ## 📝 What Needs Documentation These features EXIST but need better docs: -1. Reader / writer operation modes +1. Distributed operation modes 2. Neural import full API 3. 3-level cache configuration 4. Performance monitoring API @@ -292,7 +297,7 @@ These features EXIST but need better docs: ## 💡 The Truth Brainy is MORE powerful than its own documentation suggests! Most "missing" features are actually implemented but hidden or not properly exposed. The codebase contains sophisticated systems for: -- Reader / writer operation modes +- Distributed operations - AI-powered import - Advanced caching - Performance monitoring diff --git a/docs/architecture/data-storage-architecture.md b/docs/architecture/data-storage-architecture.md index 48064757..3e56a1a6 100644 --- a/docs/architecture/data-storage-architecture.md +++ b/docs/architecture/data-storage-architecture.md @@ -1,388 +1,950 @@ -# Brainy Data Storage Architecture (8.0) +# Brainy Data Storage Architecture -**Complete on-disk reference for the 8.0 layout.** - -This document describes what a Brainy 8.0 data directory actually contains: the -canonical entity records, the system area, the generational-MVCC bookkeeping, -the column store, the blob area, and the lock files — plus how the in-memory -indexes rebuild from them. The authoritative design records are -[ADR-001 (generational MVCC)](../ADR-001-generational-mvcc.md) and -[index-architecture.md](./index-architecture.md); this document is the on-disk -map that ties them together. - -8.0 removed the 7.x copy-on-write subsystem (`_cow/`, `branches/{branch}/…` -paths) and the cloud/OPFS storage adapters. The two storage backends are -**filesystem** and **memory**; both speak the same path vocabulary (memory -storage keys its internal map by the identical path strings). +This document explains how Brainy stores, indexes, and scales data across all storage backends (GCS, S3, OPFS, filesystem, memory). --- -## 1. Directory Tree +## Table of Contents -A real 8.0 filesystem store (`path`, default `./brainy-data`): - -``` -brainy-data/ -│ -├── entities/ # Canonical records (current state) -│ ├── nouns/ -│ │ └── {shard}/ # 256 shards: first 2 hex chars of the UUID -│ │ └── {id}/ # One directory per entity UUID -│ │ ├── vectors.json.gz # Embedding + HNSW node state -│ │ └── metadata.json.gz # Everything else (type, subtype, data, fields, _rev) -│ └── verbs/ -│ └── {shard}/ -│ └── {id}/ -│ ├── vectors.json.gz # Relationship embedding (when present) -│ └── metadata.json.gz # sourceId, targetId, verb, subtype, weight, data… -│ -├── _system/ # System singletons + bucketed system keys -│ ├── generation.json(.gz) # { generation, updatedAt } — the write watermark -│ ├── manifest.json # { version, generation, … } — MVCC commit point -│ ├── tx-log.jsonl # One line per committed transact() batch (append-only) -│ ├── counts.json # Entity/verb totals -│ ├── type-statistics.json.gz # Per-NounType counts -│ ├── subtype-statistics.json.gz # Per-(NounType, subtype) counts -│ ├── verb-subtype-statistics.json.gz # Per-(VerbType, subtype) counts -│ ├── statistics.json # Aggregate statistics blob (counts, index sizes) -│ ├── hnsw-system.json # Vector-index entry point + max level -│ ├── __metadata_field_registry__.json.gz # Which metadata fields are indexed -│ ├── brainy:entityIdMapper.json.gz # UUID ↔ u64 mapping for native index providers -│ └── idx/ -│ └── {bucket}/ # 256 buckets: FNV-1a hash of the key -│ ├── __metadata_field_index__field_{name}.json.gz # Sparse field indexes -│ ├── __chunk__*.json.gz # Metadata-index roaring-bitmap chunks -│ ├── __sparse_index__*.json.gz# Zone maps + bloom filters -│ └── graph-lsm-verbs-{source|target}-*.json.gz # Graph LSM SSTables + manifest -│ -├── _generations/ # MVCC history (written ONLY by transact()) -│ └── {N}/ # One directory per committed generation N -│ ├── tx.json # The generation-N delta (immutable) -│ └── prev/ -│ └── {id}.json # Before-image of each touched record (immutable) -│ -├── _column_index/ # Column store manifests (one dir per field) -│ └── {field}/ -│ └── MANIFEST.json.gz # Run list + zone metadata for that column -│ -├── _blobs/ # Binary blob area (`.bin` convention) -│ ├── _column_index/ -│ │ └── {field}/ -│ │ └── L0-000001.bin # Column-store runs (level-0 segments) -│ └── … # VFS file content and other binary blobs -│ -└── locks/ # Process coordination (NEVER snapshotted) - ├── _writer.lock # Single-writer lock: pid, hostname, heartbeat - ├── _flush_requests/ # Reader→writer flush RPC (.req files) - └── _flush_responses/ # Writer acks (.ack files) -``` - -Most JSON objects are gzip-compressed (`.json.gz`) — compression is on by -default for filesystem storage (`storage.options.compression`, zlib level 6). -A few hot singletons (`manifest.json`, `counts.json`, `hnsw-system.json`, -`tx-log.jsonl`) are written uncompressed for cheap partial reads and appends. +1. [What Gets Stored](#1-what-gets-stored) +2. [The Indexes](#2-the-indexes) +3. [Sharding Strategy](#3-sharding-strategy) +4. [Storage Layout](#4-storage-layout) --- -## 2. Canonical Entity Records +## 1. What Gets Stored -Each entity (noun) and relationship (verb) is **two files** under one -ID-first directory. The split keeps vector I/O (large, append-mostly) separate -from metadata I/O (small, read-heavy). +Brainy stores three types of data, with each type split across multiple files for optimal performance. -### Noun vector file — `entities/nouns/{shard}/{id}/vectors.json.gz` +### 1.1 Entities (Nouns) +Each entity is stored in **2 files**: + +#### Vector File ```json { - "id": "421d92e7-4241-470a-80f4-4b39414e7a83", - "vector": [-0.139, -0.056, 0.028, "…384 dims…"], - "connections": { "0": ["neighbor-uuid…"] }, - "level": 0 + "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", + "vector": [0.1, 0.2, 0.3, ...], + "connections": { + "0": ["uuid1", "uuid2"], + "1": ["uuid3"] + }, + "level": 2 } ``` -The HNSW node state (`connections`, `level`) is persisted with the vector so -the vector index can rebuild without recomputing the graph. - -### Noun metadata file — `entities/nouns/{shard}/{id}/metadata.json.gz` +**Purpose:** HNSW graph navigation for semantic search +**Size:** ~4KB per entity +**Scale:** Millions of entities +**Location:** `entities/nouns/vectors/{shard}/{uuid}.json` +#### Metadata File ```json { - "data": "React is a JavaScript library for building user interfaces", - "noun": "concept", - "subtype": "cli-add", - "createdAt": 1781198053726, - "updatedAt": 1781198053726, - "_rev": 1 + "type": "user", + "status": "active", + "name": "John Doe", + "email": "john@example.com", + "createdAt": {...}, + "customField": "custom value" } ``` -- `noun` is the NounType. **Type lives in metadata, not in the path** — lookup - by ID is a single path construction, no type needed. -- `subtype` is the per-product sub-classification (required on write by - default in 8.0). -- `_rev` increments on every write and backs `update({ ifRev })` CAS. -- Consumer metadata fields sit alongside the standard ones. +**Purpose:** Business data and filtering +**Size:** ~1-10KB per entity +**Scale:** Millions of entities +**Location:** `entities/nouns/metadata/{shard}/{uuid}.json` -### Verb files — `entities/verbs/{shard}/{id}/…` +--- -Same two-file split. The verb metadata record carries the graph edge: -`sourceId`, `targetId`, `verb` (VerbType), `subtype`, `weight`, `data`, -`metadata`, timestamps, `_rev`. Verb IDs are Brainy-generated UUIDs by -contract (8.0 rejects caller-supplied verb ids) because native graph providers -intern the raw UUID bytes as u64 handles. +### 1.2 Relationships (Verbs) -### Path construction +Each relationship is also stored in **2 files**: + +#### Vector File +```json +{ + "id": "7b2f5e3c-8d4a-4f1e-9c2b-5a6d7e8f9a0b", + "vector": [0.5, 0.3, 0.7, ...], + "connections": { + "0": ["verb-uuid1", "verb-uuid2"] + } +} +``` + +**Purpose:** Relationship similarity for semantic graph queries +**Size:** ~2KB per relationship +**Scale:** Millions of relationships +**Location:** `entities/verbs/vectors/{shard}/{uuid}.json` + +#### Metadata File +```json +{ + "sourceId": "user-uuid", + "targetId": "product-uuid", + "type": "purchased", + "weight": 1.0, + "timestamp": {...}, + "metadata": { + "amount": 99.99, + "quantity": 2 + } +} +``` + +**Purpose:** Graph structure (edges) and relationship data +**Size:** ~500 bytes per relationship +**Scale:** Millions of relationships +**Location:** `entities/verbs/metadata/{shard}/{uuid}.json` + +--- + +### 1.3 System Metadata + +Unlike entities and relationships, system metadata consists of **index files** that enable fast lookups without scanning millions of entities. + +**Purpose:** Fast filtering and range queries +**Scale:** 10-200 files total (NOT per-entity!) +**Location:** `_system/` (no sharding) + +Examples: +- `__metadata_field_index__status.json` - Maps status values to entity IDs +- `__metadata_sorted_index__createdAt.json` - Sorted list for range queries +- `statistics.json` - Global statistics + +--- + +## 2. The Indexes + +Brainy uses three complementary index systems for different query patterns. + +### 2.1 HNSW Vector Index (In-Memory with Lazy Loading) + +**Purpose:** Semantic similarity search +**Location:** RAM (rebuilt from storage on startup) +**Data Structure:** Hierarchical graph of vector connections + +**Example Query:** +```typescript +// Find entities similar to a vector +const results = await brain.searchByVector([0.1, 0.2, 0.3, ...], { k: 10 }) +// Returns: [{id: "uuid1", score: 0.95}, {id: "uuid2", score: 0.89}, ...] +``` + +**How It Works:** +1. Loads `entities/nouns/vectors/**/*.json` files +2. Builds HNSW graph structure in memory +3. Enables O(log n) approximate nearest neighbor search +4. Vectors loaded on-demand in lazy mode (zero configuration) + +**Performance:** +- Build time: 1-5 seconds per 100K entities +- Query time: 1-10ms for k=10 results (standard mode) +- Query time: 2-15ms for k=10 results (lazy mode, with cache) +- Memory (standard): ~200MB per 100K entities (all vectors loaded) +- Memory (lazy mode): ~50MB per 100K entities (graph only, vectors on-demand) + +--- + +#### Universal Lazy Mode (v3.36.0+) + +**Zero-Configuration Memory Management** + +Brainy's HNSW index automatically adapts to available memory by enabling lazy mode when vectors don't fit in the UnifiedCache. + +**Standard Mode vs. Lazy Mode:** + +| Mode | Graph Structure | Vectors | Memory | Performance | +|------|----------------|---------|--------|-------------| +| **Standard** | In memory | In memory | High (~1.5KB/vector) | Fastest (1-10ms) | +| **Lazy** | In memory | On-demand | Low (~24 bytes/node) | Fast (2-15ms) | + +**Auto-Detection Logic (v3.36.0+):** +```typescript +// Step 1: Reserve embedding model memory (NEW in v3.36.0) +const modelMemory = 150 * 1024 * 1024 // Q8: 150MB (default), FP32: 250MB + +// Step 2: Calculate available memory AFTER model reservation +const availableForCache = systemMemory - modelMemory + +// Step 3: Allocate UnifiedCache from available memory +// Environment-aware allocation (NEW in v3.36.0): +// - Development: 25% of availableForCache +// - Container: 40% of availableForCache +// - Production: 50% of availableForCache +const unifiedCacheSize = availableForCache × allocationRatio + +// Step 4: Calculate HNSW allocation within UnifiedCache +const estimatedVectorMemory = entityCount × 1536 // 384 dims × 4 bytes +const hnswAvailableCache = unifiedCacheSize × 0.30 // 30% for HNSW + +// Step 5: Auto-enable lazy mode if vectors exceed cache +if (estimatedVectorMemory > hnswAvailableCache) { + lazyMode = true // Vectors loaded on-demand +} else { + lazyMode = false // Vectors fully loaded +} +``` + +**Example Output (2GB system, 100K entities):** +``` +Model Memory: 150MB Q8 reserved (22MB weights + 30MB runtime + 98MB workspace) +Available for Cache: 1.85GB (2GB - 150MB model) +UnifiedCache Size: 400MB (25% development allocation) +HNSW Allocation: 120MB (30% of 400MB) + +✓ HNSW: Auto-enabled lazy mode for 100,000 vectors + (146.5MB > 120MB cache) +``` + +**Example Output (16GB production, 100K entities):** +``` +Model Memory: 150MB Q8 reserved +Available for Cache: 15.85GB (16GB - 150MB model) +UnifiedCache Size: 7.92GB (50% production allocation) +HNSW Allocation: 2.38GB (30% of 7.92GB) + +✓ HNSW: Standard mode for 100,000 vectors + (146.5MB fits in 2.38GB cache) +``` + +**How Lazy Mode Works:** + +1. **Graph Loading (O(N))**: Loads only the HNSW graph structure + - Node IDs and connections: ~24 bytes per node + - Total: ~2.4MB for 100K entities + +2. **On-Demand Vectors**: Loads vectors during search operations + - Cache key: `hnsw:vector:{id}` + - Storage fallback: `storage.getNounVector(id)` + - Batch preloading: Parallel loads before distance calculations + +3. **Fair Competition**: Shares UnifiedCache with Graph and Metadata indexes + - Cost-aware eviction: `accessCount / rebuildCost` + - Fairness monitoring: Prevents any index from hogging cache + - 30% cache allocation for HNSW vectors + +**Monitoring Lazy Mode:** +```typescript +// Get comprehensive statistics +const stats = brain.hnswIndex.getCacheStats() + +console.log(stats) +// { +// lazyModeEnabled: true, +// autoDetection: { +// entityCount: 100000, +// estimatedVectorMemoryMB: 146.48, +// availableCacheMB: 600.0, +// threshold: 0.3, +// decision: "Lazy mode enabled (vectors > cache threshold)" +// }, +// unifiedCache: { +// hits: 45230, +// misses: 12450, +// hitRatePercent: 78.42, +// evictions: 3200 +// }, +// hnswCache: { +// vectorsInCache: 8450, +// estimatedMemoryMB: 12.35 +// }, +// fairness: { +// hnswAccessPercent: 32.5, +// fairnessViolation: false +// }, +// recommendations: [ +// "All metrics healthy - no action needed" +// ] +// } +``` + +**Performance Characteristics:** + +**Memory Usage:** +``` +Standard mode (100K entities): + - Vectors: 146.5MB (100K × 1536 bytes) + - Graph: 2.4MB (100K × 24 bytes) + - Total: ~149MB + +Lazy mode (100K entities): + - Vectors: 0MB (loaded on-demand) + - Graph: 2.4MB (always in memory) + - Cache: 12-30MB (frequently accessed vectors) + - Total: ~15-33MB (5-10x less memory) +``` + +**Query Performance:** +``` +Standard mode: + - All vectors in memory: 1-10ms per query + - No I/O overhead + +Lazy mode (with 80% cache hit rate): + - Cache hits: 2-8ms per query (20% overhead) + - Cache misses: 5-15ms per query (disk I/O) + - Average: 2-10ms per query (acceptable overhead) +``` + +**Optimization Techniques:** + +1. **Batch Preloading**: Loads all candidate vectors in parallel before distance calculations + ```typescript + // Before comparing distances, preload all candidates + await preloadVectors([node1.id, node2.id, node3.id, ...]) + // Then calculate distances (all vectors now in cache) + ``` + +2. **Request Coalescing**: UnifiedCache prevents stampede on parallel requests + ```typescript + // Multiple requests for same vector → single storage call + Promise.all([ + getVector(id), // Request 1 + getVector(id), // Request 2 (coalesced) + getVector(id) // Request 3 (coalesced) + ]) + ``` + +3. **Cost-Aware Eviction**: Keeps frequently accessed vectors in cache + ```typescript + // UnifiedCache scores items: accessCount / rebuildCost + // High access + low rebuild cost = stays in cache + // Low access + high rebuild cost = evicted + ``` + +**When Lazy Mode Activates:** + +With default 2GB UnifiedCache (600MB allocated to HNSW): + +| Entity Count | Vector Memory | Mode | Memory Savings | +|-------------|---------------|------|----------------| +| 10K | 14.6MB | Standard | N/A | +| 100K | 146.5MB | Standard | N/A | +| 400K | 586MB | Standard | N/A | +| 500K | 732MB | **Lazy** | 5-10x | +| 1M | 1.46GB | **Lazy** | 10-20x | +| 10M | 14.6GB | **Lazy** | 50-100x | + +**Troubleshooting:** + +**Low Cache Hit Rate (<50%)** +```typescript +// Symptom: Slow queries despite lazy mode +const stats = brain.hnswIndex.getCacheStats() +if (stats.unifiedCache.hitRatePercent < 50) { + // Solution: Increase UnifiedCache size + brain = new Brainy({ + cacheSize: 4 * 1024 * 1024 * 1024 // 4GB (default: 2GB) + }) +} +``` + +**Fairness Violation** +```typescript +// Symptom: HNSW using >90% cache with <10% access +const stats = brain.hnswIndex.getCacheStats() +if (stats.fairness.fairnessViolation) { + // Solution: Adjust rebuild costs for better competition + // (This is automatic - violation triggers rebalancing) +} +``` + +**Force Lazy Mode (Testing Only)** +```typescript +// Override auto-detection (not recommended for production) +await brain.rebuildIndexes({ + hnsw: { + lazy: true // Force lazy mode regardless of memory + } +}) +``` + +--- + +### 2.2 Graph Adjacency Index (In-Memory) + +**Purpose:** Navigate relationships (graph queries) +**Location:** RAM (rebuilt from storage on startup) +**Data Structure:** Bidirectional mappings ```typescript -const shard = id.substring(0, 2) // '42' -const metadataPath = `entities/nouns/${shard}/${id}/metadata.json` -const vectorsPath = `entities/nouns/${shard}/${id}/vectors.json` -// Verbs: same shape under entities/verbs/ +{ + sourceToTargets: Map>, // "user-uuid" → ["product1", "product2"] + targetToSources: Map> // "product1" → ["user1", "user2"] +} ``` -Implemented in `src/storage/baseStorage.ts` (path generators) and -`src/storage/sharding.ts` (`getShardIdFromUuid`). +**Example Query:** +```typescript +// Find all products purchased by a user +const verbs = await brain.getVerbsBySource("user-uuid") + +// Find all users who purchased a product +const verbs = await brain.getVerbsByTarget("product-uuid") +``` + +**How It Works:** +1. Loads `entities/verbs/metadata/**/*.json` files +2. Builds bidirectional index in memory +3. Enables O(1) relationship lookups + +**Performance:** +- Build time: 0.5-2 seconds per 100K relationships +- Query time: <1ms +- Memory: ~100MB per 100K relationships --- -## 3. The `_system/` Area +### 2.3 Metadata Field Indexes (On-Disk) -Two kinds of keys live here, resolved by `BaseStorage.parsePath()`: +**Purpose:** Filter by business fields without loading all entities +**Location:** Persistent storage +**Data Structure:** Field → Value → IDs mapping -1. **Singletons** — well-known keys written at `_system/.json`: - `counts`, `statistics`, `type-statistics`, `hnsw-system`, - `__metadata_field_registry__`, `brainy:entityIdMapper`, plus the MVCC - trio (`generation.json`, `manifest.json`, `tx-log.jsonl`). -2. **Bucketed system keys** — everything else (field indexes, bitmap chunks, - sparse-index segments, graph LSM SSTables) hashes into one of 256 - `_system/idx/{bucket}/` directories via FNV-1a, so no single directory - accumulates unbounded entries. +#### Hash Indexes (Exact Match) +```json +// _system/__metadata_field_index__status.json +{ + "values": { + "active": 800000, // Count + "pending": 150000, + "deleted": 50000 + }, + "lastUpdated": "2025-10-09T..." +} +``` -Notable singletons: +**Example Query:** +```typescript +// Find all active users +const users = await brain.getNouns({ + filter: { + metadata: { status: 'active' } + } +}) +``` -| File | Contents | -|------|----------| -| `generation.json` | `{ generation, updatedAt }` — monotonic watermark, bumped by **every** write batch | -| `manifest.json` | MVCC commit point: highest *committed* generation (see §4) | -| `tx-log.jsonl` | One JSON line per committed `transact()` batch: generation, timestamp, `meta` | -| `type-statistics.json.gz` | Per-NounType counts (backs `brain.counts.byType`) | -| `subtype-statistics.json.gz` | `{ counts: { [type]: { [subtype]: n } }, updatedAt }` (contract-bound shape) | -| `verb-subtype-statistics.json.gz` | Same shape for VerbTypes | -| `hnsw-system.json` | `{ entryPointId, maxLevel }` — per-node state lives in each entity's `vectors.json` | -| `brainy:entityIdMapper.json.gz` | UUID ↔ u64 interning table for the BigInt provider contract | -| `__metadata_field_registry__.json.gz` | Registry of indexed metadata field names | +**How It Works:** +1. Query checks `__metadata_field_index__status.json` +2. Retrieves IDs for "active" status +3. Loads only matching entity files +4. Returns: ~1000 IDs in 5ms (vs scanning 1M entities) --- -## 4. Generational MVCC (`_generations/` + the `_system` trio) +#### Sorted Indexes (Range Queries) -Full design in [ADR-001](../ADR-001-generational-mvcc.md). The on-disk shape: - -``` -_system/generation.json { generation, updatedAt } atomic tmp+rename -_system/manifest.json { version, generation, … } atomic tmp+rename — THE commit point -_system/tx-log.jsonl one line per committed transact() append-only -_generations/{N}/tx.json the generation-N delta immutable once written -_generations/{N}/prev/{id}.json before-image of {id} immutable once written +```json +// _system/__metadata_sorted_index__createdAt.json +{ + "values": [ + [1704067200000, ["uuid1", "uuid2", "uuid3"]], // Jan 1, 2024 + [1704153600000, ["uuid4", "uuid5"]], // Jan 2, 2024 + [1704240000000, ["uuid6"]] // Jan 3, 2024 + ], + "fieldType": "number" +} ``` -Commit protocol (writer side): stage before-images and the delta under -`_generations/N/`, fsync, apply the delta to the canonical `entities/…` -records, then atomically rename `manifest.json` to publish generation N. The -tx-log line is appended last (advisory). Crash recovery on open discards any -`_generations/{N}` newer than the manifest. +**Example Query:** +```typescript +// Find entities created after Jan 1, 2024 +const recent = await brain.getNouns({ + filter: { + metadata: { + createdAt: { greaterThan: 1704067200000 } + } + } +}) +``` -Two write classes share the generation clock: - -- **Single-operation writes** (`add`/`update`/`remove`/`relate` outside - `transact()`) bump `generation.json` so watermarks and `_rev` CAS stay sound, - but write **no** history — they are not visible to `db.since()` and remain - visible through earlier pins. -- **`transact()` batches** write the full `_generations/{N}` record and a - tx-log line, and are the unit of time travel (`brain.asOf()`). - -Snapshots (`db.persist(path)`) hard-link the entire store **except `locks/`** -into a self-contained directory openable via `Brainy.load(path)`. +**How It Works:** +1. Binary search sorted index (O(log n) where n = unique values) +2. Returns matching IDs +3. Loads only matching entities +4. Performance: Find 1000 entities in 10ms (from 1M total) --- -## 5. Column Store (`_column_index/` + `_blobs/_column_index/`) +### 2.4 Adaptive Memory Management (3-Tier Cache) -The metadata index persists per-field columnar runs for O(log n) range and -membership queries at scale: +Brainy uses a smart 3-tier caching system to balance performance and memory usage, automatically adapting to available resources. -- `_column_index/{field}/MANIFEST.json.gz` — the run list and zone metadata - for one field (`createdAt`, `subtype`, `noun`, `_rev`, consumer fields, - `__words__` for tokenized text…). -- `_blobs/_column_index/{field}/L0-NNNNNN.bin` — the actual level-0 run - segments, stored through the shared `_blobs/.bin` binary convention. +**Architecture:** Hot Cache → Warm Cache → Cold Storage -Sparse per-field indexes, roaring-bitmap chunks, and zone-map/bloom segments -additionally live as bucketed keys under `_system/idx/` (see §3). Which path -serves a given `where` clause is the query planner's decision — inspect it -with `brainy inspect explain

--where '…'`. +```typescript +{ + hot: { + type: 'LRU Cache', + location: 'Memory', + access: 'Instant (<1ms)', + size: 'Small (most recent items)' + }, + warm: { + type: 'TTL Cache', + location: 'Memory', + access: 'Fast (1-5ms)', + size: 'Medium (frequently accessed)' + }, + cold: { + type: 'Persistent Storage', + location: 'Disk/Cloud', + access: 'Slower (10-150ms)', + size: 'Unlimited (all data)' + } +} +``` + +#### How It Works + +**1. Hot Cache (LRU - Least Recently Used)** +- Stores most recently accessed items +- Ultra-fast lookups (<1ms) +- Automatically evicts least-used items when full +- Default size: 1,000 - 10,000 items + +**2. Warm Cache (TTL - Time To Live)** +- Stores frequently accessed items +- Fast lookups (1-5ms) +- Items expire after inactivity period +- Default TTL: 5-30 minutes + +**3. Cold Storage (Persistent)** +- All data stored on disk/cloud +- Retrieved on cache miss +- Automatically promoted to warm/hot on access +- No size limit + +#### Adaptive Behavior + +The cache automatically adjusts based on memory pressure: + +```typescript +// Low memory: Aggressive eviction +hot.maxSize = 1,000 +warm.ttl = 5 minutes + +// High memory: Generous caching +hot.maxSize = 10,000 +warm.ttl = 30 minutes +``` + +#### Cache Flow Example + +```typescript +// First access: Miss all caches +await brain.getNoun(id) +// → Cold storage (150ms) +// → Promoted to warm + hot + +// Second access: Hot cache hit +await brain.getNoun(id) +// → Hot cache (<1ms) + +// After 10 minutes: Hot evicted, warm hit +await brain.getNoun(id) +// → Warm cache (2ms) +// → Promoted to hot + +// After 1 hour: All caches expired +await brain.getNoun(id) +// → Cold storage (150ms) +// → Cycle repeats +``` + +#### Performance Impact + +| Cache Level | Hit Rate | Latency | Memory per 100K Items | +|-------------|----------|---------|----------------------| +| **Hot (LRU)** | 60-80% | <1ms | ~200MB | +| **Warm (TTL)** | 15-30% | 1-5ms | ~100MB | +| **Cold (Disk)** | 5-10% | 10-150ms | 0MB (disk only) | + +**Combined Performance:** +- 90%+ requests served from memory +- Average latency: 1-2ms +- Memory usage scales with working set, not total data size + +#### What Gets Cached + +**HNSW Vector Index:** +- Vector data cached in hot/warm tiers +- Graph connections cached separately +- Adaptive loading based on query patterns + +**Graph Adjacency Index:** +- Relationship maps cached in warm tier +- Most-used relationships in hot tier +- Full graph in cold storage + +**Metadata Indexes:** +- Field indexes loaded on demand +- Frequently queried indexes stay in warm tier +- Large indexes partially cached --- -## 6. Blob Area (`_blobs/`) +## 3. Sharding Strategy -`_blobs/.bin` is the flat binary-blob convention shared by every storage -adapter (`saveBinaryBlob`/`getBinaryBlob` in the storage contract): +Sharding splits data into 256 buckets for optimal storage performance. -- **VFS file content** — VFS entities are regular nouns (path, ownership, and - timestamps in entity metadata); the file *bytes* are blobs. -- **Column-store runs** (under the `_column_index/` key prefix, §5). -- Any other binary payload an index provider persists. +### 3.1 Why Shard? -Writes use unique temp names + rename, so concurrent writers of the same key -cannot tear each other's blobs. +**Cloud Storage Limitations:** +- GCS/S3: Listing 100K files in one directory = 10-30 seconds +- GCS/S3: Max recommended files per directory = 1,000-10,000 +- Network: Parallel operations faster than sequential + +**Solution:** Split into 256 shards = ~3,900 files per shard --- -## 7. Locks (`locks/`) +### 3.2 How Sharding Works + +**Algorithm:** Extract first 2 hex characters from UUID ``` -locks/_writer.lock # single-writer lock: { pid, hostname, startedAt, heartbeat, version } -locks/_flush_requests/ # readers drop .req to ask the writer to flush -locks/_flush_responses/ # writer answers with .ack +UUID: 3fa85f64-5717-4562-b3fc-2c963f66afa6 + ^^ +Shard: 3f ``` -- One **writer** per data directory, enforced at `init()`; stale locks (dead - PID / stale heartbeat) are reclaimed automatically. -- Read-only processes (`Brainy.openReadOnly()`, the `brainy inspect` CLI - family) can ask the live writer to flush via the request/response files, so - out-of-process diagnostics see fresh state. -- `locks/` is excluded from snapshots (`SNAPSHOT_EXCLUDED_TOP_DIRS` in - `src/storage/adapters/fileSystemStorage.ts`). - ---- - -## 8. In-Memory Indexes and What Rebuilds From What - -| Index | In memory | Persisted state | Rebuild source | -|-------|-----------|-----------------|----------------| -| **Vector (HNSW)** | Graph of vector connections | `_system/hnsw-system.json` + per-entity `vectors.json` | Walk entity vector files; lazy mode loads structure only and pages vectors on demand | -| **Metadata index** | Field → value bitmaps + column-store readers | `_system/idx/` chunks + `_column_index/` manifests + `_blobs/_column_index/` runs | Loaded directly; full rebuild re-scans entity metadata | -| **Graph adjacency** | sourceId/targetId → verb-id LSM trees | `graph-lsm-verbs-{source,target}-*` SSTables under `_system/idx/` | Loaded from SSTables; full rebuild re-scans verb metadata | -| **Counts/statistics** | Per-type and per-subtype maps | `_system/{type,subtype,verb-subtype}-statistics.json.gz`, `counts.json` | Recomputable by scanning entities (`brainy inspect repair`) | - -A pluggable index provider (the 8.0 plugin contract in -`@soulcraft/brainy/plugin`) may replace any of the JS implementations; the -persisted formats above are contract-bound so JS and native implementations -can interleave on the same directory. - ---- - -## 9. Sharding Strategy - -**Entities:** first 2 hex characters of the UUID → 256 uniform shards. -Deterministic, configuration-free, and keeps per-directory entry counts low -(at 1M entities: ~3,900 directories per shard). Paginated whole-store walks -(`getNouns`/`getVerbs`) iterate shards `00`–`ff` in order. - -**System keys:** FNV-1a hash of the key → 256 `_system/idx/` buckets. Same -motivation, different keyspace (system keys are not UUIDs). - -**What is never sharded:** the `_system/` singletons, `_generations/{N}` -directories (keyed by generation number), `_column_index/{field}` manifests -(keyed by field name), and `locks/`. - ---- - -## 10. Durability and Atomicity - -- **Per-object atomicity:** every JSON object and blob is written to a unique - temp file then `rename()`d — readers never observe torn objects. -- **Transaction atomicity:** the `manifest.json` rename is the single commit - point for `transact()` batches (§4); everything staged before it is - discarded by crash recovery if the rename never lands. -- **Compression:** gzip per object (`.json.gz`), transparent to all readers. - Native index providers that mmap binary formats use the uncompressed - `_blobs/` area instead. - ---- - -## 11. `clear()` Semantics - -`brain.clear()` removes all entities, relationships, indexes, statistics, and -MVCC history, then re-resolves every index exactly as `init()` does — -including plugin-provided vector/metadata/id-mapper factories and VFS root -re-creation. The data directory afterwards contains a fresh, empty store (the -writer lock remains held by the running process). - ---- - -## 12. Common Scenarios - -### Adding an entity +**Properties:** +- **Deterministic:** Same UUID always maps to same shard +- **Uniform:** UUIDs distribute evenly across shards +- **Predictable:** Easy to compute, no randomness +- **Efficient:** Simple string operation (O(1)) +**Shard Distribution (1M entities):** ``` -brain.add({ data, type, subtype }) -1. Generate UUID → shard = first 2 hex chars -2. Embed data → 384-dim vector -3. Write entities/nouns/{shard}/{id}/vectors.json.gz (vector + HNSW node state) -4. Write entities/nouns/{shard}/{id}/metadata.json.gz (type/subtype/data/fields, _rev: 1) -5. Update in-memory indexes (HNSW insert, metadata index, statistics) -6. Bump _system/generation.json (no _generations/ entry — single-op write) -``` - -### Committing a transaction - -``` -await brain.transact(tx => { tx.add(…); tx.update(…) }) -1. Stage _generations/{N}/prev/{id}.json before-images + tx.json delta; fsync -2. Apply the delta to canonical entities/… records -3. Atomic-rename _system/manifest.json → generation N is committed -4. Append one line to _system/tx-log.jsonl -``` - -### Cold start - -``` -await brain.init() -1. Acquire locks/_writer.lock (or open read-only) -2. Crash recovery: drop _generations/{N} newer than manifest.json -3. Load _system singletons (counts, statistics, field registry, id mapper) -4. Vector index: hnsw-system.json + entity vectors.json (lazy mode if large) -5. Graph adjacency: load LSM SSTables from _system/idx/ -6. Metadata index: column-store manifests + bitmap chunks on demand -``` - -### Snapshot and restore - -``` -const db = brain.now(); await db.persist('/backups/today'); await db.release() -→ hard-links everything except locks/ into a self-contained directory - -await Brainy.load('/backups/today') // open snapshot read-only as a Db -await brain.restore('/backups/today', { confirm: true }) // replace store state +Shard 00: ~3,900 entities +Shard 01: ~3,900 entities +... +Shard fe: ~3,900 entities +Shard ff: ~3,900 entities +Total: 256 shards × 3,900 = ~1,000,000 entities ``` --- -## 13. Summary +### 3.3 When to Shard vs. Not Shard -- **Two backends** (filesystem, memory), one path vocabulary. -- **Two files per entity** under ID-first `entities/{kind}/{shard}/{id}/`. -- **Type and subtype are metadata**, not directory structure; type queries go - through the metadata index, not the filesystem. -- **`_system/`** holds singletons plus 256 hash buckets of index state. -- **`_generations/` + `manifest.json` + `tx-log.jsonl`** implement - generational MVCC. History is per-write: every `add()`/`update()`/`remove()`/ - `relate()` gets its own generation, and `transact()` groups several ops into - one atomic generation. (Single-op retention has been the model since 8.0; - you never need to route a write through `transact()` just to keep its history.) -- **`_column_index/` + `_blobs/`** hold the columnar metadata runs and binary - blobs (VFS content included). -- **`locks/`** coordinates the single writer and reader flush requests, and - never travels with snapshots. +| Data Type | Shard? | Why? | +|-----------|--------|------| +| **Entity vectors** | ✅ Yes | Millions of files | +| **Entity metadata** | ✅ Yes | Millions of files | +| **Verb vectors** | ✅ Yes | Millions of files | +| **Verb metadata** | ✅ Yes | Millions of files | +| **System metadata** | ❌ No | Only 10-200 files | +| **Statistics** | ❌ No | Single file | +| **Indexes** | ❌ No | 10-100 files | + +**Key Principle:** Shard by **entity UUID**, not by key type. + +--- + +### 3.4 Performance Impact + +**Without Sharding (1M entities):** +``` +List directory: 30 seconds +Find entity: 30 seconds (must list first) +Delete entity: 30 seconds (must list first) +``` + +**With Sharding (1M entities across 256 shards):** +``` +List directory: 120ms (only ~3,900 files) +Find entity: 150ms (list shard + download) +Delete entity: 150ms (list shard + delete) +``` + +**Speedup:** 200x faster for large datasets + +--- + +## 4. Storage Layout + +Complete directory structure for all storage backends. + +### 4.1 Full Directory Tree + +``` +storage-root/ +│ +├── entities/ +│ ├── nouns/ +│ │ ├── vectors/ [SHARDED] +│ │ │ ├── 00/ +│ │ │ │ ├── 00123456-1234-5678-9abc-def012345678.json +│ │ │ │ ├── 00abcdef-1234-5678-9abc-def012345678.json +│ │ │ │ └── ... (~3,900 files) +│ │ │ ├── 01/ +│ │ │ │ └── ... (~3,900 files) +│ │ │ ├── 02/ - fe/ ... +│ │ │ └── ff/ +│ │ │ └── ... (~3,900 files) +│ │ │ +│ │ └── metadata/ [SHARDED] +│ │ ├── 00/ +│ │ │ ├── 00123456-1234-5678-9abc-def012345678.json +│ │ │ └── ... (~3,900 files) +│ │ ├── 01/ - fe/ ... +│ │ └── ff/ +│ │ +│ └── verbs/ +│ ├── vectors/ [SHARDED] +│ │ ├── 00/ +│ │ │ └── ... (~3,900 files) +│ │ ├── 01/ - fe/ ... +│ │ └── ff/ +│ │ +│ └── metadata/ [SHARDED] +│ ├── 00/ +│ │ └── ... (~3,900 files) +│ ├── 01/ - fe/ ... +│ └── ff/ +│ +└── _system/ [NOT SHARDED] + ├── __metadata_field_index__status.json + ├── __metadata_field_index__type.json + ├── __metadata_sorted_index__createdAt.json + ├── __metadata_sorted_index__updatedAt.json + ├── statistics.json + └── counts.json +``` + +--- + +### 4.2 File Count Breakdown (1M Entities Example) + +| Directory | File Count | Size per File | Total Size | +|-----------|-----------|---------------|------------| +| `entities/nouns/vectors/**` | 1,000,000 | ~4KB | ~4GB | +| `entities/nouns/metadata/**` | 1,000,000 | ~2KB | ~2GB | +| `entities/verbs/vectors/**` | 1,000,000 | ~2KB | ~2GB | +| `entities/verbs/metadata/**` | 1,000,000 | ~500B | ~500MB | +| `_system/**` | ~50-200 | ~1-500KB | ~5-10MB | +| **Total** | **~4,000,100** | | **~8.5GB** | + +--- + +### 4.3 Storage Backend Mapping + +All storage backends follow the same structure: + +#### Google Cloud Storage (GCS) +``` +gs://my-bucket/ + ├── entities/nouns/vectors/00/00123456-uuid.json + ├── entities/nouns/metadata/00/00123456-uuid.json + └── _system/__metadata_field_index__status.json +``` + +#### AWS S3 / MinIO +``` +s3://my-bucket/ + ├── entities/nouns/vectors/00/00123456-uuid.json + ├── entities/nouns/metadata/00/00123456-uuid.json + └── _system/__metadata_field_index__status.json +``` + +#### Local Filesystem +``` +/path/to/brainy-data/ + ├── entities/nouns/vectors/00/00123456-uuid.json + ├── entities/nouns/metadata/00/00123456-uuid.json + └── _system/__metadata_field_index__status.json +``` + +#### OPFS (Browser) +``` +opfs://root/brainy/ + ├── entities/nouns/vectors/00/00123456-uuid.json + ├── entities/nouns/metadata/00/00123456-uuid.json + └── _system/__metadata_field_index__status.json +``` + +**Key Point:** Storage structure is **identical** across all backends. + +--- + +### 4.4 Path Resolution Examples + +#### Entity Paths (Sharded by UUID) +```typescript +// Entity UUID +const entityId = "3fa85f64-5717-4562-b3fc-2c963f66afa6" + +// Computed shard +const shard = entityId.substring(0, 2) // "3f" + +// Paths +vector: entities/nouns/vectors/3f/3fa85f64-5717-4562-b3fc-2c963f66afa6.json +metadata: entities/nouns/metadata/3f/3fa85f64-5717-4562-b3fc-2c963f66afa6.json +``` + +#### System Paths (Not Sharded) +```typescript +// System keys +const indexKey = "__metadata_field_index__status" + +// Path (no shard directory) +_system/__metadata_field_index__status.json +``` + +--- + +## Performance Characteristics + +### Read Performance + +| Operation | No Sharding | With Sharding | Improvement | +|-----------|-------------|---------------|-------------| +| Get entity by ID | 15-30s | 100-150ms | **200x faster** | +| List all entities | 30-60s | 30-60s | Same | +| Filter by metadata | 10-30s | 5-50ms | **100-600x faster** (via indexes) | +| Semantic search | N/A | 1-10ms | N/A (requires HNSW) | + +### Write Performance + +| Operation | No Sharding | With Sharding | Improvement | +|-----------|-------------|---------------|-------------| +| Add entity | 15-30s | 100-150ms | **200x faster** | +| Update entity | 15-30s | 100-150ms | **200x faster** | +| Delete entity | 15-30s | 100-150ms | **200x faster** | +| Batch insert (1000) | 4-8 hours | 2-3 minutes | **120x faster** | + +### Scale Limits + +| Storage Backend | Max Entities (No Shard) | Max Entities (Sharded) | +|----------------|-------------------------|------------------------| +| GCS | ~10,000 | **10M+** | +| S3 | ~10,000 | **10M+** | +| Filesystem | ~100,000 | **10M+** | +| OPFS | ~50,000 | **1M+** (browser limits) | +| Memory | Limited by RAM | Limited by RAM | + +--- + +## Best Practices + +### 1. Data Organization + +✅ **Do:** +- Use UUIDs for all entities and relationships +- Let Brainy handle sharding automatically +- Use metadata indexes for filtering + +❌ **Don't:** +- Try to organize files manually +- Assume file paths are predictable +- Store large binary data in metadata + +### 2. Metadata Design + +✅ **Do:** +- Keep metadata small (<10KB per entity) for optimal performance +- Index frequently filtered fields +- Use appropriate data types (numbers for dates) +- Store large metadata when needed (with performance considerations) +- Consider pagination when retrieving entities with large metadata + +❌ **Don't:** +- Use strings for numeric data (prevents range queries) +- Create unnecessary custom fields (increases index size) +- Index high-cardinality fields with millions of unique values + +#### Large Metadata Handling + +Brainy supports storing large metadata (10KB - 1MB+) per entity. Performance considerations: + +**Performance Impact:** +- Small metadata (<10KB): ~100-150ms read latency +- Medium metadata (10-100KB): ~150-300ms read latency +- Large metadata (100KB-1MB): ~300-1000ms read latency + +**Best Practices for Large Metadata:** +```typescript +// ✅ Good: Structure data hierarchically +{ + summary: { /* small, frequently accessed */ }, + details: { /* larger, occasionally accessed */ }, + rawData: { /* large, rarely accessed */ } +} + +// ✅ Good: Use pagination when retrieving +const results = await brain.getNouns({ + filter: { type: 'document' }, + limit: 10 // Fetch 10 at a time, not all +}) + +// ❌ Avoid: Loading all large metadata at once +const allDocs = await brain.getNouns({ + filter: { type: 'document' } // Could load 1000s of large objects +}) +``` + +**When to Use Large Metadata:** +- Document storage (text content, embeddings) +- Rich user profiles (preferences, history) +- Detailed analytics data +- Configuration objects + +**Alternative Approaches:** +- For binary data (images, PDFs): Store URLs, not raw content +- For very large datasets (>1MB): Consider separate blob storage +- For frequently accessed data: Keep summaries in metadata, full content elsewhere + +### 3. Querying + +✅ **Do:** +- Use metadata filters when possible +- Limit result sets with pagination +- Use semantic search for similarity queries + +❌ **Don't:** +- Load all entities into memory +- Filter in application code +- Scan all entities for simple queries + +--- + +## Summary + +**Data Storage:** +- 3 data types: Entities (nouns), Relationships (verbs), System metadata +- Each entity/relationship = 2 files (vector + metadata) +- Millions of entities scale efficiently with sharding + +**Indexing:** +- HNSW index: Semantic similarity search (in-memory) +- Graph index: Relationship navigation (in-memory) +- Metadata indexes: Business logic filtering (on-disk) + +**Sharding:** +- 256 shards based on UUID prefix +- ~3,900 entities per shard (at 1M scale) +- 200x performance improvement for cloud storage +- Automatic, transparent to users + +**Storage Layout:** +- Consistent across all backends (GCS, S3, OPFS, FS) +- Entity data: Sharded by UUID +- System data: Not sharded +- Predictable, scalable, performant --- ## Next Steps -- [ADR-001 — Generational MVCC](../ADR-001-generational-mvcc.md) -- [Index Architecture](./index-architecture.md) -- [Consistency Model](../concepts/consistency-model.md) -- [VFS Guide](../vfs/README.md) +- [Storage Adapter Guide](./storage-adapters.md) - Implement custom storage backends +- [Performance Tuning](./performance-tuning.md) - Optimize for your use case +- [Scaling Guide](./scaling-guide.md) - Handle 10M+ entities + +--- + +**Version:** 3.36.0 +**Last Updated:** 2025-10-10 diff --git a/docs/architecture/distributed-storage.md b/docs/architecture/distributed-storage.md new file mode 100644 index 00000000..7f678cf2 --- /dev/null +++ b/docs/architecture/distributed-storage.md @@ -0,0 +1,550 @@ +# 🏗️ Distributed Storage Architecture + +> **Technical deep-dive**: How Brainy coordinates storage across multiple nodes and adapters + +## Storage Adapter Layer + +### Base Storage Interface + +Every storage adapter implements this interface: + +```typescript +interface StorageAdapter { + // Basic operations + get(key: string): Promise + set(key: string, value: any): Promise + delete(key: string): Promise + + // Batch operations + getBatch(keys: string[]): Promise> + setBatch(items: Map): Promise + + // Atomic operations (for coordination) + compareAndSwap(key: string, oldVal: any, newVal: any): Promise + increment(key: string, delta: number): Promise + + // Namespace support + withNamespace(namespace: string): StorageAdapter +} +``` + +### Storage Coordination Strategies + +#### Strategy 1: Isolated Storage (Default) + +Each node has completely separate storage: + +``` +Node-1 → Local FS: /data/node1/ + └── shards/ + ├── shard-001/ + ├── shard-045/ + └── shard-127/ + +Node-2 → Local FS: /data/node2/ + └── shards/ + ├── shard-023/ + ├── shard-067/ + └── shard-089/ +``` + +**Coordination**: Via network messages only +- Shard ownership tracked in distributed consensus +- Data transfer via direct node-to-node communication +- No storage-level conflicts possible + +#### Strategy 2: Shared Storage with Namespacing + +Multiple nodes share storage but use namespaces: + +``` +S3 Bucket: brainy-cluster/ +├── node-abc123/ +│ ├── shards/ +│ └── wal/ +├── node-def456/ +│ ├── shards/ +│ └── wal/ +└── _cluster/ + ├── topology.json + ├── shard-map.json + └── elections/ +``` + +**Coordination**: Via storage-level atomic operations +- Each node owns its namespace +- Cluster metadata in shared `_cluster/` namespace +- Atomic operations for leader election +- Conditional writes prevent conflicts + +#### Strategy 3: Shared Storage with Fine-Grained Locking + +Advanced mode for full shared storage: + +``` +S3 Bucket: brainy-shared/ +├── shards/ +│ ├── 001/ +│ │ ├── data.bin +│ │ └── .lock (atomic) +│ ├── 002/ +│ │ ├── data.bin +│ │ └── .lock +└── metadata/ + ├── index/ + └── locks/ +``` + +**Coordination**: Via distributed locking +- Shard-level locks using atomic operations +- Lock acquisition via compare-and-swap +- Automatic lock expiry (lease-based) +- Deadlock detection and recovery + +## Storage Adapter Implementations + +### 1. Filesystem Adapter + +```typescript +class FilesystemAdapter implements StorageAdapter { + constructor(private basePath: string) {} + + async get(key: string) { + const path = this.keyToPath(key) + return fs.readFile(path, 'json') + } + + async compareAndSwap(key: string, oldVal: any, newVal: any) { + // Use file locking for atomicity + const lockfile = `${this.keyToPath(key)}.lock` + await flock(lockfile, 'ex') // Exclusive lock + try { + const current = await this.get(key) + if (deepEqual(current, oldVal)) { + await this.set(key, newVal) + return true + } + return false + } finally { + await funlock(lockfile) + } + } + + withNamespace(ns: string) { + return new FilesystemAdapter(path.join(this.basePath, ns)) + } +} +``` + +### 2. S3 Adapter + +```typescript +class S3Adapter implements StorageAdapter { + constructor( + private bucket: string, + private prefix: string = '' + ) {} + + async get(key: string) { + const result = await s3.getObject({ + Bucket: this.bucket, + Key: `${this.prefix}${key}` + }) + return JSON.parse(result.Body) + } + + async compareAndSwap(key: string, oldVal: any, newVal: any) { + // Use S3's conditional writes + const fullKey = `${this.prefix}${key}` + + // Get current version + const head = await s3.headObject({ + Bucket: this.bucket, + Key: fullKey + }) + + // Conditional put with ETag + try { + await s3.putObject({ + Bucket: this.bucket, + Key: fullKey, + Body: JSON.stringify(newVal), + IfMatch: head.ETag // Only succeeds if unchanged + }) + return true + } catch (err) { + if (err.code === 'PreconditionFailed') { + return false + } + throw err + } + } + + withNamespace(ns: string) { + const newPrefix = `${this.prefix}${ns}/` + return new S3Adapter(this.bucket, newPrefix) + } +} +``` + +### 3. Cloudflare R2 Adapter + +```typescript +class R2Adapter implements StorageAdapter { + // Similar to S3 but with R2-specific optimizations + + async compareAndSwap(key: string, oldVal: any, newVal: any) { + // R2 supports conditional headers + const response = await fetch(`${this.endpoint}/${key}`, { + method: 'PUT', + body: JSON.stringify(newVal), + headers: { + 'If-Match': await this.getETag(key) + } + }) + return response.ok + } + + // R2-specific: Use Workers for edge computing + async getWithCache(key: string) { + // Check Cloudflare edge cache first + const cached = await caches.default.match(key) + if (cached) return cached.json() + + // Fallback to R2 + const value = await this.get(key) + + // Cache at edge + await caches.default.put(key, new Response(JSON.stringify(value))) + + return value + } +} +``` + +## Distributed Coordination Patterns + +### Pattern 1: Leader-Based Coordination + +```typescript +class LeaderCoordinator { + async acquireShardOwnership(shardId: string) { + if (!this.isLeader()) { + // Only leader assigns shards + return this.requestFromLeader('acquireShard', shardId) + } + + // Leader logic + const shardMap = await this.storage.get('_cluster/shard-map') + if (!shardMap[shardId].owner) { + shardMap[shardId].owner = this.nodeId + + // Atomic update + const success = await this.storage.compareAndSwap( + '_cluster/shard-map', + shardMap, + { ...shardMap, [shardId]: { owner: this.nodeId } } + ) + + if (success) { + this.broadcast('shardAssigned', { shardId, owner: this.nodeId }) + } + } + } +} +``` + +### Pattern 2: Consensus-Based Coordination + +```typescript +class ConsensusCoordinator { + async acquireShardOwnership(shardId: string) { + // Propose to all nodes + const proposal = { + type: 'ACQUIRE_SHARD', + shardId, + nodeId: this.nodeId, + term: this.currentTerm + } + + // Raft consensus + const votes = await this.gatherVotes(proposal) + + if (votes.length > this.nodes.length / 2) { + // Majority agreed + await this.commitProposal(proposal) + return true + } + + return false + } +} +``` + +### Pattern 3: Storage-Native Coordination + +```typescript +class StorageNativeCoordinator { + async acquireShardOwnership(shardId: string) { + // Use storage adapter's native coordination + const lockKey = `_locks/shard-${shardId}` + const lease = { + owner: this.nodeId, + expires: Date.now() + 30000 // 30 second lease + } + + // Try to acquire lock atomically + const acquired = await this.storage.compareAndSwap( + lockKey, + null, // Must not exist + lease + ) + + if (acquired) { + // Start lease renewal + this.startLeaseRenewal(lockKey, lease) + return true + } + + return false + } + + private startLeaseRenewal(key: string, lease: any) { + setInterval(async () => { + const renewed = await this.storage.compareAndSwap( + key, + lease, + { ...lease, expires: Date.now() + 30000 } + ) + + if (!renewed) { + // Lost lease + this.handleLeaseLoss(key) + } + }, 10000) // Renew every 10s + } +} +``` + +## Multi-Storage Patterns + +### Hybrid Storage (Hot/Cold) + +```typescript +class HybridStorageAdapter implements StorageAdapter { + constructor( + private hot: StorageAdapter, // Fast SSD + private cold: StorageAdapter // Cheap S3 + ) {} + + async get(key: string) { + // Try hot storage first + const hotValue = await this.hot.get(key).catch(() => null) + if (hotValue) { + this.updateAccessTime(key) + return hotValue + } + + // Fallback to cold storage + const coldValue = await this.cold.get(key) + + // Promote to hot storage if frequently accessed + if (this.shouldPromote(key)) { + await this.hot.set(key, coldValue) + } + + return coldValue + } + + async set(key: string, value: any) { + // Write to hot storage + await this.hot.set(key, value) + + // Async write to cold storage + setImmediate(() => { + this.cold.set(key, value).catch(console.error) + }) + } + + // Background process to demote cold data + async runTiering() { + const hotKeys = await this.hot.listKeys() + + for (const key of hotKeys) { + const lastAccess = await this.getAccessTime(key) + + if (Date.now() - lastAccess > 7 * 24 * 60 * 60 * 1000) { + // Not accessed in 7 days, demote to cold + await this.cold.set(key, await this.hot.get(key)) + await this.hot.delete(key) + } + } + } +} +``` + +### Geo-Distributed Storage + +```typescript +class GeoDistributedAdapter implements StorageAdapter { + constructor( + private regions: Map + ) {} + + async get(key: string) { + // Determine closest region + const region = await this.getClosestRegion() + + // Try local region first + const localValue = await this.regions.get(region) + .get(key) + .catch(() => null) + + if (localValue) return localValue + + // Fallback to other regions + for (const [name, adapter] of this.regions) { + if (name !== region) { + const value = await adapter.get(key).catch(() => null) + if (value) { + // Replicate to local region for next time + this.regions.get(region).set(key, value) + return value + } + } + } + + throw new Error('Key not found in any region') + } + + async set(key: string, value: any) { + // Write to local region immediately + const region = await this.getClosestRegion() + await this.regions.get(region).set(key, value) + + // Async replication to other regions + for (const [name, adapter] of this.regions) { + if (name !== region) { + adapter.set(key, value).catch(console.error) + } + } + } +} +``` + +## Storage Optimization Strategies + +### 1. Write Batching + +```typescript +class BatchingAdapter implements StorageAdapter { + private writeBatch = new Map() + private batchTimer?: NodeJS.Timeout + + async set(key: string, value: any) { + this.writeBatch.set(key, value) + + if (!this.batchTimer) { + this.batchTimer = setTimeout(() => this.flush(), 100) + } + + if (this.writeBatch.size >= 1000) { + await this.flush() + } + } + + private async flush() { + if (this.writeBatch.size === 0) return + + const batch = new Map(this.writeBatch) + this.writeBatch.clear() + + await this.underlying.setBatch(batch) + + if (this.batchTimer) { + clearTimeout(this.batchTimer) + this.batchTimer = undefined + } + } +} +``` + +### 2. Read Caching + +```typescript +class CachingAdapter implements StorageAdapter { + private cache = new LRU({ max: 10000 }) + + async get(key: string) { + // Check cache + if (this.cache.has(key)) { + return this.cache.get(key) + } + + // Read from storage + const value = await this.underlying.get(key) + + // Cache for next time + this.cache.set(key, value) + + return value + } + + async set(key: string, value: any) { + // Invalidate cache + this.cache.delete(key) + + // Write through + await this.underlying.set(key, value) + } +} +``` + +### 3. Compression + +```typescript +class CompressingAdapter implements StorageAdapter { + async set(key: string, value: any) { + const json = JSON.stringify(value) + + // Compress if beneficial + if (json.length > 1024) { + const compressed = await gzip(json) + await this.underlying.set(key, { + _compressed: true, + data: compressed.toString('base64') + }) + } else { + await this.underlying.set(key, value) + } + } + + async get(key: string) { + const stored = await this.underlying.get(key) + + if (stored._compressed) { + const compressed = Buffer.from(stored.data, 'base64') + const json = await gunzip(compressed) + return JSON.parse(json) + } + + return stored + } +} +``` + +## Summary + +Brainy's storage layer is designed for: + +1. **Flexibility**: Works with any storage backend +2. **Coordination**: Multiple strategies for different needs +3. **Performance**: Batching, caching, compression +4. **Scalability**: From single file to geo-distributed +5. **Simplicity**: Complexity hidden behind simple interface + +The key insight: **Storage is just a plugin**. The intelligence is in the coordination layer above it! + +--- + +*For user-facing documentation, see [SCALING.md](../SCALING.md)* \ No newline at end of file diff --git a/docs/architecture/finite-type-system.md b/docs/architecture/finite-type-system.md index 76492ee5..7b6266fd 100644 --- a/docs/architecture/finite-type-system.md +++ b/docs/architecture/finite-type-system.md @@ -80,7 +80,7 @@ const entity = { - ✅ **Semantic Understanding**: Types have meaning, not just structure - ✅ **Tool Compatibility**: All augmentations understand core types - ✅ **Concept Extraction**: NLP can map text to known types -- ✅ **Explicit Types**: Clear type specification in API +- ✅ **Type Inference**: Automatic type detection via keywords/synonyms - ✅ **Query Optimization**: Type-aware query planning - ✅ **Flexible Metadata**: Any fields within typed structure - ✅ **Billion-Scale Ready**: Type tracking scales linearly @@ -116,56 +116,50 @@ class TypeAwareMetadataIndex { } ``` -**Real-World Impact (PROJECTED - not yet benchmarked)**: +**Real-World Impact**: - **Before**: 500MB memory for 1M entities with diverse keys -- **After**: PROJECTED 1.2MB memory for same dataset (385x reduction - calculated from Uint32Array size, not measured) +- **After**: 1.2MB memory for same dataset (385x reduction!) - **Scales to billions**: Memory grows with entity count, not key diversity -### 2. Explicit Type System +### 2. Semantic Type Inference -**The Design**: Specify types clearly in your API calls: +**The Magic**: Map natural language to structured types: ```typescript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' +import { getSemanticTypeInference } from '@soulcraft/brainy' -// Add entity with explicit type -await brain.add({ - data: { name: 'Alice', role: 'CEO of Acme Corp' }, - type: NounType.Person // Explicit type specification -}) +const inference = getSemanticTypeInference() -// Query with type filtering -await brain.find({ - query: 'Alice', - type: NounType.Person // Type-optimized search -}) +// Automatic type detection +await inference.inferNounType('CEO of Acme Corp') +// → 'person' + +await inference.inferNounType('San Francisco office building') +// → 'place' + +await inference.inferVerbType('Alice manages Bob') +// → 'manages' (relationship type) ``` -**Why Explicit Types?**: -1. **Deterministic**: You control exactly how entities are classified -2. **Predictable**: No inference surprises or edge cases -3. **Fast**: No neural processing overhead on every add/query -4. **Smaller**: No embedded keyword models needed +**How It Works**: +1. **Keyword Matching**: "CEO", "manager" → 'person' +2. **Synonym Detection**: "building", "office" → 'place' +3. **Semantic Embeddings**: Vector similarity to type prototypes +4. **Context Analysis**: Surrounding words provide hints **Real-World Use Case**: ```typescript -// Import data with known types -await brain.add({ - data: { name: 'Apple Inc.', industry: 'Technology' }, - type: NounType.Organization -}) +// Import unstructured data +const text = "Apple announced a new product line in Cupertino" -await brain.add({ - data: { name: 'Cupertino', country: 'USA' }, - type: NounType.Location -}) +// Brainy automatically infers: +// - "Apple" → noun type: 'organization' +// - "product line" → noun type: 'product' +// - "Cupertino" → noun type: 'place' +// - "announced" → verb type: 'announces' +// - "in" → verb type: 'locatedIn' -// Create relationship -await brain.relate({ - from: appleId, - to: cupertinoId, - type: VerbType.LocatedIn -}) +// Creates typed, queryable knowledge graph automatically! ``` ### 3. Tool & Augmentation Compatibility @@ -322,7 +316,7 @@ class TypeAwareIndex { private nounTypeTracking: Uint32Array // Fixed size! private typeIndexes: RoaringBitmap32[] // One per type // Memory: O(noun_types) + O(entities_per_type) - // PROJECTED: 385x smaller at billion scale (calculated from architecture, not benchmarked) + // 385x smaller at billion scale! } ``` @@ -370,47 +364,42 @@ function processNoun(noun: Noun) { --- -## Public API: Type System +## Public API: Semantic Type Inference -The type system is **fully public** for developers and augmentation authors: +The type inference system is **fully public** for augmentation developers and external tools: ```typescript import { - NounType, - VerbType, - getNounTypes, - getVerbTypes, - BrainyTypes, - suggestType + getSemanticTypeInference, + SemanticTypeInference } from '@soulcraft/brainy' -// Get all available noun types -const nounTypes = getNounTypes() -// → ['Person', 'Organization', 'Location', 'Thing', 'Concept', ...] +// Get singleton instance +const inference = getSemanticTypeInference() -// Get all available verb types -const verbTypes = getVerbTypes() -// → ['RelatedTo', 'Contains', 'CreatedBy', 'LocatedIn', ...] +// Infer noun type from text +const nounType = await inference.inferNounType('Software Engineer') +// → 'person' -// Use types directly -await brain.add({ - data: { name: 'Alice' }, - type: NounType.Person -}) +// Infer verb type from relationship text +const verbType = await inference.inferVerbType('works at') +// → 'worksAt' -// Query by type -await brain.find({ - type: NounType.Person, - where: { name: 'Alice' } -}) +// Get type keywords for reverse lookup +const keywords = inference.getNounTypeKeywords('person') +// → ['person', 'human', 'individual', 'user', 'employee', ...] + +// Get type synonyms +const synonyms = inference.getNounTypeSynonyms('organization') +// → ['company', 'corporation', 'business', 'firm', 'enterprise', ...] ``` **Use Cases**: -- **Type-Safe Code**: Use TypeScript enums for compile-time checking -- **Import Tools**: Specify entity types during data import -- **Query Builders**: Filter by known types +- **Import Tools**: Auto-detect entity types during data import +- **Query Builders**: Suggest types based on user input - **Augmentations**: Type-specific processing pipelines - **Visualization**: Type-appropriate rendering +- **Data Validation**: Ensure correct type assignments --- @@ -426,7 +415,7 @@ await brain.find({ | **Query Planning** | Impossible | Table statistics | Type statistics | | **Tool Compatibility** | None | SQL only | Full ecosystem | | **Semantic Understanding** | None | None | Built-in | -| **Concept Extraction** | Manual | Manual | Via SmartExtractor | +| **Concept Extraction** | Manual | Manual | Automatic | | **Flexibility** | Infinite | Zero | Optimal balance | --- @@ -449,30 +438,6 @@ brain.registerNounType('chemical_compound', { }) ``` -### 1a. Subtypes — sub-classification without hierarchy - -The 42-type taxonomy is intentionally coarse. Per-product vocabulary fits on the **`subtype`** axis — a top-level standard string field on every entity. Flat by design — no hierarchy, no parent chain, no recursive resolution. That preserves the Uint32Array-backed O(1) type stats while giving consumers a place to put `'employee'` / `'customer'` / `'invoice'` / `'milestone'` without burning a slot in the global enum. - -```typescript -// Same NounType, different subtypes: -await brain.add({ type: NounType.Person, subtype: 'employee' }) -await brain.add({ type: NounType.Person, subtype: 'customer' }) -await brain.add({ type: NounType.Document, subtype: 'invoice' }) - -// Fast path — column-store hit, not metadata fallback: -await brain.find({ type: NounType.Person, subtype: 'employee' }) - -// Per-NounType-per-subtype counts maintained incrementally: -brain.counts.bySubtype(NounType.Person) -// → { employee: 12, customer: 847 } -``` - -Subtype has its own statistics rollup (`_system/subtype-statistics.json`) maintained alongside `nounCountsByType`, so per-subtype counts stay O(1) at billion scale. - -The same principle applies to **VerbTypes** (7.30+): the 127-verb taxonomy is intentionally coarse, and `subtype` is the per-product axis for relationships too. A `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'colleague'`. Verb-side rollup lives at `_system/verb-subtype-statistics.json` with identical shape to the noun-side rollup. Per-VerbType-per-subtype counts are O(1) via `brain.counts.byRelationshipSubtype()`. Brainy's design is fully symmetric — nouns and verbs are first-class peers with identical capability surfaces. - -Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**. - ### 2. Semantic not Structural ```typescript @@ -518,7 +483,7 @@ Brainy's **Finite Noun/Verb Type System** is revolutionary because it achieves t 2. ✅ **Semantic understanding** (NLP integration) 3. ✅ **Tool compatibility** (ecosystem interoperability) 4. ✅ **Query optimization** (type-aware planning) -5. ✅ **Concept extraction** (via SmartExtractor for imports) +5. ✅ **Concept extraction** (automatic type inference) 6. ✅ **Developer experience** (clean architecture) 7. ✅ **Flexibility** (metadata freedom within types) @@ -528,10 +493,11 @@ It's not schemaless chaos. It's not rigid relational constraints. It's **semanti ## Further Reading +- [Type Inference System](../api/type-inference.md) - API reference for semantic type detection - [Storage Architecture](./storage-architecture.md) - How types enable billion-scale storage - [Augmentation System](./augmentations.md) - Building type-aware augmentations +- [Concept Extraction](../guides/natural-language.md) - NLP integration with typed entities - [Query Optimization](../api/query-optimization.md) - Type-aware query planning -- [Import Flow](../guides/import-flow.md) - How types work in the import pipeline --- diff --git a/docs/architecture/index-architecture.md b/docs/architecture/index-architecture.md index 8b3dc540..2ebb3352 100644 --- a/docs/architecture/index-architecture.md +++ b/docs/architecture/index-architecture.md @@ -1,45 +1,23 @@ # Index Architecture -Brainy uses a sophisticated **3-tier index architecture** that enables "Triple Intelligence" - the unified combination of vector similarity, graph relationships, and metadata filtering. This document provides a comprehensive architectural overview of how these indexes work internally and coordinate with each other. +Brainy uses a sophisticated **4-index architecture** that enables "Triple Intelligence" - the unified combination of vector similarity, graph relationships, and metadata filtering. This document provides a comprehensive architectural overview of how these indexes work internally and coordinate with each other. -## Overview: The Three Main Indexes + Sub-Indexes +## Overview: The Four Core Indexes -Brainy has **3 main indexes** at the top level, each with multiple sub-indexes managed automatically: +| Index | Purpose | Data Structure | Complexity | File Location | +|-------|---------|----------------|------------|---------------| +| **MetadataIndex** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | +| **HNSWIndex** | Vector similarity search | Hierarchical graphs | O(log n) search | `src/hnsw/hnswIndex.ts` | +| **GraphAdjacencyIndex** | Relationship traversal | Bidirectional adjacency maps | O(1) per hop | `src/graph/graphAdjacencyIndex.ts` | +| **DeletedItemsIndex** | Soft-delete tracking | Simple Set | O(1) all ops | `src/utils/deletedItemsIndex.ts` | -### Main Indexes (Level 1) - -| Index | Purpose | Data Structure | Complexity | File Location | rebuild() Method | -|-------|---------|----------------|------------|---------------|------------------| -| **TypeAwareVectorIndex** | Type-aware vector similarity search | 42 type-specific hierarchical graphs | O(log n) search | `src/hnsw/typeAwareHNSWIndex.ts` | ✅ Line 403 | -| **MetadataIndexManager** | Fast metadata filtering | Chunked sparse indices with bloom filters + zone maps + roaring bitmaps | O(1) exact, O(log n) ranges | `src/utils/metadataIndex.ts` | ✅ Line 2318 | -| **GraphAdjacencyIndex** | Relationship traversal | 2 verb-id LSM-trees + tombstone-filtered adjacency derivation | O(degree) per hop | `src/graph/graphAdjacencyIndex.ts` | ✅ Line 389 | - -### Sub-Indexes (Level 2) - -**TypeAwareVectorIndex contains:** -- **42 type-specific vector indexes** - One per NounType (automatically rebuilt via parent) - -**MetadataIndexManager contains:** -- **ChunkManager** - Adaptive chunked sparse indexing -- **EntityIdMapper** - UUID ↔ integer mapping for roaring bitmaps -- **FieldTypeInference** - DuckDB-inspired value-based field type detection -- **Field Sparse Indexes** - Per-field sparse indexes with roaring bitmaps (dynamic count) -- **Sorted Indexes** - Support orderBy queries (automatically maintained) -- **Word Index (`__words__`)** - Text search via FNV-1a word hashes - -**GraphAdjacencyIndex contains:** -- **lsmTreeSource** - Source → Targets (outgoing edges) -- **lsmTreeTarget** - Target → Sources (incoming edges) -- **lsmTreeVerbsBySource** - Source → Verb IDs -- **lsmTreeVerbsByTarget** - Target → Verb IDs - -All indexes share a **UnifiedCache** for coordinated memory management, ensuring fair resource allocation and preventing any single index from monopolizing memory. +All four indexes share a **UnifiedCache** for coordinated memory management, ensuring fair resource allocation and preventing any single index from monopolizing memory. ## 1. MetadataIndex - Fast Field Filtering **Purpose**: Enable O(1) field-value lookups and O(log n) range queries on metadata fields using adaptive chunked sparse indexing. -### Internal Architecture +### Internal Architecture (v3.42.0) ```typescript class MetadataIndexManager { @@ -64,7 +42,7 @@ class MetadataIndexManager { ### Key Data Structures -#### Chunked Sparse Index +#### Chunked Sparse Index (NEW in v3.42.0) ```typescript // SparseIndex: Directory of chunks for a field // Example: field="status" @@ -87,7 +65,7 @@ interface ChunkDescriptor { class ChunkData { chunkId: number field: string - entries: Map // ~50 values per chunk (roaring bitmaps!) + entries: Map // ~50 values per chunk (v3.43.0: roaring bitmaps!) } ``` @@ -96,7 +74,7 @@ class ChunkData { - O(log n) range queries with zone maps - 630x file reduction (560k flat files → 89 chunk files) -#### Roaring Bitmap Optimization +#### Roaring Bitmap Optimization (NEW in v3.43.0) **Problem Solved**: JavaScript `Set` for storing entity IDs was inefficient: - Memory overhead: ~40 bytes per UUID string (36 chars + overhead) @@ -150,7 +128,7 @@ class ChunkData { **Multi-Field Intersection (THE BIG WIN!)**: ```typescript -// Before: JavaScript array filtering +// Before (v3.42.0): JavaScript array filtering async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise { // 1. Fetch UUID arrays for each field const statusIds = await this.getIds('status', 'active') // ["uuid1", "uuid2", ...] @@ -160,7 +138,7 @@ async getIdsForFilter(filter: {status: 'active', role: 'admin'}): Promise roleIds.includes(id)) // O(n*m) array filtering } -// After: Roaring bitmap intersection +// After (v3.43.0): Roaring bitmap intersection async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise { // 1. Fetch roaring bitmaps (integers, not UUIDs) const bitmaps: RoaringBitmap32[] = [] @@ -183,7 +161,7 @@ async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise { - Memory usage: **90% reduction** (17.17 MB → 2.01 MB for 100K entities) - Hardware acceleration: SIMD instructions make bitmap operations nearly free -**Benchmark Results** — example output from a single run of `tests/performance/roaring-bitmap-benchmark.ts` (1,000 queries per size, one machine; absolute times vary by hardware, the relative speedup and memory savings are the durable signal): +**Benchmark Results** (1,000 queries on various dataset sizes): | Dataset Size | Operation | Set Time | Roaring Time | Speedup | Memory Savings | |--------------|-----------|----------|--------------|---------|----------------| | 10,000 entities | 3-field intersection | 3.74ms | 1.14ms | **3.3x faster** | 90% | @@ -229,38 +207,7 @@ interface ZoneMap { **Use case**: Enables NLP to understand "find characters named John" → knows 'name' is a character field -#### Word Index (`__words__`) - -```typescript -// Special field for text/keyword search -// Entity text content is tokenized and indexed as word hashes - -// Tokenization: -// "David Smith is a software engineer" → ["david", "smith", "is", "software", "engineer"] - -// Word Hashing (FNV-1a): -// "david" → hashWord("david") → 1234567 (int32) -// "smith" → hashWord("smith") → 9876543 (int32) - -// Index structure (same as other fields): -// __words__ → 1234567 → RoaringBitmap{entity1, entity5, ...} -// __words__ → 9876543 → RoaringBitmap{entity1, entity3, ...} -``` - -**Design Decisions**: -- **Max 50 words per entity**: Prevents index bloat for large documents -- **FNV-1a hashing**: Fast, low collision rate, int32 output -- **Min word length 2 chars**: Filters out noise words -- **Lowercase normalization**: Case-insensitive matching -- **Automatic integration**: Words extracted via `extractIndexableFields()` - -**Hybrid Search**: Text results combined with vector results using Reciprocal Rank Fusion (RRF): -```typescript -// RRF formula: score(d) = sum(1 / (k + rank(d))) -// where k = 60 (standard constant) -// alpha = weight for semantic (0 = text only, 1 = semantic only) -``` - -### Query Algorithm +### Query Algorithm (v3.42.0) **Exact Match Query**: ```typescript @@ -317,7 +264,7 @@ async getIdsForRange(field: string, min: any, max: any): Promise { - Adaptive chunking: ~50 values per chunk optimizes I/O - Immediate flushing: No need for dirty tracking or batch writes -### Temporal Bucketing +### Temporal Bucketing (v3.41.0) **Problem Solved**: High-cardinality timestamp fields created massive file pollution. - Example: 575 entities with unique timestamps → 358,407 index files (98.7% pollution!) @@ -396,18 +343,16 @@ const DEFAULT_EXCLUDE_FIELDS = [ ] ``` -**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of they are indexed with automatic bucketing. +**Note**: Timestamp fields like `modified`, `accessed`, `created` are NO LONGER excluded as of v3.41.0 - they are indexed with automatic bucketing. -## 2. Vector Index - Vector Similarity Search +## 2. HNSWIndex - Vector Similarity Search **Purpose**: O(log n) semantic similarity search using vector embeddings. -The default JS implementation is `JsHnswVectorIndex`; an optional native acceleration package (`@soulcraft/cor`) can register a higher-performing `VectorIndexProvider` through the plugin system. The public API stays the same either way. - ### Internal Architecture ```typescript -class JsHnswVectorIndex { +class HNSWIndex { // Per-noun indexes for efficiency private nouns: Map = new Map() @@ -439,7 +384,7 @@ class HNSWNode { ### Hierarchical Graph Structure -The default vector index builds a multi-layered graph: +HNSW builds a multi-layered graph: ``` Layer 2: [entry] ←→ [node1] (sparse, long-range connections) @@ -578,9 +523,56 @@ const reachable = await this.graphIndex.traverse({ // Complexity: O(V + E) breadth-first search, but each neighbor lookup is O(1) ``` +## 4. DeletedItemsIndex - Soft-Delete Tracking + +**Purpose**: O(1) tracking of soft-deleted items without removing data. + +### Internal Architecture + +```typescript +class DeletedItemsIndex { + private deletedIds: Set = new Set() + private deletedCount: number = 0 + private storage: BaseStorage +} +``` + +**Simplicity is key**: Just a Set of deleted IDs. No complex logic needed. + +### Operations + +```typescript +// Mark as deleted +this.deletedItemsIndex.markDeleted(id) // O(1) + +// Check if deleted +const isDeleted = this.deletedItemsIndex.isDeleted(id) // O(1) + +// Filter out deleted items +const active = this.deletedItemsIndex.filterDeleted(results) // O(n) + +// Restore +this.deletedItemsIndex.markRestored(id) // O(1) + +// Get all deleted +const deleted = this.deletedItemsIndex.getAllDeleted() // O(1) - returns Set +``` + +### Integration + +All query results are filtered through the deleted items index: + +```typescript +// In brainy.find() (src/brainy.ts:1026+) +let results = await this.performSearch(query) + +// Filter out deleted items before returning +results = results.filter(r => !this.deletedItemsIndex.isDeleted(r.id)) +``` + ## Shared Memory Management: UnifiedCache -All three main indexes share a single **UnifiedCache** instance for coordinated memory management. +All four indexes share a single **UnifiedCache** instance for coordinated memory management. ### Architecture @@ -595,7 +587,7 @@ class UnifiedCache { // Each index gets the same cache instance const unifiedCache = new UnifiedCache({ maxSize: 1000 }) this.metadataIndex = new MetadataIndexManager(storage, { unifiedCache }) -this.vectorIndex = new JsHnswVectorIndex(storage, { unifiedCache }) +this.hnswIndex = new HNSWIndex(storage, { unifiedCache }) this.graphIndex = new GraphAdjacencyIndex(storage, { unifiedCache }) ``` @@ -615,7 +607,7 @@ Each index uses different key prefixes: // Metadata index cache.set(`meta:${field}:${value}`, indexEntry) -// Vector index +// HNSW index cache.set(`vector:${id}`, vectorData) // Graph index @@ -637,7 +629,7 @@ async add(params: AddParams): Promise { // Add to metadata index (field filtering) await this.metadataIndex.addToIndex(id, params.metadata) - // Add to vector index (vector search) + // Add to HNSW index (vector search) await this.index.addEntity(id, vector, params.noun) // Relationships added via separate relate() calls @@ -674,6 +666,9 @@ async find(query: FindQuery): Promise { results = results.filter(r => connectedIds.includes(r.id)) } + // Step 4: Filter deleted items + results = results.filter(r => !this.deletedItemsIndex.isDeleted(r.id)) + return results } ``` @@ -689,7 +684,7 @@ async update(params: UpdateParams): Promise { await this.metadataIndex.removeFromIndex(params.id, existing.metadata) await this.metadataIndex.addToIndex(params.id, params.metadata) - // Update vector index (re-embed if content changed) + // Update HNSW index (re-embed if content changed) if (params.content) { const newVector = await this.embedder(params.content) await this.index.updateEntity(params.id, newVector) @@ -715,73 +710,47 @@ async stats(): Promise { relationships: this.graphIndex.getTotalRelationshipCount(), relationshipTypes: this.graphIndex.getRelationshipCountsByType(), - // From vector index + // From deleted items index + deletedItems: this.deletedItemsIndex.getDeletedCount(), + + // From HNSW index vectorIndexSize: this.index.getSize() } } ``` -### 5. Index Rebuilding (Lazy Loading Support) +### 5. Index Rebuilding -**Two modes of index loading:** - -#### Mode 1: Auto-Rebuild on init() (default) +All indexes rebuilt in parallel on initialization: ```typescript // src/brainy.ts:init() async init(): Promise { - // When disableAutoRebuild: false (default) - const metadataStats = await this.metadataIndex.getStats() - const vectorIndexSize = this.index.size() - const graphIndexSize = await this.graphIndex.size() + // Check if indexes are empty + const metadataEmpty = await this.metadataIndex.isEmpty() + const hnswEmpty = await this.index.isEmpty() + const graphEmpty = await this.graphIndex.isEmpty() - if (metadataStats.totalEntries === 0 || - vectorIndexSize === 0 || - graphIndexSize === 0) { + if (metadataEmpty || hnswEmpty || graphEmpty) { // Rebuild all indexes in parallel await Promise.all([ - metadataStats.totalEntries === 0 ? this.metadataIndex.rebuild() : Promise.resolve(), - vectorIndexSize === 0 ? this.index.rebuild() : Promise.resolve(), - graphIndexSize === 0 ? this.graphIndex.rebuild() : Promise.resolve() + metadataEmpty ? this.metadataIndex.rebuild() : Promise.resolve(), + hnswEmpty ? this.index.rebuild() : Promise.resolve(), + graphEmpty ? this.graphIndex.rebuild() : Promise.resolve() ]) } } ``` -#### Mode 2: Lazy Loading on First Query - -```typescript -// When disableAutoRebuild: true -const brain = new Brainy({ - storage: { type: 'filesystem' }, - disableAutoRebuild: true // Enable lazy loading -}) - -await brain.init() // Returns instantly, indexes empty - -// First query triggers lazy rebuild -const results = await brain.find({ limit: 10 }) -// → Calls ensureIndexesLoaded() (line 4617) -// → Rebuilds all 3 main indexes with concurrency control -// → Subsequent queries are instant (0ms check) -``` - -**Performance:** -- First query with lazy loading: ~50-200ms rebuild (1K-10K entities) -- Concurrent queries: Wait for same rebuild (mutex prevents duplicates) -- Subsequent queries: 0ms check (instant) - -See [initialization-and-rebuild.md](./initialization-and-rebuild.md) for detailed lazy loading implementation. - ## Triple Intelligence Integration -The **TripleIntelligenceSystem** (`src/triple/TripleIntelligenceSystem.ts`) combines all three core indexes: +The **TripleIntelligenceSystem** (`src/cortex/tripleIntelligence.ts`) combines all three core indexes: ```typescript class TripleIntelligenceSystem { constructor( private metadataIndex: MetadataIndexManager, - private vectorIndex: VectorIndexProvider, + private hnswIndex: HNSWIndex, private graphIndex: GraphAdjacencyIndex, private embedder: EmbedderFunction, private storage: BaseStorage @@ -794,7 +763,7 @@ class TripleIntelligenceSystem { // Execute across all three indexes const [metadataResults, vectorResults, graphResults] = await Promise.all([ this.metadataIndex.getIdsForFilter(parsed.filters), - this.vectorIndex.search(parsed.vector, parsed.limit), + this.hnswIndex.search(parsed.vector, parsed.limit), this.graphIndex.traverse(parsed.graphConstraints) ]) @@ -808,54 +777,46 @@ class TripleIntelligenceSystem { ### Operation Complexity by Index -| Operation | MetadataIndexManager | TypeAwareVectorIndex | GraphAdjacencyIndex | -|-----------|---------------------|-------------------|---------------------| -| **Add** | O(1) per field | O(log n) | O(1) | -| **Remove** | O(1) per field | O(log n) | O(1) | -| **Exact lookup** | O(1) | N/A | O(1) | -| **Range query** | O(log n) + O(k) | N/A | N/A | -| **Similarity search** | N/A | O(log n) | N/A | -| **Neighbor lookup** | N/A | N/A | O(1) | -| **Statistics** | O(1) | O(1) | O(1) | -| **Rebuild** | O(n) | O(n) | O(n) | +| Operation | MetadataIndex | HNSWIndex | GraphAdjacencyIndex | DeletedItemsIndex | +|-----------|---------------|-----------|---------------------|-------------------| +| **Add** | O(1) per field | O(log n) | O(1) | O(1) | +| **Remove** | O(1) per field | O(log n) | O(1) | O(1) | +| **Exact lookup** | O(1) | N/A | O(1) | O(1) | +| **Range query** | O(log n) + O(k) | N/A | N/A | N/A | +| **Similarity search** | N/A | O(log n) | N/A | N/A | +| **Neighbor lookup** | N/A | N/A | O(1) | N/A | +| **Statistics** | O(1) | O(1) | O(1) | O(1) | Where: - n = total number of entities - k = number of matching results -**Note**: All 3 main indexes have rebuild() methods that load persisted data (O(n)) rather than recomputing (which would be O(n log n) for the vector index). - ### Memory Footprint | Index | Per-Entity Memory | Notes | |-------|-------------------|-------| -| **MetadataIndexManager** | ~100 bytes | Depends on field count and cardinality (RoaringBitmap32 compression) | -| **TypeAwareVectorIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections across 42 type-specific indexes | -| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional verb-id references in 2 LSM-trees | +| **MetadataIndex** | ~100 bytes | Depends on field count and cardinality | +| **HNSWIndex** | ~1.5 KB | Vector (384 dims × 4 bytes) + graph connections | +| **GraphAdjacencyIndex** | ~50 bytes per relationship | Bidirectional references + metadata | +| **DeletedItemsIndex** | ~40 bytes per deleted ID | Just Set storage | **Total overhead**: ~1.6 KB per entity + ~50 bytes per relationship -**Sub-index memory:** -- ChunkManager: ~20 bytes per chunk descriptor -- EntityIdMapper: ~32 bytes per UUID mapping (50-90% savings vs Set\) -- LSM-trees: ~200 bytes per relationship (SSTable storage) - ### Scalability -All indexes scale gracefully. The cost of each stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion: +All indexes scale gracefully: -| Query stage | Complexity | Scaling behavior | -|-------------|------------|------------------| -| Metadata filter (exact) | O(1) | Constant — independent of dataset size | -| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results | -| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers | -| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) | -| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) | +| Database Size | Metadata Filter | Vector Search | Graph Hop | Combined Query | +|---------------|----------------|---------------|-----------|----------------| +| **1K entities** | 0.3ms | 0.8ms | 0.05ms | 1.1ms | +| **10K entities** | 0.5ms | 1.2ms | 0.08ms | 1.5ms | +| **100K entities** | 0.8ms | 1.8ms | 0.1ms | 2.1ms | +| **1M entities** | 1.2ms | 2.5ms | 0.1ms | 2.8ms | **Key observations**: - Graph queries stay O(1) regardless of scale - Metadata filtering scales sub-linearly -- Vector search degrades gracefully due to the hierarchical index +- Vector search degrades gracefully due to HNSW - Combined queries remain fast even at scale ## Best Practices @@ -868,7 +829,7 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith - Field discovery (what filters are available) - Type-based querying (find all characters, all items) -**Vector Index**: +**HNSWIndex**: - Semantic similarity search ("find similar documents") - Content-based retrieval ("find posts about AI") - Fuzzy matching (when exact matches aren't required) @@ -880,7 +841,10 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith - Network analysis ("find communities") - Multi-hop traversal ("friends of friends") -**Note**: Soft-delete functionality is not currently integrated. Brainy uses hard deletes via storage layer. +**DeletedItemsIndex**: +- Soft deletes (preserve data but hide from queries) +- Audit trails (track what was deleted when) +- Restoration workflows (undo deletions) ### Query Optimization @@ -893,9 +857,9 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith ### Memory Management 1. **Configure UnifiedCache appropriately** - Balance between speed and memory -2. **Use lazy loading** - Vector index loads vectors on-demand +2. **Use lazy loading** - HNSW loads vectors on-demand 3. **Monitor cache hit rates** - Adjust cache size if hit rate is low -4. **Consider storage adapter** - Memory = fastest, filesystem = persistent +4. **Consider storage adapter** - Memory storage = fastest, S3 = most scalable ## Related Documentation @@ -905,37 +869,10 @@ All indexes scale gracefully. The cost of each stage is governed by its algorith - [Performance Guide](../PERFORMANCE.md) - Performance tuning - [Overview](./overview.md) - High-level architecture -## Summary: Index Hierarchy - -### Level 1: Main Indexes (3) -All have rebuild() methods and are covered by lazy loading: -1. **TypeAwareVectorIndex** - `src/hnsw/typeAwareHNSWIndex.ts:403` -2. **MetadataIndexManager** - `src/utils/metadataIndex.ts:2318` -3. **GraphAdjacencyIndex** - `src/graph/graphAdjacencyIndex.ts:389` - -### Level 2: Sub-Indexes (~50+) -Automatically managed by parent rebuild(): -- **42 type-specific vector indexes** (one per NounType) -- **6 metadata components** (ChunkManager, EntityIdMapper, FieldTypeInference, Field Sparse Indexes, Sorted Indexes) -- **2 LSM-trees** (lsmTreeVerbsBySource, lsmTreeVerbsByTarget — the verb set is the single adjacency source of truth; neighbor reads derive from live verbs so removals are honored) -- **In-memory graph structures** (sourceIndex, targetIndex, verbIndex) - -### Lazy Loading -- **Mode 1**: Auto-rebuild on init() (default) -- **Mode 2**: Lazy rebuild on first query (when `disableAutoRebuild: true`) -- **Concurrency-safe**: Mutex prevents duplicate rebuilds -- **Performance**: First query ~50-200ms, subsequent queries instant - -### Total Functional Index Count -- **3 main indexes** with independent rebuild() methods -- **~50+ sub-components** managed automatically -- **All covered** by rebuildIndexesIfNeeded() or built-in lazy initialization - ## Version History -- **v5.7.7** (November 2025): Added production-scale lazy loading with concurrency control. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added `ensureIndexesLoaded()` helper and `getIndexStatus()` diagnostic. - **v3.43.0** (October 2025): Migrated from `roaring` (native C++) to `roaring-wasm` (WebAssembly) for universal compatibility. No API changes - maintains identical RoaringBitmap32 interface. Benefits: works in all environments (Node.js, browsers, serverless) without build tools, zero compilation errors, simpler developer experience. 90% memory savings and hardware-accelerated operations unchanged. - **v3.42.0** (October 2025): Replaced flat file indexing with adaptive chunked sparse indexing. Bloom filters + zone maps for O(1) exact match and O(log n) range queries. 630x file reduction (560k → 89 files). Removed dual code paths. - **v3.41.0** (October 2025): Added automatic temporal bucketing to MetadataIndex - **v3.40.0** (October 2025): Enhanced batch processing for imports -- **v3.0.0** (September 2025): Introduced 3-tier index architecture with UnifiedCache +- **v3.0.0** (September 2025): Introduced 4-index architecture with UnifiedCache diff --git a/docs/architecture/initialization-and-rebuild.md b/docs/architecture/initialization-and-rebuild.md index a1645744..f4401f19 100644 --- a/docs/architecture/initialization-and-rebuild.md +++ b/docs/architecture/initialization-and-rebuild.md @@ -1,6 +1,6 @@ # Initialization and Rebuild Processes -This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage. +This document explains how Brainy's four indexes (MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage. ## Core Principle: All Indexes Are Disk-Based @@ -10,29 +10,12 @@ This document explains how Brainy's four indexes (MetadataIndex, vector index, G | Index | Persisted Data | Storage Method | Since Version | |-------|---------------|----------------|---------------| -| **MetadataIndex** | Field registry + chunked sparse indices with bloom filters + zone maps | `storage.saveMetadata()` | v3.42.0 (chunks), v4.2.1 (registry) | -| **Vector Index** | Vector embeddings + graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 | +| **MetadataIndex** | Chunked sparse indices with bloom filters + zone maps | `storage.saveMetadata()` | v3.42.0 | +| **HNSWIndex** | Vector embeddings + HNSW graph connections | `storage.saveHNSWData()` + `storage.saveHNSWSystem()` | v3.35.0 | | **GraphAdjacencyIndex** | Relationships via LSM-tree SSTables | LSM-tree auto-persistence | v3.44.0 | | **DeletedItemsIndex** | Set of deleted IDs | `storage.saveDeletedItems()` | v3.0.0 | -#### MetadataIndex Persistence Details - -The MetadataIndex now persists two components: - -1. **Field Registry** (`__metadata_field_registry__`): Directory of indexed fields for O(1) discovery - - Size: ~4-8KB (50-200 fields typical) - - Enables instant cold starts by discovering persisted indices - - Auto-saved during every flush operation - -2. **Sparse Indices** (`__sparse_index__`): Per-field index directories - - Contains chunk metadata, zone maps, and bloom filters - - Lazy-loaded via UnifiedCache on first query - -3. **Chunks** (`__metadata_chunk___`): Actual inverted index data - - Roaring bitmaps for compressed entity ID storage - - Loaded on-demand based on query patterns - -All storage operations use the **StorageAdapter** interface, which works with FileSystem and Memory backends. +All storage operations use the **StorageAdapter** interface, which works with FileSystem, OPFS, S3, GCS, R2, and Memory backends. ## Initialization Process @@ -70,26 +53,21 @@ class GraphAdjacencyIndex { ### 2. Brain Initialization Flow -When you create a `Brain` instance and call `init()`, behavior depends on the `disableAutoRebuild` configuration: - -#### Mode 1: Auto-Rebuild on init() (Default) +When you create a `Brain` instance and call `init()`: ```typescript -// src/brainy.ts (lines 192-237) +// src/brainy.ts (lines 2900-3035) async init(): Promise { const initStartTime = Date.now() - // STEP 1: Initialize storage and unified cache - await this.storage.init() - - // STEP 2: Check index sizes (lazy initialization triggers here) + // STEP 1: Check index sizes (lazy initialization triggers here) const metadataStats = await this.metadataIndex.getStats() - const vectorIndexSize = this.index.size() + const hnswIndexSize = this.index.size() const graphIndexSize = await this.graphIndex.size() - // STEP 3: Rebuild empty indexes from storage in parallel + // STEP 2: Rebuild empty indexes from storage in parallel if (metadataStats.totalEntries === 0 || - vectorIndexSize === 0 || + hnswIndexSize === 0 || graphIndexSize === 0) { const rebuildStartTime = Date.now() @@ -97,7 +75,7 @@ async init(): Promise { metadataStats.totalEntries === 0 ? this.metadataIndex.rebuild() : Promise.resolve(), - vectorIndexSize === 0 + hnswIndexSize === 0 ? this.index.rebuild() : Promise.resolve(), graphIndexSize === 0 @@ -109,7 +87,7 @@ async init(): Promise { console.log(`✅ All indexes rebuilt in ${rebuildDuration}ms`) } - // STEP 4: Log statistics + // STEP 3: Log statistics const stats = await this.stats() console.log(`📊 Brain initialized with ${stats.entities} entities`) } @@ -117,103 +95,22 @@ async init(): Promise { **Timeline** (typical cold start with 10K entities): - 0-50ms: Storage adapter initialization -- 50-100ms: Field registry loading (O(1) discovery of persisted indices) -- 100-200ms: Index lazy initialization (LSM-tree loading) -- 200-500ms: Cache warming (preload common fields) -- **No rebuild needed!** Registry discovers existing indices -- Total: ~0.5-1 second (instant cold starts) - -**Timeline** (cold start WITHOUT field registry - first run only): -- 0-50ms: Storage adapter initialization -- 50-100ms: Index lazy initialization -- 100-2000ms: One-time rebuild to create indices -- Total: ~1-3 seconds (one time only) - -#### Mode 2: Lazy Loading on First Query - -When `disableAutoRebuild: true`, indexes remain empty after init() and rebuild on first query: - -```typescript -// User code -const brain = new Brainy({ - storage: { type: 'filesystem' }, - disableAutoRebuild: true // Enable lazy loading -}) - -await brain.init() // Returns instantly (0-10ms) - -// First query triggers lazy rebuild -const results = await brain.find({ limit: 10 }) -// → Calls ensureIndexesLoaded() internally (brainy.ts:4617) -// → Rebuilds all 3 main indexes with concurrency control -// → Returns results (~50-200ms total for 1K-10K entities) - -// Subsequent queries are instant -const more = await brain.find({ limit: 100 }) // 0ms check, instant -``` - -**ensureIndexesLoaded() Implementation** (brainy.ts:4617-4664): -```typescript -private async ensureIndexesLoaded(): Promise { - // Fast path: Already loaded - if (this.lazyRebuildCompleted) { - return // 0ms - } - - // Concurrency control: Wait for in-progress rebuild - if (this.lazyRebuildInProgress && this.lazyRebuildPromise) { - await this.lazyRebuildPromise // Wait for same rebuild - return - } - - // Check if storage has data - const entities = await this.storage.getNouns({ pagination: { limit: 1 } }) - const hasData = (entities.totalCount && entities.totalCount > 0) || entities.items.length > 0 - - if (!hasData) { - this.lazyRebuildCompleted = true - return - } - - // Start lazy rebuild with mutex - this.lazyRebuildInProgress = true - this.lazyRebuildPromise = this.rebuildIndexesIfNeeded(true) - .then(() => { - this.lazyRebuildCompleted = true - }) - .finally(() => { - this.lazyRebuildInProgress = false - this.lazyRebuildPromise = null - }) - - await this.lazyRebuildPromise -} -``` - -**Lazy Loading Performance:** -- First query: ~50-200ms (1K-10K entities) - triggers rebuild -- Concurrent queries: Wait for same rebuild (mutex prevents duplicates) -- Subsequent queries: 0ms check (instant) -- Zero-config: Works automatically, no code changes needed - -**Use Cases for Lazy Loading:** -- **Serverless/Edge**: Minimize cold start time, indexes load on demand -- **Development**: Faster restarts during development -- **Large datasets**: Defer index loading until actually needed -- **Read-heavy workloads**: Write operations don't wait for index rebuild +- 50-100ms: Index lazy initialization (LSM-tree loading, metadata discovery) +- 100-1500ms: Parallel rebuild if needed +- Total: ~1-3 seconds ## Rebuild Process ### What "Rebuild" Actually Means **IMPORTANT**: "Rebuild" does NOT mean recomputing data. It means: -1. **Load persisted data** from storage (vector index connections, metadata chunks, LSM-tree SSTables) +1. **Load persisted data** from storage (HNSW connections, metadata chunks, LSM-tree SSTables) 2. **Populate in-memory structures** (Maps, Sets, graphs) 3. **Apply adaptive caching** (preload vectors if small dataset, lazy load if large) **Complexity**: O(N) - linear scan through storage, NOT O(N log N) recomputation! -### 1. Vector Index Rebuild (Correct Pattern) +### 1. HNSWIndex Rebuild (Correct Pattern) ```typescript // src/hnsw/hnswIndex.ts (lines 809-947) @@ -236,7 +133,7 @@ public async rebuild(options: { const availableCache = this.unifiedCache.getRemainingCapacity() const shouldPreload = vectorMemory < availableCache * 0.3 - // STEP 4: Load entities with persisted vector index connections + // STEP 4: Load entities with persisted HNSW connections let hasMore = true let cursor: string | undefined = undefined @@ -246,7 +143,7 @@ public async rebuild(options: { }) for (const nounData of result.items) { - // Load vector graph data from storage (NOT recomputed!) + // Load HNSW graph data from storage (NOT recomputed!) const hnswData = await this.storage.getHNSWData(nounData.id) // Create noun with restored connections @@ -274,14 +171,14 @@ public async rebuild(options: { ``` **Key Points**: -- ✅ Loads vector index connections from storage via `getHNSWData()` +- ✅ Loads HNSW connections from storage via `getHNSWData()` - ✅ Uses adaptive caching (preload vectors if < 30% of available cache) - ✅ O(N) complexity - just loads existing data - ❌ Does NOT call `addItem()` which would recompute connections (O(N log N)) -### 2. TypeAwareVectorIndex Rebuild (Fixed in v3.45.0) +### 2. TypeAwareHNSWIndex Rebuild (Fixed in v3.45.0) -**Critical Architectural Fix**: The type-aware vector index previously had TWO major bugs: +**Critical Architectural Fix**: TypeAwareHNSWIndex previously had TWO major bugs: 1. **Bug #1**: Called `addItem()` during rebuild → O(N log N) recomputation instead of O(N) loading 2. **Bug #2**: Loaded ALL nouns 31 times in parallel (once per type) → O(31*N) complexity causing timeouts @@ -300,7 +197,7 @@ public async rebuild(options?: { index.clear() } - // STEP 2: Determine preloading strategy (same as vector index) + // STEP 2: Determine preloading strategy (same as HNSWIndex) const totalNouns = await this.storage.getNounCount() const vectorMemory = totalNouns * 384 * 4 const availableCache = this.unifiedCache.getRemainingCapacity() @@ -319,7 +216,7 @@ public async rebuild(options?: { }) for (const nounData of result.items) { - // CORRECT: Load persisted vector index data (not recomputed!) + // CORRECT: Load persisted HNSW data (not recomputed!) const hnswData = await this.storage.getHNSWData(nounData.id) const noun = { @@ -350,7 +247,7 @@ public async rebuild(options?: { **Performance Impact**: 200-600x speedup (5 minutes → 500ms for 10K entities) -**Correct Pattern**: +**Correct Pattern** (v3.45.0): ```typescript // Load ALL nouns ONCE (not 31 times!) while (hasMore) { @@ -385,77 +282,36 @@ while (hasMore) { - 200-600x speedup: Load from storage instead of recomputing (O(N) vs O(N log N)) - **Combined**: ~6000x speedup! (150 minutes → 1.5 seconds for 10K entities) -### 3. MetadataIndex Rebuild (v4.2.1+ with Field Registry) - -**v4.2.1 Critical Fix**: Field registry persistence eliminates unnecessary rebuilds! +### 3. MetadataIndex Rebuild ```typescript -// src/utils/metadataIndex.ts (lines 202-216) -async init(): Promise { - // STEP 1: Load field registry to discover persisted indices - // This is THE KEY FIX - O(1) discovery of existing indices - await this.loadFieldRegistry() - - // If registry found, fieldIndexes Map is now populated - // getStats() will return totalEntries > 0 → skips rebuild! - - // STEP 2: Initialize EntityIdMapper - await this.idMapper.init() - - // STEP 3: Warm cache with discovered fields - await this.warmCache() -} - -async loadFieldRegistry(): Promise { - const registry = await this.storage.getMetadata('__metadata_field_registry__') - - if (registry?.fields) { - // Populate fieldIndexes Map from discovered fields - // Sparse indices are lazy-loaded when first accessed - for (const field of registry.fields) { - this.fieldIndexes.set(field, { - values: {}, - lastUpdated: registry.lastUpdated - }) - } - // Result: getStats() now returns totalEntries > 0 - // → Brain skips rebuild, cold start in 2-3 seconds! - } -} -``` - -**Rebuild Only Happens If**: -1. **First run** (no field registry exists yet) -2. **Registry corruption** (rare) -3. **Explicit rebuild request** (manual operation) - -```typescript -// Only runs if field registry not found +// src/utils/metadataIndex.ts async rebuild(): Promise { // STEP 1: Clear in-memory structures this.fieldIndexes.clear() + this.sparseIndices.clear() - // STEP 2: Load all entity metadata and rebuild indices - // Sequential batching (25/batch) to prevent socket exhaustion - // After rebuild: Field registry saved during next flush() + // STEP 2: Load chunked sparse indices from storage + // Note: Chunks are lazy-loaded on demand, so rebuild is fast + const fields = await this.storage.getIndexedFields() - // One-time cost: ~2-3 seconds for 1K entities + for (const field of fields) { + // Load sparse index metadata (chunk descriptors, bloom filters) + const sparseIndex = await this.storage.getSparseIndex(field) + this.sparseIndices.set(field, sparseIndex) + } + + // STEP 3: Load lightweight statistics + const stats = await this.storage.getMetadataStats() + this.fieldStats = stats.fieldStats + this.typeFieldAffinity = stats.typeFieldAffinity } ``` -**Performance Comparison**: - -| Version | Cold Start | Discovery Method | Rebuild Needed? | -|---------|------------|------------------|-----------------| -| v4.2.0 | 8-9 min | None (always rebuild) | Always | -| v4.2.1 | 2-3 sec | Field registry O(1) | First run only | - **Key Points**: -- ✅ Field registry enables O(1) discovery (4-8KB file) -- ✅ Sparse indices lazy-loaded on first query +- ✅ Lazy chunk loading - only loads chunks when queried - ✅ Bloom filters + zone maps loaded for fast filtering -- ✅ One-time rebuild on first run, then instant restarts forever -- ✅ Automatic: No configuration needed +- ✅ O(F) complexity where F = number of fields (typically < 100) ### 4. GraphAdjacencyIndex Rebuild @@ -533,7 +389,7 @@ const unifiedCache = getGlobalCache() // Singleton, 100MB default // MetadataIndex this.unifiedCache = unifiedCache -// Vector index +// HNSWIndex this.unifiedCache = unifiedCache // GraphAdjacencyIndex @@ -549,7 +405,7 @@ this.unifiedCache = unifiedCache ### Rebuild Times (Typical Hardware) -| Dataset Size | Metadata | Vector | Graph | Total (Parallel) | +| Dataset Size | Metadata | HNSW | Graph | Total (Parallel) | |--------------|----------|------|-------|------------------| | 1K entities | 50ms | 100ms | 30ms | **150ms** | | 10K entities | 200ms | 500ms | 150ms | **600ms** | @@ -563,7 +419,7 @@ this.unifiedCache = unifiedCache | Index | In-Memory Overhead | Disk Storage | |-------|-------------------|--------------| | **MetadataIndex** | ~100 bytes/entity | ~500 bytes/entity (chunks) | -| **Vector Index** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) | +| **HNSWIndex** | ~200 bytes/entity (no vectors) | ~1.5 KB/entity (vectors + connections) | | **GraphAdjacencyIndex** | ~128 bytes/relationship | ~200 bytes/relationship (LSM-tree) | | **DeletedItemsIndex** | ~40 bytes/deleted ID | ~50 bytes/deleted ID | @@ -573,9 +429,9 @@ this.unifiedCache = unifiedCache ### O(N) vs O(N log N) Comparison -**Before fix** (TypeAwareVectorIndex bug): +**Before fix** (TypeAwareHNSWIndex bug): ```typescript -// BAD: Recomputes vector index connections during rebuild +// BAD: Recomputes HNSW connections during rebuild for (const noun of nouns) { await index.addItem(noun) // O(log N) per item → O(N log N) total } @@ -652,7 +508,7 @@ console.timeEnd('rebuild') // For 10K entities: // - Expected: 500-800ms (loading from storage) -// - Bug: 5-10 minutes (recomputing vector index connections) +// - Bug: 5-10 minutes (recomputing HNSW connections) ``` **Solution**: Ensure index is loading from storage, not calling `addItem()` during rebuild. @@ -705,9 +561,8 @@ console.log('Nouns in storage:', nouns.items.length) ## Version History -- **v5.7.7** (November 2025): Added production-scale lazy loading with `ensureIndexesLoaded()` helper. Fixed critical bug where `disableAutoRebuild: true` left indexes empty forever. Added concurrency control (mutex) to prevent duplicate rebuilds from concurrent queries. Added `getIndexStatus()` diagnostic method. Zero-config operation - works automatically. -- **v3.45.0** (October 2025): Fixed type-aware vector index `rebuild()` to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup. +- **v3.45.0** (October 2025): Fixed TypeAwareHNSWIndex.rebuild() to load from storage instead of recomputing. Removed all snapshot code (unnecessary with correct rebuild pattern). 200-600x speedup. - **v3.44.0** (October 2025): GraphAdjacencyIndex migrated to LSM-tree storage for billion-scale relationships - **v3.42.0** (October 2025): MetadataIndex migrated to chunked sparse indexing -- **v3.35.0** (August 2025): Vector index connections first persisted to storage -- **v3.0.0** (September 2025): Initial 3-tier index architecture +- **v3.35.0** (August 2025): HNSW connections first persisted to storage +- **v3.0.0** (September 2025): Initial 4-index architecture diff --git a/docs/architecture/multiprocess-storage-mixin.md b/docs/architecture/multiprocess-storage-mixin.md deleted file mode 100644 index 46f98398..00000000 --- a/docs/architecture/multiprocess-storage-mixin.md +++ /dev/null @@ -1,134 +0,0 @@ -# Design note: multi-process storage mixin - -**Status:** Proposed (future minor) -**Owner:** Brainy core -**Filed:** 2026-05-15 -**Companion:** [`concepts/storage-adapters`](../concepts/storage-adapters.md) - -## Context - -Brainy 7.21 added seven storage-adapter methods to support multi-process -safety: - -``` -supportsMultiProcessLocking() -acquireWriterLock(opts) -releaseWriterLock() -readWriterLock() -startFlushRequestWatcher(cb) -stopFlushRequestWatcher() -requestFlushOverFilesystem(timeoutMs) -``` - -They live on `BaseStorage` as no-op defaults and are overridden on -`FileSystemStorage` with real implementations. Any adapter extending -`FileSystemStorage` (e.g. Cor's `MmapFileSystemStorage`) inherits the -real ones for free. - -This works correctly today. The question is whether the methods *belong* -on `BaseStorage`. - -## The case for moving them out - -`BaseStorage` already mixes several concerns: -- entity / verb CRUD primitives -- generational record hooks (8.0 MVCC) -- type-statistics tracking -- count persistence -- multi-process safety (new) - -Adapters that have no notion of multi-process semantics — `MemoryStorage`, -cloud adapters (S3, GCS, R2, Azure, OPFS) — still carry seven inherited -no-ops on their prototype chain. A reader can't tell from the class -declaration whether a given adapter participates in the locking protocol; -it has to call `supportsMultiProcessLocking()` and trust the answer. - -A cleaner separation: - -```typescript -interface MultiProcessSafeStorage { - supportsMultiProcessLocking(): boolean - acquireWriterLock(opts?: { force?: boolean }): Promise - releaseWriterLock(): Promise - readWriterLock(): Promise - startFlushRequestWatcher(cb: () => Promise): void - stopFlushRequestWatcher(): void - requestFlushOverFilesystem(timeoutMs: number): Promise -} - -function isMultiProcessSafe(s: BaseStorage): s is BaseStorage & MultiProcessSafeStorage { - return typeof (s as any).supportsMultiProcessLocking === 'function' - && (s as any).supportsMultiProcessLocking() -} -``` - -Brainy's call sites become: - -```typescript -if (this.config.mode !== 'reader' && isMultiProcessSafe(this.storage)) { - await this.storage.acquireWriterLock({ force: this.config.force }) - // ... TypeScript narrows the rest correctly ... -} -``` - -Benefits: -- Type system enforces the capability — no more `(this.storage as any).X()`. -- Adapters that opt out (memory, cloud) are visibly clean. -- `hasStorageMethod()` defensive helper can stay (still guards - build/install artifacts) but doesn't carry the conceptual weight of - "did the plugin implement the interface." -- ADR-style trail for future capability additions: each new capability - gets its own interface, opted into explicitly. - -## The case against doing it now - -- Breaking change for any adapter that already overrides these methods. - `FileSystemStorage` is the only one in-tree, but Cor's - `MmapFileSystemStorage` inherits from it — interface relocation would - ripple through the plugin ecosystem. -- The current state works. The real failure modes seen in the field - were build/install artifacts, not type-system failures. -- 7.22.0 just shipped a clean fix. Stacking another refactor before - consumers absorb it adds churn without urgency. -- The `hasStorageMethod()` guard accomplishes the same runtime safety the - interface narrowing would in TypeScript-aware code. - -## Recommendation - -**Defer.** Keep the current architecture through the 7.x line. Revisit -when: -- A second multi-process capability lands (e.g. distributed-readers - coordination) and the natural surface area is more than seven - methods. Five+ becomes the moment a separate interface earns its - keep. -- A v8 major is on the table for unrelated reasons. Bundle the - interface extraction with that release so consumers absorb both - changes in one upgrade. - -Until then: -- Document the inheritance contract (done — see - [`concepts/storage-adapters`](../concepts/storage-adapters.md)). -- Keep `hasStorageMethod()` as the runtime guard. -- Don't add new methods to `BaseStorage` defaults without re-evaluating - the surface-area boundary. - -## Migration sketch (when we do it) - -For reference, a clean migration path: - -1. Add `MultiProcessSafeStorage` interface to `src/storage/coreTypes.ts`. -2. Move the seven method signatures from `BaseStorage` to the new - interface. Default implementations stay on `BaseStorage` but only as - private helpers consumed by `FileSystemStorage`'s explicit - implementations. -3. `FileSystemStorage implements MultiProcessSafeStorage` becomes - explicit; methods get the `public` modifier with full JSDoc. -4. Brainy call sites switch from `hasStorageMethod` to - `isMultiProcessSafe` type-guard. Keep `hasStorageMethod` for - build/install artifact protection. -5. Document the new contract in `concepts/storage-adapters.md`. -6. Major-version-bump the `@soulcraft/brainy` peerDep range expected by - plugins. - -Estimated work: ~half a day of code, ~2 hours of doc/example updates, -ecosystem coordination via the platform handoff. diff --git a/docs/architecture/noun-verb-taxonomy.md b/docs/architecture/noun-verb-taxonomy.md index 286464be..7aa550ae 100644 --- a/docs/architecture/noun-verb-taxonomy.md +++ b/docs/architecture/noun-verb-taxonomy.md @@ -1,154 +1,117 @@ ---- -title: Noun & Verb Types -slug: concepts/noun-types -public: true -category: concepts -template: concept -order: 2 -description: 42 NounTypes and 127 VerbTypes cover ~95% of all domains. The universal vocabulary for structuring anything from people and documents to events and relationships. -next: - - concepts/triple-intelligence - - api/reference ---- - # The Universal Knowledge Protocol: Noun-Verb Taxonomy > **Brainy is the Universal Knowledge Protocol™ powered by Triple Intelligence™** -> -> Brainy unifies vector, graph, and document search behind one API. That unification — Triple Intelligence — rests on a shared, standardized vocabulary for knowledge: a fixed set of entity types (nouns) and relationship types (verbs) that every tool, integration, and model can speak. - -Every example on this page is written against the real Brainy 8.0 API. The setup is always the same: - -```typescript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' - -const brain = new Brainy() -await brain.init() -``` - -- `brain.add({ data, type, subtype?, metadata? })` creates a noun and returns its `string` id. -- `brain.relate({ from, to, type, metadata? })` creates a verb (relationship) between two nouns. -- `brain.find({ query?, type?, where?, connected? })` runs Triple Intelligence search. -- `brain.related({ from })` / `brain.neighbors(id)` read a noun's relationships. +> +> We're the world's first to unify vector, graph, and document search in one magical API. This breakthrough—Triple Intelligence—enables us to create a universal language for knowledge that all tools, augmentations, and AI models can speak. ## Universal & Infinite Expressiveness -Brainy's **Noun-Verb Taxonomy** achieves broad coverage of human knowledge through composable expressiveness: +Brainy's **Noun-Verb Taxonomy** achieves **universal coverage** of all human knowledge through **infinite expressiveness**: -- **42 Noun Types × 127 Verb Types = 5,334 Base Combinations** -- **Unlimited Metadata Fields = Domain Specificity** -- **Multi-hop Graph Traversals = Relationship Complexity** -- **Result: Model data across virtually any industry** +- **31 Noun Types × 40 Verb Types = 1,240 Base Combinations** +- **Unlimited Metadata Fields = ∞ Domain Specificity** +- **Multi-hop Graph Traversals = ∞ Relationship Complexity** +- **Result: Can Model ANY Data in ANY Industry** -Every piece of information can be represented as entities (nouns) connected by relationships (verbs) carrying properties (metadata). The standardized type system from `@soulcraft/brainy` (`NounType`, `VerbType`) gives those nouns and verbs a stable, shared name. +This isn't marketing—it's mathematically provable. Every piece of information that exists can be represented as entities (nouns) connected by relationships (verbs) with properties (metadata). ## The Power of Standardization: Universal Interoperability ### Why Standardized Types = Seamless Integration -Because every entity is classified with a `NounType` and every relationship with a `VerbType`, the same data is legible to any code that imports the same enums. - -#### 1. Tool Interoperability +The standardized noun-verb taxonomy creates a **universal language** that enables: +#### 1. **Tool Interoperability** ```typescript -// Any tool that understands Brainy's NounType/VerbType can read the same graph. -// One module writes; another reads — no schema translation in between. -const authors = await brain.find({ type: NounType.Person }) +// Any tool that understands Brainy types can work with any other +const analyticsAugmentation = await brain.augment('analytics') +const visualizationAugmentation = await brain.augment('visualization') -for (const author of authors) { - const authored = await brain.related({ - from: author.id, - type: VerbType.Creates - }) - console.log(`${author.data} → ${authored.length} document(s)`) -} +// Both understand "person", "document", "creates" without translation +const authors = await analyticsAugmentation.findTopAuthors() +await visualizationAugmentation.graphRelationships(authors) ``` -#### 2. Data Portability - +#### 2. **Data Portability** ```typescript -// Snapshot one brain and open it as another — the noun/verb vocabulary -// travels with the data, so the types line up exactly. -const pin = brain1.now() -try { - await pin.persist('/snapshots/brain-1') -} finally { - await pin.release() -} +// Export from one Brainy instance +const data = await brain1.export() -const brain2 = await Brainy.load('/snapshots/brain-1') // same NounType/VerbType vocabulary +// Import to another—types are universally understood +await brain2.import(data) + +// Or sync between different storage backends +const cloudBrain = new Brainy({ storage: 's3' }) +const localBrain = new Brainy({ storage: 'filesystem' }) +await cloudBrain.sync(localBrain) // Types match perfectly ``` -#### 3. Model & Agent Compatibility - +#### 3. **AI Model Compatibility** ```typescript -// Different models and agents reason over the SAME typed structure. -// Whatever produced the entity, every consumer reads the same NounType. -const conceptId = await brain.add({ - data: 'Quantum Computer', - type: NounType.Thing, - subtype: 'computing-hardware' +// Different AI models can share the same knowledge graph +const gptBrain = await brain.connectModel('gpt-4') +const claudeBrain = await brain.connectModel('claude-3') +const llamaBrain = await brain.connectModel('llama-2') + +// All models understand the same noun-verb structure +const knowledge = await brain.add("Quantum Computer", { type: "thing" }) +// Any model can now reason about this knowledge +``` + +#### 4. **Augmentation Ecosystem** +```typescript +// Augmentations build on standard types, ensuring compatibility +await brain.augment.install('medical-records') // Extends "person" type +await brain.augment.install('financial-analysis') // Extends "transaction" events +await brain.augment.install('social-graph') // Uses "follows", "likes" verbs + +// All augmentations work together seamlessly +const patient = await brain.find("patient with financial transactions who follows Dr. Smith") +``` + +#### 5. **Cross-Platform Integration** +```typescript +// Standard types enable integration with external systems +// CRM understands "person" and "organization" +await brain.sync.salesforce({ + mapping: { + Contact: "person", + Account: "organization", + Opportunity: "event" + } }) -// Any downstream consumer can now retrieve and reason about this entity. -const concept = await brain.get(conceptId) -console.log(concept?.type) // 'thing' -``` - -#### 4. Extensibility Without Forking the Schema - -```typescript -// Subtypes extend a standard NounType for a specific domain — no schema -// migration, no new noun type. The base type stays universally understood. -await brain.add({ data: 'Patient #12345', type: NounType.Person, subtype: 'patient' }) -await brain.add({ data: 'Invoice #4471', type: NounType.Document, subtype: 'invoice' }) -await brain.add({ data: 'Follower edge', type: NounType.Relationship, subtype: 'social-graph' }) - -// Domains that share the base types interoperate even with custom subtypes. -const patients = await brain.find({ type: NounType.Person, subtype: 'patient' }) -``` - -#### 5. Cross-Platform Integration - -```typescript -// Map external systems onto the standard taxonomy. A CRM's Contact/Account/ -// Opportunity become Person/Organization/Event — the same vocabulary everywhere. -const externalRecords = [ - { kind: 'Contact', name: 'Dana Lee', type: NounType.Person }, - { kind: 'Account', name: 'Acme Corp', type: NounType.Organization }, - { kind: 'Opportunity', name: 'Q3 Renewal', type: NounType.Event } -] - -for (const record of externalRecords) { - await brain.add({ - data: record.name, - type: record.type, - metadata: { source: 'crm', externalKind: record.kind } - }) -} +// Project management understands "task" and "project" +await brain.sync.jira({ + mapping: { + Issue: "task", + Epic: "project", + Sprint: "event" + } +}) ``` ### The Network Effect: Brainy as the Universal Knowledge Protocol -Like **HTTP** became the protocol for the web and **TCP/IP** for the internet, Brainy's noun-verb taxonomy aims to be a **Universal Knowledge Protocol**: +Like **HTTP** became the protocol for the web and **TCP/IP** for the internet, Brainy's noun-verb taxonomy is becoming the **Universal Knowledge Protocol**: -- **Learn Once**: Developers learn 42 nouns + 127 verbs, not thousands of bespoke schemas +- **Learn Once**: Developers learn 31 nouns + 40 verbs, not 1000s of schemas - **Build Anywhere**: Tools built for one domain work in others - **Share Everything**: Knowledge graphs are universally shareable -- **Compose Freely**: Subtypes and metadata extend types without schema migrations +- **Compose Freely**: Augmentations compose without conflicts -This isn't just a database — it's a **shared model for how knowledge is represented**. +This isn't just a database—it's a **protocol for how humanity represents knowledge**. ## Overview -Brainy's **Noun-Verb Taxonomy** models data as entities (nouns) and relationships (verbs), creating a semantic knowledge graph that mirrors how humans naturally think about information. +Brainy 2.0 introduces a revolutionary **Noun-Verb Taxonomy** that models data as entities (nouns) and relationships (verbs), creating a semantic knowledge graph that mirrors how humans naturally think about information. ## Why Noun-Verb? Traditional databases force you to think in tables, documents, or nodes. Brainy lets you think naturally: - **Nouns**: Things that exist (people, documents, products, concepts) -- **Verbs**: How things relate (creates, owns, references, related-to) +- **Verbs**: How things relate (creates, owns, references, similar-to) This simple mental model scales from basic storage to complex knowledge graphs while remaining intuitive. @@ -156,78 +119,70 @@ This simple mental model scales from basic storage to complex knowledge graphs w ### Nouns (Entities) -Nouns represent any entity in your system. `add()` takes a single object and returns the new entity's id: +Nouns represent any entity in your system: ```typescript // Add any entity as a noun -const personId = await brain.add({ - data: 'John Smith, Senior Engineer', - type: NounType.Person, - subtype: 'employee', - metadata: { department: 'engineering', skills: ['TypeScript', 'React', 'Node.js'] } +const personId = await brain.add("John Smith, Senior Engineer", { + type: "person", + department: "engineering", + skills: ["TypeScript", "React", "Node.js"] }) -const documentId = await brain.add({ - data: 'Q3 2024 Financial Report', - type: NounType.Document, - subtype: 'report', - metadata: { category: 'financial', confidential: true, created: '2024-10-01' } +const documentId = await brain.add("Q3 2024 Financial Report", { + type: "document", + category: "financial", + confidential: true, + created: "2024-10-01" }) -const conceptId = await brain.add({ - data: 'Machine Learning', - type: NounType.Concept, - metadata: { domain: 'technology', complexity: 'advanced' } +const conceptId = await brain.add("Machine Learning", { + type: "concept", + domain: "technology", + complexity: "advanced" }) ``` #### Noun Properties Every noun automatically gets: -- **Unique ID**: System-generated UUID, or supply your own via `id` -- **Vector Embedding**: `data` is embedded for semantic similarity -- **Metadata**: Flexible, queryable JSON properties -- **Timestamps**: `createdAt` / `updatedAt` tracking -- **Indexing**: Automatic field indexing for `where` filters +- **Unique ID**: System-generated or custom +- **Vector Embedding**: For semantic similarity +- **Metadata**: Flexible JSON properties +- **Timestamps**: Created/updated tracking +- **Indexing**: Automatic field indexing ### Verbs (Relationships) -Verbs define how nouns relate to each other. `relate()` also takes a single object: +Verbs define how nouns relate to each other: ```typescript // Create relationships between entities -await brain.relate({ - from: personId, - to: documentId, - type: VerbType.Creates, - metadata: { role: 'primary_author', contribution: '80%' } +await brain.relate(personId, documentId, "authored", { + role: "primary_author", + contribution: "80%" }) -await brain.relate({ - from: documentId, - to: conceptId, - type: VerbType.Describes, - metadata: { sections: ['methodology', 'results'], depth: 'detailed' } +await brain.relate(documentId, conceptId, "discusses", { + sections: ["methodology", "results"], + depth: "detailed" }) -await brain.relate({ - from: personId, - to: conceptId, - type: VerbType.RelatedTo, - subtype: 'expertise', - metadata: { yearsExperience: 5, certification: 'Advanced ML Certification' } +await brain.relate(personId, conceptId, "expert_in", { + years_experience: 5, + certification: "Advanced ML Certification" }) ``` #### Verb Properties Every verb includes: -- **Source** (`from`): The noun initiating the relationship -- **Target** (`to`): The noun receiving the relationship -- **Type**: The `VerbType` classification -- **Subtype**: Optional per-product sub-classification (fast-path indexed) -- **Metadata**: Relationship-specific queryable data -- **Weight**: Optional relationship strength (0–1) +- **Source**: The noun initiating the relationship +- **Target**: The noun receiving the relationship +- **Type**: The relationship type/name +- **Direction**: Directional or bidirectional +- **Metadata**: Relationship-specific data +- **Strength**: Optional relationship weight ## Benefits @@ -235,104 +190,92 @@ Every verb includes: ```typescript // Think naturally about your data -const taskId = await brain.add({ data: 'Implement payment system', type: NounType.Task }) -const userId = await brain.add({ data: 'Alice Johnson', type: NounType.Person }) -const projectId = await brain.add({ data: 'E-commerce Platform', type: NounType.Project }) +const taskId = await brain.add("Implement payment system") +const userId = await brain.add("Alice Johnson") +const projectId = await brain.add("E-commerce Platform") // Express relationships clearly -await brain.relate({ from: userId, to: taskId, type: VerbType.ParticipatesIn, subtype: 'assignee' }) -await brain.relate({ from: taskId, to: projectId, type: VerbType.PartOf }) -await brain.relate({ from: userId, to: projectId, type: VerbType.ParticipatesIn, subtype: 'manager' }) +await brain.relate(userId, taskId, "assigned_to") +await brain.relate(taskId, projectId, "part_of") +await brain.relate(userId, projectId, "manages") ``` ### 2. Semantic Understanding -The noun-verb model preserves meaning. `find()` accepts a natural-language string or a structured query: +The noun-verb model preserves meaning: ```typescript -// Natural language — embedded and matched semantically -const results = await brain.find({ query: 'tasks for the payment system' }) +// The system understands semantic relationships +const results = await brain.find("tasks assigned to Alice") +// Automatically understands: assigned_to verb + Alice noun -// Structured — type + graph traversal in one call -const aliceTasks = await brain.find({ - type: NounType.Task, - connected: { from: userId, via: VerbType.ParticipatesIn } -}) +const related = await brain.find("people who manage projects with payment tasks") +// Traverses: person -> manages -> project -> contains -> task ``` ### 3. Flexible Schema -No rigid schema requirements — add any type, extend with a `subtype`: +No rigid schema requirements: ```typescript // Add any noun type without schema changes -const sensorId = await brain.add({ - data: 'New IoT Sensor', - type: NounType.Thing, - subtype: 'iot-device', - metadata: { protocol: 'MQTT', location: 'Building A' } +await brain.add("New IoT Sensor", { + type: "device", + protocol: "MQTT", + location: "Building A" }) -const buildingId = await brain.add({ data: 'Building A', type: NounType.Location }) - -// Relationships carry their own structured metadata -await brain.relate({ - from: sensorId, - to: buildingId, - type: VerbType.Measures, - metadata: { metrics: ['temperature', 'humidity'], interval: '5 minutes' } +// Create new relationship types on the fly +await brain.relate(sensorId, buildingId, "monitors", { + metrics: ["temperature", "humidity"], + interval: "5 minutes" }) ``` ### 4. Graph Traversal -Navigate relationships naturally with `connected`: +Navigate relationships naturally: ```typescript -// Find documents reachable from a team via two relationship hops +// Find all documents authored by team members const teamDocs = await brain.find({ - type: NounType.Document, connected: { from: teamId, - via: [VerbType.MemberOf, VerbType.Creates], + through: ["member_of", "authored"], depth: 2 } }) -// Find products two hops out from a user +// Find similar products purchased by similar users const recommendations = await brain.find({ - type: NounType.Product, connected: { from: userId, - via: VerbType.Owns, - depth: 2 + through: ["similar_to", "purchased"], + depth: 2, + type: "product" } }) ``` ### 5. Temporal Relationships -Track how relationships change over time by storing dates in edge metadata: +Track how relationships change over time: ```typescript -await brain.relate({ - from: employeeId, - to: companyId, - type: VerbType.MemberOf, - subtype: 'past-employment', - metadata: { from: '2020-01-01', to: '2023-12-31', position: 'Senior Developer' } +// Relationships with temporal data +await brain.relate(employeeId, companyId, "worked_at", { + from: "2020-01-01", + to: "2023-12-31", + position: "Senior Developer" }) -await brain.relate({ - from: employeeId, - to: newCompanyId, - type: VerbType.MemberOf, - subtype: 'current-employment', - metadata: { from: '2024-01-01', position: 'Tech Lead' } +await brain.relate(employeeId, newCompanyId, "works_at", { + from: "2024-01-01", + position: "Tech Lead" }) -// Query with natural language -const employment = await brain.find({ query: 'where did this person work in 2022' }) +// Query historical relationships +const employment = await brain.find("where did John work in 2022") ``` ## Real-World Use Cases @@ -341,108 +284,97 @@ const employment = await brain.find({ query: 'where did this person work in 2022 ```typescript // Documents and their relationships -const paperId = await brain.add({ - data: 'Neural Networks Paper', - type: NounType.Document, - subtype: 'research-paper', - metadata: { year: 2024 } +const paperId = await brain.add("Neural Networks Paper", { + type: "research_paper", + year: 2024 }) -const authorId = await brain.add({ - data: 'Dr. Sarah Chen', - type: NounType.Person, - subtype: 'researcher' +const authorId = await brain.add("Dr. Sarah Chen", { + type: "researcher" }) -const topicId = await brain.add({ data: 'Deep Learning', type: NounType.Concept }) -const otherPaperId = await brain.add({ data: 'Backpropagation Survey', type: NounType.Document }) +const topicId = await brain.add("Deep Learning", { + type: "topic" +}) // Rich relationship network -await brain.relate({ from: authorId, to: paperId, type: VerbType.Creates }) -await brain.relate({ from: paperId, to: topicId, type: VerbType.Describes }) -await brain.relate({ from: paperId, to: otherPaperId, type: VerbType.References }) -await brain.relate({ from: authorId, to: topicId, type: VerbType.RelatedTo, subtype: 'research-focus' }) +await brain.relate(authorId, paperId, "authored") +await brain.relate(paperId, topicId, "covers") +await brain.relate(paperId, otherPaperId, "cites") +await brain.relate(authorId, topicId, "researches") // Query the knowledge graph -const related = await brain.find({ query: 'papers about deep learning by Sarah Chen' }) +const related = await brain.find("papers about deep learning by Sarah Chen") ``` ### Social Networks ```typescript // Users and connections -const user1 = await brain.add({ data: 'Alice', type: NounType.Person }) -const user2 = await brain.add({ data: 'Bob', type: NounType.Person }) -const post = await brain.add({ data: 'Great article on AI!', type: NounType.Message, subtype: 'post' }) +const user1 = await brain.add("Alice", { type: "user" }) +const user2 = await brain.add("Bob", { type: "user" }) +const post = await brain.add("Great article on AI!", { type: "post" }) // Social interactions -await brain.relate({ from: user1, to: user2, type: VerbType.Follows }) -await brain.relate({ from: user2, to: user1, type: VerbType.Follows }) // mutual -await brain.relate({ from: user1, to: post, type: VerbType.Creates }) -await brain.relate({ from: user2, to: post, type: VerbType.Likes }) -await brain.relate({ from: user2, to: post, type: VerbType.Communicates, subtype: 'share' }) +await brain.relate(user1, user2, "follows") +await brain.relate(user2, user1, "follows") // Mutual +await brain.relate(user1, post, "created") +await brain.relate(user2, post, "liked") +await brain.relate(user2, post, "shared") // Find social patterns -const influencers = await brain.find({ query: 'people who post about AI with many followers' }) +const influencers = await brain.find("users with most followers who post about AI") ``` ### E-commerce ```typescript // Products and purchases -const product = await brain.add({ - data: 'Wireless Headphones', - type: NounType.Product, - metadata: { price: 99.99, category: 'electronics' } +const product = await brain.add("Wireless Headphones", { + type: "product", + price: 99.99, + category: "electronics" }) -const customer = await brain.add({ - data: 'Customer #12345', - type: NounType.Person, - subtype: 'customer', - metadata: { tier: 'premium' } +const customer = await brain.add("Customer #12345", { + type: "customer", + tier: "premium" }) -// Purchase and review relationships -await brain.relate({ - from: customer, - to: product, - type: VerbType.Owns, - subtype: 'purchase', - metadata: { date: '2024-01-15', quantity: 1, price: 99.99 } +// Purchase relationships +await brain.relate(customer, product, "purchased", { + date: "2024-01-15", + quantity: 1, + price: 99.99 }) -await brain.relate({ - from: customer, - to: product, - type: VerbType.Evaluates, - subtype: 'review', - metadata: { rating: 5, text: 'Excellent sound quality!' } +await brain.relate(customer, product, "reviewed", { + rating: 5, + text: "Excellent sound quality!" }) // Recommendation queries -const recs = await brain.find({ query: 'products bought by customers who bought headphones' }) +const recs = await brain.find("products purchased by customers who bought headphones") ``` ### Project Management ```typescript // Projects, tasks, and teams -const project = await brain.add({ data: 'Website Redesign', type: NounType.Project }) -const task = await brain.add({ data: 'Update homepage', type: NounType.Task }) -const otherTask = await brain.add({ data: 'Design system audit', type: NounType.Task }) -const developer = await brain.add({ data: 'Jane Developer', type: NounType.Person, subtype: 'employee' }) -const designer = await brain.add({ data: 'John Designer', type: NounType.Person, subtype: 'employee' }) +const project = await brain.add("Website Redesign", { type: "project" }) +const task = await brain.add("Update homepage", { type: "task" }) +const developer = await brain.add("Jane Developer", { type: "person" }) +const designer = await brain.add("John Designer", { type: "person" }) // Work relationships -await brain.relate({ from: task, to: project, type: VerbType.PartOf }) -await brain.relate({ from: developer, to: task, type: VerbType.ParticipatesIn, subtype: 'assignee' }) -await brain.relate({ from: designer, to: developer, type: VerbType.WorksWith }) -await brain.relate({ from: task, to: otherTask, type: VerbType.DependsOn }) +await brain.relate(task, project, "belongs_to") +await brain.relate(developer, task, "assigned_to") +await brain.relate(designer, developer, "collaborates_with") +await brain.relate(task, otherTask, "depends_on") // Project queries -const blockers = await brain.find({ query: 'tasks blocked by incomplete work' }) -const workload = await brain.find({ query: 'people assigned to multiple active projects' }) +const blockers = await brain.find("tasks that depend on incomplete tasks") +const workload = await brain.find("people assigned to multiple active projects") ``` ## Advanced Patterns @@ -450,56 +382,54 @@ const workload = await brain.find({ query: 'people assigned to multiple active p ### Bidirectional Relationships ```typescript -// Symmetric relationships create the inverse edge automatically -await brain.relate({ from: user1, to: user2, type: VerbType.FriendOf, bidirectional: true }) +// Some relationships are naturally bidirectional +await brain.relate(user1, user2, "friend_of", { bidirectional: true }) +// Automatically creates inverse relationship ``` ### Weighted Relationships ```typescript -// Add strength/weight to relationships (top-level weight, 0–1) -await brain.relate({ - from: doc1, - to: doc2, - type: VerbType.SimilarityDegree, - weight: 0.95, - metadata: { algorithm: 'cosine' } +// Add strength/weight to relationships +await brain.relate(doc1, doc2, "similar_to", { + similarity_score: 0.95, + algorithm: "cosine" }) -// Weights come back on the Relation, so you can filter on them -const edges = await brain.related({ from: doc1, type: VerbType.SimilarityDegree }) -const stronglyRelated = edges.filter((edge) => (edge.weight ?? 0) >= 0.8) -``` - -### Relationship Chains (Multi-hop) - -```typescript -// Follow a chain of relationship types out to a fixed depth. -// `via` accepts an array of VerbTypes; `depth` bounds the traversal. -const results = await brain.find({ - type: NounType.Thing, +// Use weights in queries +const stronglyRelated = await brain.find({ connected: { - from: userId, - via: [VerbType.Owns, VerbType.Creates, VerbType.Uses], - depth: 3 + type: "similar_to", + minWeight: 0.8 } }) -// Finds: things used by products made by companies owned by the user +``` + +### Relationship Chains + +```typescript +// Follow relationship chains +const results = await brain.find({ + connected: { + from: userId, + chain: [ + { type: "owns", to: "company" }, + { type: "produces", to: "product" }, + { type: "uses", to: "technology" } + ] + } +}) +// Finds: technologies used by products made by companies owned by user ``` ### Meta-Relationships -Relationships can themselves be reasoned about. The `Relationship` NounType reifies an edge as a first-class entity, and the meta-level verbs (`Endorses`, `Supports`, `Contradicts`, `Supersedes`) express second-order claims between entities: - ```typescript -// A second person endorses a claim, and a third supports it with evidence. -const claim = await brain.add({ data: 'X improves retention', type: NounType.Proposition }) -await brain.relate({ from: user2, to: claim, type: VerbType.Endorses }) -await brain.relate({ - from: user3, - to: claim, - type: VerbType.Supports, - metadata: { reason: 'Matches the A/B test', trustScore: 0.9 } +// Relationships about relationships +const verbId = await brain.relate(user1, user2, "recommends") +await brain.relate(user3, verbId, "endorses", { + reason: "Accurate recommendation", + trust_score: 0.9 }) ``` @@ -509,1048 +439,1121 @@ await brain.relate({ ```typescript // By type -const people = await brain.find({ type: NounType.Person }) +const people = await brain.find({ where: { type: "person" } }) -// By type + metadata filters (bare operators — no `$` prefixes) +// By properties const documents = await brain.find({ - type: NounType.Document, where: { + type: "document", confidential: false, - created: { gte: '2024-01-01' } + created: { $gte: "2024-01-01" } } }) -// By semantic similarity — use `query`, optionally narrowed by type +// By similarity const similar = await brain.find({ - query: 'machine learning research', - type: NounType.Document + like: "machine learning research", + where: { type: "document" } }) ``` -> **`where` operators** are bare (never dollar-prefixed): `eq`/`equals`/`is`, `ne`/`notEquals`, `in`/`oneOf`, `gt`/`greaterThan`, `gte`/`greaterThanOrEqual`, `lt`/`lessThan`, `lte`/`lessThanOrEqual`, `between`, `contains`, `exists`, `missing`, plus the logical combinators `allOf`/`anyOf`/`not`. - -### Finding Verbs (Relationships) +### Finding Verbs ```typescript -// All relationships originating from a noun -const outgoing = await brain.related({ from: nounId }) +// Get all relationships for a noun +const relationships = await brain.getVerbs(nounId) -// Every edge touching a noun, in either direction -const incident = await brain.related({ node: nounId }) +// Find specific relationship types +const authorships = await brain.find({ + verb: { + type: "authored", + from: authorId + } +}) -// Filter by relationship type -const authorships = await brain.related({ from: authorId, type: VerbType.Creates }) - -// Filter returned relationships by their metadata (Relation carries `.metadata`) -const purchases = await brain.related({ from: customerId, type: VerbType.Owns, subtype: 'purchase' }) -const recentPurchases = purchases.filter((edge) => edge.metadata?.date >= '2024-01-01') - -// Just the count of relationships in the graph -const totalEdges = await brain.getVerbCount() -``` - -### Combined Queries (Query → Expand) - -```typescript -// Start from a semantic query, then expand along the graph. -// Vector + graph in a single find() call. -const results = await brain.find({ - query: 'AI research', - connected: { - via: VerbType.Creates, - depth: 2 +// Find by relationship properties +const recentPurchases = await brain.find({ + verb: { + type: "purchased", + where: { + date: { $gte: "2024-01-01" } + } } }) ``` -## The Complete Noun Taxonomy (42 Types) - -`NounType` is the stable, exported vocabulary for classifying entities. Every value is a plain string, so you can write `NounType.Person` or the literal `'person'`. Pick the closest standard type and refine with `subtype` and `metadata`. +### Combined Queries ```typescript -const physicistId = await brain.add({ - data: 'Albert Einstein', - type: NounType.Person, - metadata: { role: 'physicist', born: '1879-03-14' } +// Find nouns through relationships +const results = await brain.find({ + // Start with similar documents + like: "AI research", + // That are authored by + connected: { + through: "authored", + // People who work at + where: { + connected: { + to: "Stanford", + type: "works_at" + } + } + } }) ``` -### Core Entity Types (7) +## Performance Optimizations -| NounType | Value | Use for | -|----------|-------|---------| -| `Person` | `'person'` | Individual human entities | -| `Organization` | `'organization'` | Companies, institutions, collectives | -| `Location` | `'location'` | Geographic and named spatial entities | -| `Thing` | `'thing'` | Discrete physical objects and artifacts | -| `Concept` | `'concept'` | Abstract ideas, principles, intangibles | -| `Event` | `'event'` | Temporal occurrences and happenings | -| `Agent` | `'agent'` | Non-human autonomous actors (AI agents, bots) | +### Noun Indexing +- Automatic vector indexing for similarity +- Field indexing for metadata queries +- Full-text indexing for content search -### Biological & Material Types (2) +### Verb Indexing +- Relationship type indexing +- Source/target indexing +- Temporal indexing for time-based queries -| NounType | Value | Use for | -|----------|-------|---------| -| `Organism` | `'organism'` | Living biological entities (animals, plants, bacteria) | -| `Substance` | `'substance'` | Physical materials and matter (water, iron, DNA) | +### Query Optimization +- Automatic query plan optimization +- Parallel execution of independent operations +- Result caching for repeated queries -### Property, Temporal & Functional Types (3) +## Best Practices -| NounType | Value | Use for | -|----------|-------|---------| -| `Quality` | `'quality'` | Properties and attributes that inhere in entities | -| `TimeInterval` | `'timeInterval'` | Temporal regions, periods, durations | -| `Function` | `'function'` | Purposes, capabilities, functional roles | +1. **Use Descriptive Types**: Make noun and verb types self-documenting +2. **Rich Metadata**: Include relevant metadata for better querying +3. **Consistent Naming**: Use consistent verb names across your application +4. **Temporal Data**: Include timestamps for time-based analysis +5. **Bidirectional When Appropriate**: Mark symmetric relationships as bidirectional -### Informational Type (1) - -| NounType | Value | Use for | -|----------|-------|---------| -| `Proposition` | `'proposition'` | Statements, claims, assertions, declarative content | - -### Digital/Content Types (4) - -| NounType | Value | Use for | -|----------|-------|---------| -| `Document` | `'document'` | Text-based files and written content | -| `Media` | `'media'` | Non-text media (audio, video, images) | -| `File` | `'file'` | Generic digital files and data blobs | -| `Message` | `'message'` | Communication content and correspondence | - -### Collection Types (2) - -| NounType | Value | Use for | -|----------|-------|---------| -| `Collection` | `'collection'` | Groups and sets of items | -| `Dataset` | `'dataset'` | Structured data collections and databases | - -### Business/Application Types (4) - -| NounType | Value | Use for | -|----------|-------|---------| -| `Product` | `'product'` | Commercial products and offerings | -| `Service` | `'service'` | Service offerings and intangible products | -| `Task` | `'task'` | Actions, todos, work items | -| `Project` | `'project'` | Organized initiatives and programs | - -### Descriptive Types (6) - -| NounType | Value | Use for | -|----------|-------|---------| -| `Process` | `'process'` | Workflows, procedures, ongoing activities | -| `State` | `'state'` | Conditions, status, situational contexts | -| `Role` | `'role'` | Positions, responsibilities, classifications | -| `Language` | `'language'` | Natural and formal languages | -| `Currency` | `'currency'` | Monetary units and exchange mediums | -| `Measurement` | `'measurement'` | Metrics, quantities, measured values | - -### Scientific & Legal Types (4) - -| NounType | Value | Use for | -|----------|-------|---------| -| `Hypothesis` | `'hypothesis'` | Scientific theories, research hypotheses | -| `Experiment` | `'experiment'` | Controlled studies, trials, methodologies | -| `Contract` | `'contract'` | Legal agreements, terms, binding documents | -| `Regulation` | `'regulation'` | Laws, rules, compliance requirements | - -### Technical Infrastructure Types (2) - -| NounType | Value | Use for | -|----------|-------|---------| -| `Interface` | `'interface'` | APIs, protocols, specifications, endpoints | -| `Resource` | `'resource'` | Compute, bandwidth, storage, infrastructure assets | - -### Social Structure Types (3) - -| NounType | Value | Use for | -|----------|-------|---------| -| `SocialGroup` | `'socialGroup'` | Informal social groups and collectives | -| `Institution` | `'institution'` | Formal social structures and practices | -| `Norm` | `'norm'` | Social norms, conventions, expectations | - -### Information Theory Types (2) - -| NounType | Value | Use for | -|----------|-------|---------| -| `InformationContent` | `'informationContent'` | Abstract information (stories, ideas, schemas) | -| `InformationBearer` | `'informationBearer'` | Physical or digital carrier of information | - -### Meta-Level & Extensible Types (2) - -| NounType | Value | Use for | -|----------|-------|---------| -| `Relationship` | `'relationship'` | Relationships reified as first-class entities | -| `Custom` | `'custom'` | Domain-specific entities outside the standard set | - -## The Complete Verb Taxonomy (127 Types) - -`VerbType` is the exported vocabulary for classifying relationships. As with nouns, every value is a plain string — write `VerbType.Creates` or `'creates'`. Where no verb is an exact fit, choose the closest base verb and refine it with `subtype` and `metadata` (see [Coverage Completeness](#coverage-completeness-analysis)). +## Migration from Traditional Models +### From Relational (SQL) ```typescript -await brain.relate({ from: authorId, to: documentId, type: VerbType.Creates }) +// Instead of JOIN queries +// SELECT * FROM users JOIN orders ON users.id = orders.user_id + +// Use noun-verb relationships +const userId = await brain.add("User", userData) +const orderId = await brain.add("Order", orderData) +await brain.relate(userId, orderId, "placed") + +// Query naturally +const userOrders = await brain.find({ + connected: { from: userId, type: "placed" } +}) ``` -### Foundational Ontological (3) +### From Document (NoSQL) +```typescript +// Instead of embedded documents +// { user: { orders: [...] } } -| VerbType | Value | Meaning | -|----------|-------|---------| -| `InstanceOf` | `'instanceOf'` | Individual to class (Fido instanceOf Dog) | -| `SubclassOf` | `'subclassOf'` | Taxonomic hierarchy (Dog subclassOf Mammal) | -| `ParticipatesIn` | `'participatesIn'` | Entity participation in events/processes | +// Use explicit relationships +const userId = await brain.add("User", userData) +for (const order of orders) { + const orderId = await brain.add("Order", order) + await brain.relate(userId, orderId, "has_order") +} +``` -### Core Relationships (4) +### From Graph Databases +```typescript +// Similar to graph databases but with added benefits: +// 1. Automatic vector embeddings for similarity +// 2. Natural language querying +// 3. Unified with metadata filtering -| VerbType | Value | Meaning | -|----------|-------|---------| -| `RelatedTo` | `'relatedTo'` | Generic relationship (fallback) | -| `Contains` | `'contains'` | Containment relationship | -| `PartOf` | `'partOf'` | Part-whole (mereological) relationship | -| `References` | `'references'` | Citation and referential relationship | +// Enhanced graph queries +const results = await brain.find("similar users who purchased similar products") +``` -### Spatial (2) +## Universal Knowledge Coverage -| VerbType | Value | Meaning | -|----------|-------|---------| -| `LocatedAt` | `'locatedAt'` | Spatial location relationship | -| `AdjacentTo` | `'adjacentTo'` | Spatial proximity relationship | +The Noun-Verb taxonomy is designed to represent **all human knowledge** through a comprehensive set of types that can be combined infinitely. -### Temporal (3) +### Complete Noun Types (31 Types) -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Precedes` | `'precedes'` | Temporal sequence (before) | -| `During` | `'during'` | Temporal containment | -| `OccursAt` | `'occursAt'` | Temporal location | +#### Core Entity Types (6) -### Causal & Dependency (5) +##### 1. **Person** - Individual human entities +```typescript +await brain.add("Albert Einstein", { + type: "person", + role: "physicist", + born: "1879-03-14" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Causes` | `'causes'` | Direct causal relationship | -| `Enables` | `'enables'` | Enablement without direct causation | -| `Prevents` | `'prevents'` | Prevention relationship | -| `DependsOn` | `'dependsOn'` | Dependency relationship | -| `Requires` | `'requires'` | Necessity relationship | +##### 2. **Organization** - Collective entities +```typescript +await brain.add("OpenAI", { + type: "organization", + industry: "AI research", + founded: 2015 +}) +``` -### Creation & Transformation (5) +##### 3. **Location** - Geographic and spatial entities +```typescript +await brain.add("San Francisco", { + type: "location", + category: "city", + coordinates: [37.7749, -122.4194] +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Creates` | `'creates'` | Creation relationship | -| `Transforms` | `'transforms'` | Transformation relationship | -| `Becomes` | `'becomes'` | State change relationship | -| `Modifies` | `'modifies'` | Modification relationship | -| `Consumes` | `'consumes'` | Consumption relationship | +##### 4. **Thing** - Physical objects +```typescript +await brain.add("Tesla Model 3", { + type: "thing", + category: "vehicle", + manufacturer: "Tesla" +}) +``` -### Lifecycle (1) +##### 5. **Concept** - Abstract ideas and intangibles +```typescript +await brain.add("Machine Learning", { + type: "concept", + domain: "technology", + complexity: "advanced" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Destroys` | `'destroys'` | Termination and destruction relationship | +##### 6. **Event** - Temporal occurrences +```typescript +await brain.add("Product Launch 2024", { + type: "event", + date: "2024-09-15", + attendees: 500 +}) +``` -### Ownership & Attribution (2) +#### Digital/Content Types (5) -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Owns` | `'owns'` | Ownership relationship | -| `AttributedTo` | `'attributedTo'` | Attribution relationship | +##### 7. **Document** - Text-based files +```typescript +await brain.add("Quarterly Report", { + type: "document", + format: "PDF", + pages: 47 +}) +``` -### Property & Quality (2) +##### 8. **Media** - Non-text media files +```typescript +await brain.add("Product Demo Video", { + type: "media", + format: "MP4", + duration: "5:30" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `HasQuality` | `'hasQuality'` | Entity to quality attribution | -| `Realizes` | `'realizes'` | Function realization relationship | +##### 9. **File** - Generic digital files +```typescript +await brain.add("config.json", { + type: "file", + size: "2KB", + modified: Date.now() +}) +``` -### Effects & Experience (1) +##### 10. **Message** - Communication content +```typescript +await brain.add("Support ticket #1234", { + type: "message", + priority: "high", + channel: "email" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Affects` | `'affects'` | Patient/experiencer relationship | +##### 11. **Content** - Generic content +```typescript +await brain.add("Landing page copy", { + type: "content", + category: "marketing", + language: "en" +}) +``` -### Composition (2) +#### Collection Types (2) -| VerbType | Value | Meaning | -|----------|-------|---------| -| `ComposedOf` | `'composedOf'` | Material composition (distinct from partOf) | -| `Inherits` | `'inherits'` | Inheritance relationship | +##### 12. **Collection** - Groups of items +```typescript +await brain.add("Premium Features", { + type: "collection", + items: 25, + category: "features" +}) +``` -### Social & Organizational (8) +##### 13. **Dataset** - Structured data collections +```typescript +await brain.add("Customer Analytics", { + type: "dataset", + records: 10000, + schema: "v2" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `MemberOf` | `'memberOf'` | Membership relationship | -| `WorksWith` | `'worksWith'` | Professional collaboration | -| `FriendOf` | `'friendOf'` | Friendship relationship | -| `Follows` | `'follows'` | Following/subscription relationship | -| `Likes` | `'likes'` | Liking/favoriting relationship | -| `ReportsTo` | `'reportsTo'` | Hierarchical reporting relationship | -| `Mentors` | `'mentors'` | Mentorship relationship | -| `Communicates` | `'communicates'` | Communication relationship | +#### Business/Application Types (5) -### Descriptive & Functional (8) +##### 14. **Product** - Commercial offerings +```typescript +await brain.add("Pro Subscription", { + type: "product", + price: 99.99, + tier: "premium" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Describes` | `'describes'` | Descriptive relationship | -| `Defines` | `'defines'` | Definition relationship | -| `Categorizes` | `'categorizes'` | Categorization relationship | -| `Measures` | `'measures'` | Measurement relationship | -| `Evaluates` | `'evaluates'` | Evaluation relationship | -| `Uses` | `'uses'` | Utilization relationship | -| `Implements` | `'implements'` | Implementation relationship | -| `Extends` | `'extends'` | Extension relationship | +##### 15. **Service** - Service offerings +```typescript +await brain.add("Cloud Hosting", { + type: "service", + sla: "99.9%", + region: "us-west" +}) +``` -### Advanced Relationships (5) +##### 16. **User** - User accounts +```typescript +await brain.add("user@example.com", { + type: "user", + tier: "enterprise", + created: Date.now() +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `EquivalentTo` | `'equivalentTo'` | Equivalence/identity relationship | -| `Believes` | `'believes'` | Epistemic relationship (cognitive state) | -| `Conflicts` | `'conflicts'` | Conflict relationship | -| `Synchronizes` | `'synchronizes'` | Synchronization relationship | -| `Competes` | `'competes'` | Competition relationship | +##### 17. **Task** - Actions and todos +```typescript +await brain.add("Deploy v2.0", { + type: "task", + priority: "high", + assignee: "devops" +}) +``` -### Modal (6) +##### 18. **Project** - Organized initiatives +```typescript +await brain.add("Website Redesign", { + type: "project", + deadline: "2024-12-31", + status: "active" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `CanCause` | `'canCause'` | Potential causation (possibility) | -| `MustCause` | `'mustCause'` | Necessary causation (necessity) | -| `WouldCauseIf` | `'wouldCauseIf'` | Counterfactual causation | -| `CouldBe` | `'couldBe'` | Possible states | -| `MustBe` | `'mustBe'` | Necessary identity | -| `Counterfactual` | `'counterfactual'` | General counterfactual relationship | +#### Descriptive Types (7) -### Epistemic States (9) +##### 19. **Process** - Workflows and procedures +```typescript +await brain.add("CI/CD Pipeline", { + type: "process", + steps: 7, + automated: true +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Knows` | `'knows'` | Knowledge (justified true belief) | -| `Doubts` | `'doubts'` | Uncertainty/skepticism | -| `Desires` | `'desires'` | Want/preference | -| `Intends` | `'intends'` | Intentionality | -| `Fears` | `'fears'` | Fear/anxiety | -| `Loves` | `'loves'` | Strong positive emotional attitude | -| `Hates` | `'hates'` | Strong negative emotional attitude | -| `Hopes` | `'hopes'` | Hopeful expectation | -| `Perceives` | `'perceives'` | Sensory perception | +##### 20. **State** - Conditions or status +```typescript +await brain.add("System Health", { + type: "state", + status: "operational", + uptime: "99.99%" +}) +``` -### Learning & Cognition (1) +##### 21. **Role** - Positions or responsibilities +```typescript +await brain.add("Admin Role", { + type: "role", + permissions: ["read", "write", "delete"], + level: "superuser" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Learns` | `'learns'` | Cognitive acquisition and learning | +##### 22. **Topic** - Subjects or themes +```typescript +await brain.add("Machine Learning", { + type: "topic", + field: "AI", + popularity: "high" +}) +``` -### Uncertainty & Probability (4) +##### 23. **Language** - Languages or linguistic entities +```typescript +await brain.add("English", { + type: "language", + iso_code: "en", + speakers_millions: 1500 +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `ProbablyCauses` | `'probablyCauses'` | Probabilistic causation | -| `UncertainRelation` | `'uncertainRelation'` | Unknown relationship with confidence bounds | -| `CorrelatesWith` | `'correlatesWith'` | Statistical correlation (not causation) | -| `ApproximatelyEquals` | `'approximatelyEquals'` | Fuzzy equivalence | +##### 24. **Currency** - Monetary units +```typescript +await brain.add("US Dollar", { + type: "currency", + symbol: "$", + code: "USD" +}) +``` -### Scalar Properties (5) +##### 25. **Measurement** - Metrics or quantities +```typescript +await brain.add("Temperature Reading", { + type: "measurement", + value: 23.5, + unit: "celsius" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `GreaterThan` | `'greaterThan'` | Scalar comparison | -| `SimilarityDegree` | `'similarityDegree'` | Graded similarity | -| `MoreXThan` | `'moreXThan'` | Comparative property | -| `HasDegree` | `'hasDegree'` | Scalar property assignment | -| `PartiallyHas` | `'partiallyHas'` | Graded possession | +#### Scientific/Research Types (2) -### Information Theory (2) +##### 26. **Hypothesis** - Scientific theories and propositions +```typescript +await brain.add("String Theory", { + type: "hypothesis", + field: "physics", + status: "unproven" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Carries` | `'carries'` | Bearer carries content | -| `Encodes` | `'encodes'` | Encoding relationship | +##### 27. **Experiment** - Studies and research trials +```typescript +await brain.add("Clinical Trial XYZ", { + type: "experiment", + phase: 3, + participants: 1000 +}) +``` -### Deontic (5) +#### Legal/Regulatory Types (2) -| VerbType | Value | Meaning | -|----------|-------|---------| -| `ObligatedTo` | `'obligatedTo'` | Moral/legal obligation | -| `PermittedTo` | `'permittedTo'` | Permission/authorization | -| `ProhibitedFrom` | `'prohibitedFrom'` | Prohibition/forbidden | -| `ShouldDo` | `'shouldDo'` | Normative expectation | -| `MustNotDo` | `'mustNotDo'` | Strong prohibition | +##### 28. **Contract** - Legal agreements and terms +```typescript +await brain.add("Service Agreement", { + type: "contract", + duration: "2 years", + value: 100000 +}) +``` -### Context & Perspective (5) +##### 29. **Regulation** - Laws and compliance requirements +```typescript +await brain.add("GDPR", { + type: "regulation", + jurisdiction: "EU", + category: "data protection" +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `TrueInContext` | `'trueInContext'` | Context-dependent truth | -| `PerceivedAs` | `'perceivedAs'` | Subjective perception | -| `InterpretedAs` | `'interpretedAs'` | Interpretation relationship | -| `ValidInFrame` | `'validInFrame'` | Frame-dependent validity | -| `TrueFrom` | `'trueFrom'` | Perspective-dependent truth | +#### Technical Infrastructure Types (2) -### Advanced Temporal (6) +##### 30. **Interface** - APIs and protocols +```typescript +await brain.add("REST API", { + type: "interface", + version: "v2", + endpoints: 45 +}) +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Overlaps` | `'overlaps'` | Partial temporal overlap | -| `ImmediatelyAfter` | `'immediatelyAfter'` | Direct temporal succession | -| `EventuallyLeadsTo` | `'eventuallyLeadsTo'` | Long-term consequence | -| `SimultaneousWith` | `'simultaneousWith'` | Exact temporal alignment | -| `HasDuration` | `'hasDuration'` | Temporal extent | -| `RecurringWith` | `'recurringWith'` | Cyclic temporal relationship | +##### 31. **Resource** - Infrastructure and compute assets +```typescript +await brain.add("Database Server", { + type: "resource", + capacity: "1TB", + availability: "99.9%" +}) +``` -### Advanced Spatial (9) +### Complete Verb Types (40 Types) -| VerbType | Value | Meaning | -|----------|-------|---------| -| `ContainsSpatially` | `'containsSpatially'` | Spatial containment | -| `OverlapsSpatially` | `'overlapsSpatially'` | Spatial overlap | -| `Surrounds` | `'surrounds'` | Encirclement | -| `ConnectedTo` | `'connectedTo'` | Topological connection | -| `Above` | `'above'` | Vertical (superior position) | -| `Below` | `'below'` | Vertical (inferior position) | -| `Inside` | `'inside'` | Within containment boundaries | -| `Outside` | `'outside'` | Beyond containment boundaries | -| `Facing` | `'facing'` | Directional orientation | +#### Core Relationship Types (5) -### Social Structures (5) +##### 1. **RelatedTo** - Generic relationship (default) +```typescript +await brain.relate(entityA, entityB, "relatedTo") +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Represents` | `'represents'` | Representative relationship | -| `Embodies` | `'embodies'` | Exemplification or personification | -| `Opposes` | `'opposes'` | Opposition relationship | -| `AlliesWith` | `'alliesWith'` | Alliance relationship | -| `ConformsTo` | `'conformsTo'` | Norm conformity | +##### 2. **Contains** - Containment relationship +```typescript +await brain.relate(folderId, fileId, "contains") +``` -### Measurement (4) +##### 3. **PartOf** - Part-whole relationship +```typescript +await brain.relate(componentId, systemId, "partOf") +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `MeasuredIn` | `'measuredIn'` | Unit relationship | -| `ConvertsTo` | `'convertsTo'` | Unit conversion | -| `HasMagnitude` | `'hasMagnitude'` | Quantitative value | -| `DimensionallyEquals` | `'dimensionallyEquals'` | Dimensional analysis | +##### 4. **LocatedAt** - Spatial relationship +```typescript +await brain.relate(deviceId, locationId, "locatedAt") +``` -### Change & Persistence (4) +##### 5. **References** - Citation relationship +```typescript +await brain.relate(paperId, sourceId, "references") +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `PersistsThrough` | `'persistsThrough'` | Persistence through change | -| `GainsProperty` | `'gainsProperty'` | Property acquisition | -| `LosesProperty` | `'losesProperty'` | Property loss | -| `RemainsSame` | `'remainsSame'` | Identity through time | +#### Temporal/Causal Types (5) -### Parthood Variations (4) +##### 6. **Precedes** - Temporal sequence (before) +```typescript +await brain.relate(event1Id, event2Id, "precedes") +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `FunctionalPartOf` | `'functionalPartOf'` | Functional component | -| `TopologicalPartOf` | `'topologicalPartOf'` | Spatial part | -| `TemporalPartOf` | `'temporalPartOf'` | Temporal slice | -| `ConceptualPartOf` | `'conceptualPartOf'` | Abstract decomposition | +##### 7. **Succeeds** - Temporal sequence (after) +```typescript +await brain.relate(event2Id, event1Id, "succeeds") +``` -### Dependency Variations (3) +##### 8. **Causes** - Causal relationship +```typescript +await brain.relate(actionId, effectId, "causes") +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `RigidlyDependsOn` | `'rigidlyDependsOn'` | Necessary dependency | -| `FunctionallyDependsOn` | `'functionallyDependsOn'` | Operational dependency | -| `HistoricallyDependsOn` | `'historicallyDependsOn'` | Causal history dependency | +##### 9. **DependsOn** - Dependency relationship +```typescript +await brain.relate(moduleId, libraryId, "dependsOn") +``` -### Meta-Level (4) +##### 10. **Requires** - Necessity relationship +```typescript +await brain.relate(taskId, resourceId, "requires") +``` -| VerbType | Value | Meaning | -|----------|-------|---------| -| `Endorses` | `'endorses'` | Second-order validation | -| `Contradicts` | `'contradicts'` | Logical contradiction | -| `Supports` | `'supports'` | Evidential support | -| `Supersedes` | `'supersedes'` | Replacement relationship | +#### Creation/Transformation Types (5) + +##### 11. **Creates** - Creation relationship +```typescript +await brain.relate(authorId, documentId, "creates") +``` + +##### 12. **Transforms** - Transformation relationship +```typescript +await brain.relate(processId, dataId, "transforms") +``` + +##### 13. **Becomes** - State change relationship +```typescript +await brain.relate(caterpillarId, butterflyId, "becomes") +``` + +##### 14. **Modifies** - Modification relationship +```typescript +await brain.relate(editorId, fileId, "modifies") +``` + +##### 15. **Consumes** - Consumption relationship +```typescript +await brain.relate(processId, resourceId, "consumes") +``` + +#### Ownership/Attribution Types (4) + +##### 16. **Owns** - Ownership relationship +```typescript +await brain.relate(userId, assetId, "owns") +``` + +##### 17. **AttributedTo** - Attribution relationship +```typescript +await brain.relate(quoteId, authorId, "attributedTo") +``` + +##### 18. **CreatedBy** - Creation attribution +```typescript +await brain.relate(productId, teamId, "createdBy") +``` + +##### 19. **BelongsTo** - Belonging relationship +```typescript +await brain.relate(itemId, collectionId, "belongsTo") +``` + +#### Social/Organizational Types (9) + +##### 20. **MemberOf** - Membership relationship +```typescript +await brain.relate(userId, organizationId, "memberOf") +``` + +##### 21. **WorksWith** - Professional relationship +```typescript +await brain.relate(employee1Id, employee2Id, "worksWith") +``` + +##### 22. **FriendOf** - Friendship relationship +```typescript +await brain.relate(user1Id, user2Id, "friendOf") +``` + +##### 23. **Follows** - Following relationship +```typescript +await brain.relate(followerId, influencerId, "follows") +``` + +##### 24. **Likes** - Liking relationship +```typescript +await brain.relate(userId, postId, "likes") +``` + +##### 25. **ReportsTo** - Reporting relationship +```typescript +await brain.relate(employeeId, managerId, "reportsTo") +``` + +##### 26. **Supervises** - Supervisory relationship +```typescript +await brain.relate(managerId, employeeId, "supervises") +``` + +##### 27. **Mentors** - Mentorship relationship +```typescript +await brain.relate(seniorId, juniorId, "mentors") +``` + +##### 28. **Communicates** - Communication relationship +```typescript +await brain.relate(sender, receiver, "communicates") +``` + +#### Descriptive/Functional Types (8) + +##### 29. **Describes** - Descriptive relationship +```typescript +await brain.relate(documentId, conceptId, "describes") +``` + +##### 30. **Defines** - Definition relationship +```typescript +await brain.relate(glossaryId, termId, "defines") +``` + +##### 31. **Categorizes** - Categorization relationship +```typescript +await brain.relate(taxonomyId, itemId, "categorizes") +``` + +##### 32. **Measures** - Measurement relationship +```typescript +await brain.relate(sensorId, metricId, "measures") +``` + +##### 33. **Evaluates** - Evaluation relationship +```typescript +await brain.relate(reviewerId, proposalId, "evaluates") +``` + +##### 34. **Uses** - Utilization relationship +```typescript +await brain.relate(applicationId, libraryId, "uses") +``` + +##### 35. **Implements** - Implementation relationship +```typescript +await brain.relate(classId, interfaceId, "implements") +``` + +##### 36. **Extends** - Extension relationship +```typescript +await brain.relate(childClassId, parentClassId, "extends") +``` + +#### Enhanced Relationships (4) + +##### 37. **Inherits** - Inheritance relationship +```typescript +await brain.relate(childId, parentId, "inherits") +``` + +##### 38. **Conflicts** - Conflict relationship +```typescript +await brain.relate(policy1Id, policy2Id, "conflicts") +``` + +##### 39. **Synchronizes** - Synchronization relationship +```typescript +await brain.relate(service1Id, service2Id, "synchronizes") +``` + +##### 40. **Competes** - Competition relationship +```typescript +await brain.relate(company1Id, company2Id, "competes") +``` ## Coverage Completeness Analysis -### Is Anything Missing? +### Is Anything Missing? -The taxonomy is intentionally bounded. When no type is an exact fit, three escape hatches keep it complete: +While we could add more specific verb types (like "approves", "delegates", "shares"), our current taxonomy is **mathematically complete** for several reasons: #### 1. Generic Fallbacks +- **`Custom` noun type**: For any entity that doesn't fit standard types +- **`RelatedTo` verb type**: For any relationship not explicitly defined +- **Unlimited metadata**: Any additional semantics via properties -- **`Custom` noun type**: For any entity that doesn't fit a standard type -- **`RelatedTo` verb type**: For any relationship not explicitly named -- **`subtype` + metadata**: Refine a base type with domain-specific semantics +#### 2. Semantic Flexibility Through Metadata -#### 2. Semantic Flexibility Through Subtype & Metadata - -Instead of adding ever more verb types, refine a base verb with a `subtype` and structured metadata: +Instead of adding dozens more verb types, we use metadata for specificity: ```typescript -// "approves" → Evaluates with a result -await brain.relate({ - from: managerId, - to: requestId, - type: VerbType.Evaluates, - subtype: 'approval', - metadata: { result: 'approved', timestamp: Date.now() } +// Instead of adding "approves" verb: +await brain.relate(managerId, requestId, "evaluates", { + result: "approved", + timestamp: Date.now() }) -// "shares" → Communicates with an action -await brain.relate({ - from: userId, - to: documentId, - type: VerbType.Communicates, - subtype: 'share', - metadata: { permissions: 'read-only' } +// Instead of adding "shares" verb: +await brain.relate(userId, documentId, "communicates", { + action: "shared", + permissions: "read-only" }) -// "delegates" → ParticipatesIn with a role + delegation metadata -await brain.relate({ - from: employeeId, - to: taskId, - type: VerbType.ParticipatesIn, - subtype: 'delegate', - metadata: { delegatedBy: managerId, authority: 'full' } +// Instead of adding "delegates" verb: +await brain.relate(managerId, taskId, "creates", { + delegatedTo: employeeId, + authority: "full" }) ``` #### 3. Edge Cases Are Covered -Even exotic scenarios fit the standard types: +Even exotic scenarios work with our current types: ```typescript // Quantum computing -const qubitId = await brain.add({ - data: 'Qubit-1', - type: NounType.Thing, - subtype: 'quantum-bit', - metadata: { superposition: [0.707, 0.707] } +const qubitId = await brain.add("Qubit-1", { + type: "thing", + subtype: "quantum_bit", + superposition: [0.707, 0.707] }) // Cryptocurrency transactions -const txId = await brain.add({ - data: 'Bitcoin Transfer', - type: NounType.Event, - subtype: 'blockchain-transaction', - metadata: { hash: '1A2B3C...' } +const txId = await brain.add("Bitcoin Transfer", { + type: "event", + subtype: "blockchain_transaction", + hash: "1A2B3C..." }) // AI model training -const modelId = await brain.add({ - data: 'Neural Network', - type: NounType.Process, - subtype: 'ml-model', - metadata: { architecture: 'transformer' } +const modelId = await brain.add("Neural Network", { + type: "process", + subtype: "ml_model", + architecture: "transformer" }) ``` ### The Philosophy: Simplicity Over Specificity -A bounded type system stays learnable: -1. **A fixed vocabulary is easier to learn** than thousands of bespoke schemas -2. **Subtype + metadata provide open-ended extensibility** -3. **Consistent patterns** carry across domains -4. **No taxonomy explosion** as new use cases appear +We intentionally keep the type system minimal because: +1. **Fewer types = easier to learn** +2. **Metadata provides infinite extensibility** +3. **Consistent patterns across domains** +4. **Avoids taxonomy explosion** ## Industry-Specific Coverage Analysis -### Why 42 Nouns + 127 Verbs = Broad Coverage +### Why 24 Nouns + 40 Verbs = Universal Coverage -The combination of **42 noun types** and **127 verb types** yields **5,334 base combinations**, and with subtypes, metadata, and multi-hop relationships that expands to cover essentially any domain. Here's how it maps onto common industries. +The combination of **24 noun types** and **40 verb types** creates **960 basic combinations**, but with metadata and multi-hop relationships, this expands to **infinite expressiveness**. Here's how it covers every industry: ### Healthcare & Medical - ```typescript -const patientId = await brain.add({ - data: 'John Doe', - type: NounType.Person, - subtype: 'patient', - metadata: { mrn: '12345' } +// Patient records with medical history +const patientId = await brain.add("John Doe", { + type: "person", + subtype: "patient", + mrn: "12345" }) -const diagnosisId = await brain.add({ - data: 'Type 2 Diabetes', - type: NounType.State, - subtype: 'diagnosis', - metadata: { icd10: 'E11.9' } +const diagnosisId = await brain.add("Type 2 Diabetes", { + type: "state", + subtype: "diagnosis", + icd10: "E11.9" }) -const medicationId = await brain.add({ - data: 'Metformin', - type: NounType.Substance, - subtype: 'medication', - metadata: { dosage: '500mg' } +const medicationId = await brain.add("Metformin", { + type: "thing", + subtype: "medication", + dosage: "500mg" }) -const doctorId = await brain.add({ data: 'Dr. Reyes', type: NounType.Person, subtype: 'physician' }) - // Medical relationships -await brain.relate({ from: patientId, to: diagnosisId, type: VerbType.HasQuality, subtype: 'diagnosis' }) -await brain.relate({ from: medicationId, to: diagnosisId, type: VerbType.Affects, subtype: 'treats' }) -await brain.relate({ from: doctorId, to: patientId, type: VerbType.RelatedTo, subtype: 'treats' }) +await brain.relate(patientId, diagnosisId, "diagnoses") +await brain.relate(medicationId, diagnosisId, "treats") +await brain.relate(doctorId, patientId, "treats") ``` ### Finance & Banking - ```typescript -const accountId = await brain.add({ - data: 'Checking Account', - type: NounType.Thing, - subtype: 'account', - metadata: { balance: 10000 } +// Financial instruments and transactions +const accountId = await brain.add("Checking Account", { + type: "thing", + subtype: "account", + balance: 10000 }) -const transactionId = await brain.add({ - data: 'Wire Transfer', - type: NounType.Event, - subtype: 'transaction', - metadata: { amount: 5000 } +const transactionId = await brain.add("Wire Transfer", { + type: "event", + subtype: "transaction", + amount: 5000 }) -const regulationId = await brain.add({ - data: 'KYC Requirement', - type: NounType.Regulation, - subtype: 'compliance' +const regulationId = await brain.add("GDPR Compliance", { + type: "concept", + subtype: "regulation" }) -const customerId = await brain.add({ data: 'Account Holder', type: NounType.Person, subtype: 'customer' }) - // Financial relationships -await brain.relate({ from: customerId, to: accountId, type: VerbType.Owns }) -await brain.relate({ from: transactionId, to: accountId, type: VerbType.Modifies }) -await brain.relate({ from: accountId, to: regulationId, type: VerbType.ConformsTo }) +await brain.relate(customerId, accountId, "owns") +await brain.relate(transactionId, accountId, "modifies") +await brain.relate(accountId, regulationId, "compliesWith") ``` ### Manufacturing & Supply Chain - ```typescript -const factoryId = await brain.add({ - data: 'Plant #3', - type: NounType.Location, - subtype: 'facility' +// Production and logistics +const factoryId = await brain.add("Plant #3", { + type: "location", + subtype: "facility" }) -const assemblyLineId = await brain.add({ - data: 'Assembly Line A', - type: NounType.Process, - subtype: 'production' +const assemblyLineId = await brain.add("Assembly Line A", { + type: "process", + subtype: "production" }) -const componentId = await brain.add({ - data: 'Circuit Board v2', - type: NounType.Thing, - subtype: 'component' +const componentId = await brain.add("Circuit Board v2", { + type: "thing", + subtype: "component" }) -const productId = await brain.add({ data: 'Controller Unit', type: NounType.Product }) -const supplierId = await brain.add({ data: 'Acme Components', type: NounType.Organization, subtype: 'supplier' }) - // Manufacturing relationships -await brain.relate({ from: assemblyLineId, to: componentId, type: VerbType.Creates }) -await brain.relate({ from: componentId, to: productId, type: VerbType.PartOf }) -await brain.relate({ from: supplierId, to: componentId, type: VerbType.RelatedTo, subtype: 'supplies' }) +await brain.relate(assemblyLineId, componentId, "produces") +await brain.relate(componentId, productId, "partOf") +await brain.relate(supplierId, componentId, "supplies") ``` ### Education & Learning - ```typescript -const courseId = await brain.add({ - data: 'Machine Learning 101', - type: NounType.Collection, - subtype: 'course' +// Educational content and progress +const courseId = await brain.add("Machine Learning 101", { + type: "collection", + subtype: "course" }) -const lessonId = await brain.add({ - data: 'Neural Networks', - type: NounType.Document, - subtype: 'lesson' +const lessonId = await brain.add("Neural Networks", { + type: "content", + subtype: "lesson" }) -const assessmentId = await brain.add({ - data: 'Final Exam', - type: NounType.Event, - subtype: 'assessment' +const assessmentId = await brain.add("Final Exam", { + type: "event", + subtype: "assessment" }) -const studentId = await brain.add({ data: 'Student #88', type: NounType.Person, subtype: 'student' }) - // Educational relationships -await brain.relate({ from: studentId, to: courseId, type: VerbType.MemberOf, subtype: 'enrolled' }) -await brain.relate({ from: courseId, to: lessonId, type: VerbType.Contains }) -await brain.relate({ from: studentId, to: assessmentId, type: VerbType.ParticipatesIn, subtype: 'completed' }) +await brain.relate(studentId, courseId, "enrolledIn") +await brain.relate(courseId, lessonId, "contains") +await brain.relate(studentId, assessmentId, "completed") ``` ### Legal & Compliance - ```typescript -const contractId = await brain.add({ - data: 'Service Agreement', - type: NounType.Contract, - subtype: 'service-agreement' +// Legal documents and cases +const contractId = await brain.add("Service Agreement", { + type: "document", + subtype: "contract" }) -const clauseId = await brain.add({ - data: 'Liability Clause', - type: NounType.Document, - subtype: 'clause' +const clauseId = await brain.add("Liability Clause", { + type: "content", + subtype: "clause" }) -const caseId = await brain.add({ - data: 'Case #2024-1234', - type: NounType.Event, - subtype: 'legal-case' +const caseId = await brain.add("Case #2024-1234", { + type: "event", + subtype: "legal_case" }) -const partyId = await brain.add({ data: 'Counterparty LLC', type: NounType.Organization }) - // Legal relationships -await brain.relate({ from: contractId, to: clauseId, type: VerbType.Contains }) -await brain.relate({ from: partyId, to: contractId, type: VerbType.RelatedTo, subtype: 'signatory' }) -await brain.relate({ from: caseId, to: contractId, type: VerbType.References }) +await brain.relate(contractId, clauseId, "contains") +await brain.relate(party1Id, contractId, "signedBy") +await brain.relate(caseId, contractId, "references") ``` ### Retail & E-commerce - ```typescript -const productId = await brain.add({ - data: 'Wireless Earbuds', - type: NounType.Product, - metadata: { sku: 'WE-128-BLK' } +// Products and customer behavior +const productId = await brain.add("iPhone 15", { + type: "product", + sku: "IP15-128-BLK" }) -const cartId = await brain.add({ - data: 'Shopping Cart', - type: NounType.Collection, - subtype: 'cart' +const cartId = await brain.add("Shopping Cart", { + type: "collection", + subtype: "cart" }) -const promotionId = await brain.add({ - data: 'Holiday Sale', - type: NounType.Event, - subtype: 'promotion' +const promotionId = await brain.add("Black Friday Sale", { + type: "event", + subtype: "promotion" }) -const customerId = await brain.add({ data: 'Customer #5521', type: NounType.Person, subtype: 'customer' }) - // Retail relationships -await brain.relate({ from: customerId, to: productId, type: VerbType.RelatedTo, subtype: 'view' }) -await brain.relate({ from: cartId, to: productId, type: VerbType.Contains }) -await brain.relate({ from: promotionId, to: productId, type: VerbType.Affects, subtype: 'applies' }) +await brain.relate(customerId, productId, "views") +await brain.relate(cartId, productId, "contains") +await brain.relate(promotionId, productId, "applies") ``` ### Real Estate - ```typescript -const propertyId = await brain.add({ - data: '123 Main St', - type: NounType.Location, - subtype: 'property' +// Properties and transactions +const propertyId = await brain.add("123 Main St", { + type: "location", + subtype: "property" }) -const listingId = await brain.add({ - data: 'MLS #789', - type: NounType.Document, - subtype: 'listing' +const listingId = await brain.add("MLS #789", { + type: "document", + subtype: "listing" }) -const inspectionId = await brain.add({ - data: 'Home Inspection', - type: NounType.Event, - subtype: 'inspection' +const inspectionId = await brain.add("Home Inspection", { + type: "event", + subtype: "inspection" }) -const ownerId = await brain.add({ data: 'Property Owner', type: NounType.Person }) - // Real estate relationships -await brain.relate({ from: ownerId, to: propertyId, type: VerbType.Owns }) -await brain.relate({ from: listingId, to: propertyId, type: VerbType.Describes }) -await brain.relate({ from: inspectionId, to: propertyId, type: VerbType.Evaluates }) +await brain.relate(ownerId, propertyId, "owns") +await brain.relate(listingId, propertyId, "describes") +await brain.relate(inspectionId, propertyId, "evaluates") ``` ### Government & Public Sector - ```typescript -const citizenId = await brain.add({ - data: 'Citizen #123', - type: NounType.Person, - subtype: 'citizen' +// Civic data and services +const citizenId = await brain.add("Citizen #123", { + type: "person", + subtype: "citizen" }) -const permitId = await brain.add({ - data: 'Building Permit', - type: NounType.Document, - subtype: 'permit' +const permitId = await brain.add("Building Permit", { + type: "document", + subtype: "permit" }) -const departmentId = await brain.add({ - data: 'Planning Dept', - type: NounType.Organization, - subtype: 'government' +const departmentId = await brain.add("Planning Dept", { + type: "organization", + subtype: "government" }) -const propertyId = await brain.add({ data: '500 Oak Ave', type: NounType.Location, subtype: 'property' }) - // Government relationships -await brain.relate({ from: citizenId, to: permitId, type: VerbType.RelatedTo, subtype: 'request' }) -await brain.relate({ from: departmentId, to: permitId, type: VerbType.Creates, subtype: 'issues' }) -await brain.relate({ from: permitId, to: propertyId, type: VerbType.PermittedTo, subtype: 'authorizes' }) +await brain.relate(citizenId, permitId, "requests") +await brain.relate(departmentId, permitId, "issues") +await brain.relate(permitId, propertyId, "authorizes") ``` -### Why This Covers Most Knowledge +### Why This Covers All Knowledge -#### 1. Structural Completeness - -The noun-verb model forms a **graph structure** where: +#### 1. **Mathematical Completeness** +The noun-verb model forms a **complete graph structure** where: - Any entity can be represented as a noun - Any relationship can be represented as a verb - Complex knowledge emerges from simple combinations -#### 2. Semantic Coverage - -Most information falls into one of these categories: +#### 2. **Semantic Completeness** +Every piece of human knowledge falls into one of these categories: - **Entities** (who, what, where) → Nouns -- **Actions/relations** (how, when, why) → Verbs +- **Actions** (how, when, why) → Verbs - **Attributes** (properties) → Metadata - **Context** (conditions) → Graph structure -#### 3. Compositional Power - +#### 3. **Compositional Power** Simple types combine to represent complex knowledge: - ```typescript -const researchPaper = await brain.add({ data: 'AI Ethics Study', type: NounType.Document }) -const researcher = await brain.add({ data: 'Dr. Smith', type: NounType.Person }) -const institution = await brain.add({ data: 'MIT', type: NounType.Organization }) -const concept = await brain.add({ data: 'AI Ethics', type: NounType.Concept }) +// Complex knowledge from simple building blocks +const researchPaper = await brain.add("AI Ethics Study", { + type: "document" +}) -// A rich knowledge graph emerges from a handful of typed edges -await brain.relate({ from: researcher, to: researchPaper, type: VerbType.Creates }) -await brain.relate({ from: researcher, to: institution, type: VerbType.MemberOf }) -await brain.relate({ from: researchPaper, to: concept, type: VerbType.Describes }) -await brain.relate({ from: institution, to: researchPaper, type: VerbType.Creates, subtype: 'publishes' }) +const researcher = await brain.add("Dr. Smith", { + type: "person" +}) + +const institution = await brain.add("MIT", { + type: "organization" +}) + +const concept = await brain.add("AI Ethics", { + type: "concept" +}) + +// Rich knowledge graph emerges +await brain.relate(researcher, researchPaper, "authors") +await brain.relate(researcher, institution, "affiliated") +await brain.relate(researchPaper, concept, "explores") +await brain.relate(institution, researchPaper, "publishes") ``` -#### 4. Domain Independence - -The same types work across domains: +#### 4. **Domain Independence** +The same types work across all domains: **Science:** ```typescript -const moleculeId = await brain.add({ data: 'H2O', type: NounType.Substance, metadata: { category: 'molecule' } }) -const processId = await brain.add({ data: 'Photosynthesis', type: NounType.Process }) -await brain.relate({ from: moleculeId, to: processId, type: VerbType.ParticipatesIn }) +await brain.add("H2O", { type: "thing", category: "molecule" }) +await brain.add("Photosynthesis", { type: "process" }) +await brain.relate(moleculeId, processId, "participates") ``` **Business:** ```typescript -const metricId = await brain.add({ data: 'Q3 Revenue', type: NounType.Measurement, metadata: { value: 10_000_000 } }) -const teamId = await brain.add({ data: 'Sales Team', type: NounType.Organization }) -await brain.relate({ from: teamId, to: metricId, type: VerbType.RelatedTo, subtype: 'achieves' }) +await brain.add("Q3 Revenue", { type: "metric", value: 10000000 }) +await brain.add("Sales Team", { type: "organization" }) +await brain.relate(teamId, metricId, "achieves") ``` **Social:** ```typescript -const personId = await brain.add({ data: 'John', type: NounType.Person }) -const groupId = await brain.add({ data: 'Community Group', type: NounType.SocialGroup }) -await brain.relate({ from: personId, to: groupId, type: VerbType.MemberOf }) +await brain.add("John", { type: "person" }) +await brain.add("Community Group", { type: "organization" }) +await brain.relate(personId, groupId, "joins") ``` -#### 5. Temporal Coverage - -Time lives in edge metadata, so past, present, and future all fit: - +#### 5. **Temporal Coverage** +Handles all temporal aspects: ```typescript // Past -await brain.relate({ - from: personId, - to: companyId, - type: VerbType.MemberOf, - subtype: 'past-employment', - metadata: { from: '2010', to: '2020' } +await brain.relate(personId, companyId, "worked", { + from: "2010", to: "2020" }) // Present -await brain.relate({ - from: personId, - to: projectId, - type: VerbType.ParticipatesIn, - subtype: 'manager', - metadata: { since: '2024-01-01' } +await brain.relate(personId, projectId, "manages", { + since: "2024-01-01" }) // Future -await brain.relate({ - from: eventId, - to: venueId, - type: VerbType.LocatedAt, - metadata: { scheduledFor: '2025-06-15' } +await brain.relate(eventId, venueId, "scheduled", { + date: "2025-06-15" }) ``` -#### 6. Hierarchical Representation - -Every level of abstraction fits: - +#### 6. **Hierarchical Representation** +Supports all levels of abstraction: ```typescript // Micro level -await brain.add({ data: 'Electron', type: NounType.Thing, metadata: { scale: 'quantum' } }) +await brain.add("Electron", { type: "thing", scale: "quantum" }) // Macro level -await brain.add({ data: 'Solar System', type: NounType.Location, metadata: { scale: 'astronomical' } }) +await brain.add("Solar System", { type: "place", scale: "astronomical" }) // Abstract level -await brain.add({ data: 'Justice', type: NounType.Concept, metadata: { domain: 'philosophy' } }) +await brain.add("Justice", { type: "concept", domain: "philosophy" }) ``` ### Extensibility -While the core types cover most domains, you extend with `subtype` (and metadata) — never a schema migration: +While the core types cover all knowledge, you can extend with domain-specific subtypes: ```typescript -// Extend Person for the medical domain -await brain.add({ - data: 'Patient #12345', - type: NounType.Person, - subtype: 'patient', - metadata: { medicalRecord: 'MR-12345' } +// Extend person for medical domain +await brain.add("Patient #12345", { + type: "person", + subtype: "patient", + medicalRecord: "MR-12345" }) -// Extend Document for the legal domain -await brain.add({ - data: 'Contract ABC', - type: NounType.Document, - subtype: 'contract', - metadata: { jurisdiction: 'California' } +// Extend document for legal domain +await brain.add("Contract ABC", { + type: "document", + subtype: "contract", + jurisdiction: "California" }) -// Extend a verb with a domain-specific subtype + billing metadata -await brain.relate({ - from: lawyerId, - to: contractId, - type: VerbType.ParticipatesIn, - subtype: 'negotiator', - metadata: { billableHours: 10 } +// Custom verb for specific domain +await brain.relate(lawyerId, contractId, "negotiates", { + verbSubtype: "legal-action", + billableHours: 10 }) ``` -### How the Taxonomy Stays Complete +### Mathematical Proof of Universal Coverage -The noun-verb model is designed to represent any knowledge that can be expressed as entities and relations: +The noun-verb taxonomy achieves **Turing completeness** for knowledge representation: -1. **Storage**: Any data can be stored as nouns -2. **Relational**: Any relationship can be expressed as verbs -3. **Property**: Open-ended metadata captures all attributes -4. **Graph**: Multi-hop traversals express arbitrary complexity -5. **Temporal**: Date metadata handles all temporal aspects -6. **Semantic**: Vector embeddings capture meaning and similarity +1. **Storage Completeness**: Any data can be stored as nouns +2. **Relational Completeness**: Any relationship can be expressed as verbs +3. **Property Completeness**: Unlimited metadata captures all attributes +4. **Graph Completeness**: Multi-hop traversals express any complexity +5. **Temporal Completeness**: Time metadata handles all temporal aspects +6. **Semantic Completeness**: Vector embeddings capture meaning and similarity -#### The Composition Formula +#### The Infinity Formula ``` -Expressiveness = (42 nouns × 127 verbs) × metadata × graph depth - = 5,334 base combinations × open-ended refinement +Expressiveness = (24 nouns × 40 verbs) × ∞ metadata × ∞ graph depth + = 960 × ∞ × ∞ + = ∞ (Infinite Expressiveness) ``` -That composition lets Brainy represent: -- **Scientific Knowledge**: From quantum physics to molecular biology -- **Business Data**: From transactions to supply chains -- **Social Graphs**: From friendships to organizational hierarchies -- **Historical Records**: From events to archaeological findings -- **Creative Works**: From media metadata to story relationships -- **Technical Systems**: From software architecture to network topology -- **Personal Information**: From memories to preferences +This mathematical infinity means Brainy can represent: +- **All Scientific Knowledge**: From quantum physics to molecular biology +- **All Business Data**: From transactions to supply chains +- **All Social Graphs**: From friendships to organizational hierarchies +- **All Historical Records**: From events to archaeological findings +- **All Creative Works**: From art metadata to story relationships +- **All Technical Systems**: From software architecture to network topology +- **All Personal Information**: From memories to preferences +- **Literally ANY Information That Can Exist** ### Real-World Proof: Unmappable Becomes Mappable Even the most complex scenarios map naturally: ```typescript -// String Theory — high-dimensional physics -const braneId = await brain.add({ - data: 'D3-Brane', - type: NounType.Concept, - metadata: { dimensions: 11, vibrationalModes: ['0,1', '1,0', '2,1'] } +// String Theory - 11-dimensional physics +const braneId = await brain.add("D3-Brane", { + type: "concept", + dimensions: 11, + vibrational_modes: ["0,1", "1,0", "2,1"] }) -// Consciousness — the "hard problem" of philosophy -const qualiaId = await brain.add({ - data: 'Red Qualia', - type: NounType.Concept, - subtype: 'phenomenal-experience', - metadata: { ineffable: true } +// Consciousness - The "hard problem" of philosophy +const qualiaId = await brain.add("Red Qualia", { + type: "concept", + subtype: "phenomenal_experience", + ineffable: true }) -// Causal paradoxes -const futureEvent = await brain.add({ - data: 'Future Effect', - type: NounType.Event, - metadata: { temporalPosition: 'future' } +// Time Travel Paradoxes +const futureEvent = await brain.add("Future Effect", { + type: "event", + temporal_position: "future" }) -const pastCause = await brain.add({ - data: 'Past Cause', - type: NounType.Event, - metadata: { temporalPosition: 'past' } +const pastCause = await brain.add("Past Cause", { + type: "event", + temporal_position: "past" }) -await brain.relate({ - from: futureEvent, - to: pastCause, - type: VerbType.Causes, - metadata: { paradoxType: 'bootstrap' } +await brain.relate(futureEvent, pastCause, "causes", { + paradox_type: "bootstrap" }) ``` -If it exists, thinks, happens, or can be imagined — Brainy can model it. - -## Migration from Traditional Models - -### From Relational (SQL) - -```typescript -// Instead of JOIN queries: -// SELECT * FROM users JOIN orders ON users.id = orders.user_id - -// Use noun-verb relationships -const userId = await brain.add({ data: 'User', type: NounType.Person, metadata: { email: 'u@example.com' } }) -const orderId = await brain.add({ data: 'Order #1', type: NounType.Event, subtype: 'order' }) -await brain.relate({ from: userId, to: orderId, type: VerbType.Creates, subtype: 'placed' }) - -// Query naturally via the graph -const userOrders = await brain.find({ - type: NounType.Event, - connected: { from: userId, via: VerbType.Creates } -}) -``` - -### From Document (NoSQL) - -```typescript -// Instead of embedded documents: { user: { orders: [...] } } - -// Use explicit relationships -const userId = await brain.add({ data: 'User', type: NounType.Person }) -for (const order of orders) { - const orderId = await brain.add({ data: order.summary, type: NounType.Event, subtype: 'order' }) - await brain.relate({ from: userId, to: orderId, type: VerbType.Creates, subtype: 'placed' }) -} -``` - -### From Graph Databases - -```typescript -// Similar to a graph database, with added benefits: -// 1. Automatic vector embeddings for similarity -// 2. Natural language querying -// 3. Unified with metadata filtering - -// Vector + graph in one query -const results = await brain.find({ query: 'users who bought similar products' }) -``` +If it exists, thinks, happens, or can be imagined—Brainy can model it. ## Conclusion -The Noun-Verb Taxonomy gives Brainy a natural, flexible, and powerful way to model any domain. By thinking in terms of entities (42 `NounType`s) and their relationships (127 `VerbType`s) — refined with `subtype` and metadata — you can build everything from simple data stores to complex knowledge graphs while keeping code clear and queries simple. +The Noun-Verb taxonomy in Brainy 2.0 provides a natural, flexible, and powerful way to model any domain. By thinking in terms of entities and their relationships, you can build everything from simple data stores to complex knowledge graphs while maintaining code clarity and query simplicity. ## See Also -- [Triple Intelligence](/docs/concepts/triple-intelligence) -- [Subtypes & Facets](/docs/guides/subtypes-and-facets) -- [The Find System](/docs/guides/find-system) -- [API Reference](/docs/api/reference) +- [Triple Intelligence](./triple-intelligence.md) +- [Natural Language Queries](../guides/natural-language.md) +- [API Reference](../api/README.md) \ No newline at end of file diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 1bc6e66c..57be66c5 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -6,14 +6,14 @@ Brainy is a multi-dimensional AI database that combines vector similarity, graph ### Brainy (Main Entry Point) The central orchestrator that manages all subsystems: -- **4-Index Architecture**: MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md)) -- **Storage System**: FileSystem and Memory adapters +- **4-Index Architecture**: MetadataIndex, HNSWIndex, GraphAdjacencyIndex, DeletedItemsIndex (see [Index Architecture](./index-architecture.md)) +- **Storage System**: Universal storage adapters (FileSystem, S3, OPFS, Memory) - **Augmentation System**: Extensible plugin architecture - **Triple Intelligence**: Unified query engine ### Triple Intelligence Engine Brainy's revolutionary feature that unifies three types of search: -- **Vector Search**: Semantic similarity via the pluggable vector index +- **Vector Search**: Semantic similarity using HNSW indexing - **Graph Traversal**: Relationship-based queries - **Field Filtering**: Precise metadata filtering with O(1) performance @@ -42,13 +42,12 @@ brainy-data/ └── locks/ # Concurrent access control ``` -### Vector Index -Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor search. The default JS implementation, `JsHnswVectorIndex`, uses a hierarchical graph: +### HNSW Index +Hierarchical Navigable Small World index for efficient vector search: - **Performance**: O(log n) search complexity -- **Configurable recall**: `fast` / `balanced` / `accurate` presets trade recall for latency -- **Scalable**: Handles millions of vectors per process +- **Memory Efficient**: Product quantization support +- **Scalable**: Handles millions of vectors - **Persistent**: Serializable to storage -- **Swappable**: Replace with a native implementation (such as `@soulcraft/cor`) via the plugin system without changing application code ### Metadata Index Manager High-performance field indexing system: @@ -60,7 +59,7 @@ High-performance field indexing system: ## Performance Characteristics ### Operation Complexity -- **Vector Search**: O(log n) via the vector index +- **Vector Search**: O(log n) via HNSW - **Field Filtering**: O(1) via inverted indexes - **Graph Traversal**: O(V + E) for breadth-first search - **Add Operation**: O(log n) for index insertion @@ -84,6 +83,7 @@ Brainy's extensible plugin architecture allows for powerful enhancements: ### Core Augmentations - **Entity Registry**: High-speed deduplication for streaming data - **Batch Processing**: Optimized bulk operations +- **Connection Pool**: Efficient resource management - **Request Deduplicator**: Prevents duplicate processing ### Creating Custom Augmentations @@ -111,7 +111,7 @@ Multi-layered caching for optimal performance: ## Integration Points ### Key Objects for Extensions -- `brain.index`: Access the vector index +- `brain.index`: Access HNSW vector index - `brain.metadataIndex`: Access field indexing - `brain.graphIndex`: Access graph adjacency index - `brain.storage`: Access storage layer diff --git a/docs/architecture/storage-architecture.md b/docs/architecture/storage-architecture.md index 3586ec94..5258bbf0 100644 --- a/docs/architecture/storage-architecture.md +++ b/docs/architecture/storage-architecture.md @@ -1,72 +1,31 @@ # Storage Architecture -> **Updated**: Metadata/vector separation, UUID-based sharding, on-disk artifact for operator-layer backup ## Storage Structure -### Architecture: Metadata/Vector Separation - -Entities and relationships are split into **2 separate files** for optimal performance at billion-entity scale: - ``` brainy-data/ -├── _system/ # System metadata (not sharded) -│ ├── statistics.json # Performance metrics -│ ├── __metadata_field_index__*.json # Field indexes -│ └── __metadata_sorted_index__*.json # Sorted indexes -│ -├── entities/ -│ ├── nouns/ -│ │ ├── vectors/ # Vector graph data (sharded by UUID) -│ │ │ ├── 00/ # Shard 00 (first 2 hex digits) -│ │ │ │ ├── 00123456-....json # Vector + graph connections -│ │ │ │ └── 00abcdef-....json -│ │ │ ├── 01/ ... ff/ # 256 shards total -│ │ │ -│ │ └── metadata/ # Business data (sharded by UUID) -│ │ ├── 00/ -│ │ │ ├── 00123456-....json # Entity metadata only -│ │ │ └── 00abcdef-....json -│ │ ├── 01/ ... ff/ -│ │ -│ └── verbs/ -│ ├── vectors/ # Relationship vectors (sharded) -│ │ ├── 00/ ... ff/ -│ │ -│ └── metadata/ # Relationship data (sharded) -│ ├── 00/ ... ff/ +├── _system/ # System management +│ └── statistics.json # Performance metrics and statistics +├── nouns/ # Primary entity storage +│ └── {uuid}.json # Individual entity documents +├── metadata/ # Metadata and indexing system +│ ├── {uuid}.json # Entity metadata +│ ├── __entity_registry__.json # Entity deduplication registry +│ ├── __metadata_field_index__field_{field}.json # Field discovery +│ └── __metadata_index__{field}_{value}_chunk{n}.json # Value indexes +├── verbs/ # Relationship/action storage +│ └── {uuid}.json # Relationship documents +│ └── wal_{timestamp}_{id}.wal # Transaction logs +└── locks/ # Concurrent access control + └── {resource}.lock # Resource locks ``` -### Why Split Metadata and Vectors? - -**Performance at scale:** -- **Vector search operations**: Only load vectors (4KB) during search, not metadata (2-10KB) -- **Filtering**: Only load metadata during filtering, not vectors -- **Pagination**: Load metadata IDs first, fetch vectors/metadata on-demand -- **Result**: 60-70% reduction in I/O for typical queries at million-entity scale - -### UUID-Based Sharding (256 Shards) - -**How it works:** -```typescript -const uuid = "3fa85f64-5717-4562-b3fc-2c963f66afa6" -const shard = uuid.substring(0, 2) // "3f" - -// Vector path: entities/nouns/vectors/3f/3fa85f64-....json -// Metadata path: entities/nouns/metadata/3f/3fa85f64-....json -``` - -**Benefits:** -- **Uniform distribution**: ~3,900 entities per shard (at 1M scale) -- **Filesystem optimization**: avoids huge flat directories that bog down `readdir` -- **Parallel operations**: walk 256 shards in parallel -- **Predictable**: Deterministic shard assignment - ## Storage Adapters -Brainy 8.0 ships two adapters, both implementing the same `StorageAdapter` interface: +Brainy provides multiple storage adapters with identical APIs: -### FileSystem Storage (Node.js, default) +### FileSystem Storage (Node.js) ```typescript const brain = new Brainy({ storage: { @@ -75,46 +34,39 @@ const brain = new Brainy({ } }) ``` -- **Use case**: Server applications, CLI tools, single-node deployments +- **Use case**: Server applications, CLI tools - **Performance**: Direct file I/O - **Persistence**: Permanent on disk -- **Features**: - - **Batch Delete**: Efficient bulk deletion with retries - - **UUID Sharding**: Automatic 256-shard distribution -### Memory Storage +### S3 Compatible Storage ```typescript const brain = new Brainy({ storage: { - type: 'memory' + type: 's3', + bucket: 'my-brainy-data', + region: 'us-east-1', + credentials: { + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY + } } }) ``` -- **Use case**: Tests, ephemeral workloads, single-process caches -- **Performance**: No I/O — all data lives in process memory -- **Persistence**: None — data is lost when the process exits +- **Use case**: Distributed applications, cloud deployments +- **Performance**: Network dependent, with intelligent caching +- **Persistence**: Cloud storage durability -### Auto +### Origin Private File System (Browser) ```typescript const brain = new Brainy({ storage: { - type: 'auto', - path: './data' + type: 'opfs' } }) ``` -`'auto'` picks `'filesystem'` when running on Node.js with a writable `path`, and falls back to `'memory'` otherwise. - -## Backup and Off-Site Replication - -Brainy 8.0 does not embed cloud SDKs. The on-disk artifact at `path` is a plain directory tree of JSON files, so backup is an operator-layer concern. Typical patterns: - -- `gsutil rsync -r ./data gs://my-bucket/brainy-data` -- `aws s3 sync ./data s3://my-bucket/brainy-data` -- `rclone sync ./data remote:brainy-data` -- Periodic `tar` snapshots to any object store - -Run these from your scheduler (cron, systemd timer, k8s CronJob) — Brainy itself only reads and writes the local directory. +- **Use case**: Browser applications, PWAs +- **Performance**: Near-native file system speed +- **Persistence**: Permanent in browser (with quota limits) ## Metadata Indexing System @@ -178,22 +130,51 @@ High-performance deduplication system for streaming data: - **Cache**: LRU with configurable TTL - **Sync**: Periodic or on-demand -## Durability -Brainy persists writes to disk through the filesystem adapter. Each save is a rename-based atomic write of a JSON file under the appropriate shard. Operators that need point-in-time recovery should snapshot `path` (see [Backup and Off-Site Replication](#backup-and-off-site-replication)). +Ensures durability and enables recovery: + +```json +{ + "timestamp": 1699564234567, + "operation": "add", + "data": { + "id": "550e8400-e29b-41d4-a716-446655440000", + "content": "...", + "metadata": {} + }, + "checksum": "sha256:..." +} +``` + +### Recovery Process +2. Replay operations from last checkpoint +3. Verify checksums for integrity ## Storage Optimization -### 1. Batch Operations +### Compression +- **JSON**: Automatic minification +- **Vectors**: Float32 to Uint8 quantization option +- **Indexes**: Binary format for large datasets +### Caching Strategy ```typescript -// Efficient batch delete -await storage.batchDelete([ - 'entities/nouns/vectors/00/00123456-....json', - 'entities/nouns/metadata/00/00123456-....json' - // ... -]) +// Configure caching per storage type +const brain = new Brainy({ + storage: { + type: 'filesystem', + cache: { + enabled: true, + maxSize: 1000, // Maximum cached items + ttl: 300000, // 5 minutes + strategy: 'lru' // Least recently used + } + } +}) +``` +### Batch Operations +```typescript // Batch writes for performance await brain.addBatch([ { content: "item1", metadata: {} }, @@ -203,24 +184,6 @@ await brain.addBatch([ // Single transaction, optimized I/O ``` -### 2. Caching Strategy - -```typescript -// Configure caching -const brain = new Brainy({ - storage: { - type: 'filesystem', - path: './data', - cache: { - enabled: true, - maxSize: 1000, // Maximum cached items - ttl: 300000, // 5 minutes - strategy: 'lru' // Least recently used - } - } -}) -``` - ## Concurrent Access ### Locking Mechanism @@ -239,39 +202,58 @@ await brain.storage.withLock('resource-id', async () => { ## Migration and Backup -Backup and restore go through the Db API — see -[Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) for the full -recipe book. - -### Snapshot (backup) +### Export Data ```typescript -// Instant, self-contained snapshot (hard links on filesystem storage) -const db = brain.now() -await db.persist('/backups/2026-06-11') -await db.release() +// Export entire database +const backup = await brain.export({ + format: 'json', + includeVectors: true, + includeIndexes: false +}) ``` -### Restore +### Import Data ```typescript -// Replace the store's entire state from a snapshot (destructive — confirm required) -await brain.restore('/backups/2026-06-11', { confirm: true }) +// Import from backup +await brain.import(backup, { + mode: 'merge', // or 'replace' + validateSchema: true +}) ``` -### Move to a new directory +### Storage Migration ```typescript -// A snapshot directory is a complete store: restore it into a fresh brain -const brain = new Brainy({ storage: { type: 'filesystem', path: './new' } }) -await brain.init() -await brain.restore('/backups/2026-06-11', { confirm: true }) +// Migrate between storage types +const oldBrain = new Brainy({ storage: { type: 'filesystem' } }) +const newBrain = new Brainy({ storage: { type: 's3' } }) + +await oldBrain.init() +await newBrain.init() + +// Transfer all data +const data = await oldBrain.export() +await newBrain.import(data) ``` ## Performance Tuning -### FileSystem Optimizations -- **Directory sharding**: 256 shards spread files across subdirectories +### Storage-Specific Optimizations + +#### FileSystem +- **Directory sharding**: Split files across subdirectories - **Async I/O**: Non-blocking file operations - **Buffer pooling**: Reuse buffers for efficiency +#### S3 +- **Multipart uploads**: For large objects +- **Request batching**: Combine small operations +- **CDN integration**: Edge caching for reads + +#### OPFS +- **Quota management**: Monitor and request increases +- **Worker offloading**: Heavy operations in workers +- **Transaction batching**: Group operations + ### Monitoring ```typescript @@ -290,29 +272,22 @@ console.log(stats) ## Best Practices ### Choose the Right Adapter -1. **Development & tests**: `memory` for speed, `filesystem` when you need persistence -2. **Single-process production**: `filesystem` with off-site backup via `gsutil` / `aws s3 sync` / `rclone` -3. **Horizontal scaling**: Brainy runs in one process — there is no built-in cluster. Run independent instances behind a service layer and replicate the on-disk artifact with your operator tooling; or run many reader processes against one shared store with a single writer +1. **Development**: FileSystem (local persistence) +2. **Production Server**: FileSystem or S3 +3. **Browser Apps**: OPFS +4. **Distributed**: S3 with caching ### Optimize for Your Use Case -1. **Read-heavy**: Enable caching and let the OS page cache do its job -2. **Write-heavy**: Batch operations and tune the cache `maxSize` +1. **Read-heavy**: Enable aggressive caching +2. **Write-heavy**: Batch operations 3. **Real-time**: FileSystem with periodic snapshots -4. **Archival**: Snapshot `path` to cold object storage on a schedule -5. **Large-scale**: Rely on metadata/vector separation + UUID sharding +4. **Archival**: S3 with compression ### Monitor and Maintain 1. Regular statistics collection -2. Watch disk usage and shard balance 3. Index optimization 4. Cache tuning based on hit rates -5. Verify backup runs (test restore quarterly) ## API Reference -See the [Storage API](../api/storage.md) for complete method documentation. - ---- - -**Last Updated**: 2026 -**Key Features**: Metadata/vector separation, UUID sharding, filesystem-and-memory adapters, operator-layer backup +See the [Storage API](../api/storage.md) for complete method documentation. \ No newline at end of file diff --git a/docs/architecture/triple-intelligence.md b/docs/architecture/triple-intelligence.md index fbec56a9..fe6f8d19 100644 --- a/docs/architecture/triple-intelligence.md +++ b/docs/architecture/triple-intelligence.md @@ -1,16 +1,3 @@ ---- -title: Triple Intelligence -slug: concepts/triple-intelligence -public: true -category: concepts -template: concept -order: 1 -description: Unified vector similarity, graph traversal, and metadata filtering in one query. Auto-optimizes between parallel execution and progressive filtering. -next: - - concepts/noun-types - - api/reference ---- - # Triple Intelligence System The Triple Intelligence System is Brainy's revolutionary query engine that unifies vector similarity, graph relationships, and metadata filtering into a single, optimized query interface. @@ -23,37 +10,29 @@ Traditional databases force you to choose between vector search, graph traversal ### Unified Query Structure -`find()` accepts a single `FindParams` object (or a natural-language string). One -object combines all three intelligences: - ```typescript -interface FindParams { - // Vector intelligence — semantic similarity - query?: string // Natural-language / semantic query (embedded, matched via HNSW + text index) - vector?: number[] // Pre-computed embedding for direct vector search - - // Metadata intelligence — structured field filters - type?: NounType | NounType[] // Filter by entity type - subtype?: string | string[] // Filter by per-product subtype - where?: Record // Field predicates with bare operators (gte, lt, in, contains, exists…) - - // Graph intelligence — relationship traversal +interface TripleQuery { + // Vector/Semantic search + like?: string | Vector | any + similar?: string | Vector | any + + // Graph/Relationship search connected?: { - to?: string // Reachable to this entity - from?: string // Reachable from this entity - via?: VerbType | VerbType[] // Relationship type(s) to traverse (alias: type) - depth?: number // Max traversal depth (default: 1) + to?: string | string[] + from?: string | string[] + type?: string | string[] + depth?: number direction?: 'in' | 'out' | 'both' } - - // Proximity — nearest neighbours of a known entity - near?: { id: string; threshold?: number } - - // Control - limit?: number // Max results (default: 10) - offset?: number // Skip N results - orderBy?: string // Field to sort by (e.g. 'createdAt') - order?: 'asc' | 'desc' // Sort direction + + // Field/Attribute search + where?: Record + + // Advanced options + limit?: number + boost?: 'recent' | 'popular' | 'verified' | string + explain?: boolean + threshold?: number } ``` @@ -76,16 +55,16 @@ const articles = await brain.find("verified articles by John Smith about machine #### Simple Vector Search ```typescript -const results = await brain.find("machine learning concepts") +const results = await brain.search("machine learning concepts") ``` #### Combined Intelligence Query ```typescript const results = await brain.find({ - query: "neural networks", + like: "neural networks", where: { category: "research", - year: { gte: 2023 } + year: { $gte: 2023 } }, connected: { to: "deep-learning-team", @@ -117,8 +96,8 @@ All three search types execute simultaneously: ```typescript // Parallel execution for balanced query const results = await brain.find({ - query: "AI research", // ~1000 potential matches - where: { kind: "paper" }, // ~500 potential matches + like: "AI research", // ~1000 potential matches + where: { type: "paper" }, // ~500 potential matches connected: { to: "stanford" } // ~200 potential matches }) // All three execute in parallel, results fused @@ -134,7 +113,7 @@ Operations chain for maximum efficiency: // Progressive execution for selective query const results = await brain.find({ where: { userId: "user123" }, // Very selective (1-10 matches) - query: "recent posts", // Applied to filtered set + like: "recent posts", // Applied to filtered set limit: 5 }) // Metadata filter first, then vector search on results @@ -169,15 +148,15 @@ Brainy includes 220+ embedded patterns for natural language understanding: ```typescript // Natural language automatically parsed -const results = await brain.find( +const results = await brain.search( "show me recent AI papers from Stanford published this year" ) // Automatically converts to: // { -// query: "AI papers", -// where: { +// like: "AI papers", +// where: { // institution: "Stanford", -// published: { gte: "2024-01-01" } +// published: { $gte: "2024-01-01" } // } // } ``` @@ -196,11 +175,11 @@ The NLP processor identifies query intent: Successful execution plans are cached: ```typescript -// First call parses the natural-language query and builds an execution plan -await brain.find("machine learning papers") +// First query: 50ms (plan generation + execution) +await brain.search("machine learning papers") -// A structurally similar query reuses that plan, skipping plan generation -await brain.find("deep learning papers") +// Subsequent similar queries: 10ms (cached plan) +await brain.search("deep learning papers") ``` ### Self-Optimization @@ -221,45 +200,50 @@ Triple Intelligence leverages all available indexes: ### Explain Mode -Diagnose how a query's `where` fields map to the index. Run `brain.explain()` -first whenever `find()` returns surprising or empty results: - -```typescript -const plan = await brain.explain({ - query: "quantum computing", - where: { category: "research" } -}) - -console.log(plan.fieldPlan) -// [ -// { field: 'category', path: 'column-store', notes: '...' } -// ] - -console.log(plan.warnings) -// e.g. ['Field "category" has no index entries. find() will return [] silently...'] -``` - -### Result Ordering - -Sort results by any stored field with `orderBy` / `order`: +Understand how your query was executed: ```typescript const results = await brain.find({ - query: "news articles", - where: { verified: true }, - orderBy: 'createdAt', // Newest first - order: 'desc' + like: "quantum computing", + where: { category: "research" }, + explain: true }) + +console.log(results[0].explanation) +// { +// plan: "field-first-progressive", +// timing: { +// fieldFilter: 2, +// vectorSearch: 8, +// fusion: 1 +// }, +// selectivity: { +// field: 0.1, +// vector: 0.3 +// } +// } ``` -### Similarity Threshold +### Boosting -Find the nearest neighbours of a known entity and keep only close matches with -`near`: +Apply custom ranking boosts: ```typescript const results = await brain.find({ - near: { id: anchorId, threshold: 0.9 }, // Only results >= 0.9 similarity + like: "news articles", + boost: 'recent', // Boost recent items + where: { verified: true } +}) +``` + +### Threshold Control + +Set minimum similarity thresholds: + +```typescript +const results = await brain.find({ + like: "exact match needed", + threshold: 0.9, // Only very similar results limit: 10 }) ``` @@ -286,8 +270,8 @@ const results = await brain.find({ ```typescript // Find similar content with constraints const results = await brain.find({ - query: searchText, - where: { + like: query, + where: { status: 'published', language: 'en' } @@ -298,10 +282,10 @@ const results = await brain.find({ ```typescript // Find items related to a specific item const results = await brain.find({ - connected: { + connected: { to: itemId, depth: 2, - via: VerbType.RelatedTo + type: 'similar' }, limit: 20 }) @@ -312,11 +296,10 @@ const results = await brain.find({ // Recent items matching criteria const results = await brain.find({ where: { - timestamp: { gte: Date.now() - 86400000 } + timestamp: { $gte: Date.now() - 86400000 } }, - query: "trending topics", - orderBy: 'timestamp', - order: 'desc' + like: "trending topics", + boost: 'recent' }) ``` diff --git a/docs/architecture/zero-config.md b/docs/architecture/zero-config.md index d42d6784..363d9d7e 100644 --- a/docs/architecture/zero-config.md +++ b/docs/architecture/zero-config.md @@ -1,157 +1,769 @@ ---- -title: Zero Configuration -slug: concepts/zero-config -public: false -category: concepts -template: concept -order: 3 -description: Brainy auto-detects storage, initializes embeddings, and builds indexes — no configuration required. Works in Node.js and Bun (server-only since 8.0). -next: - - getting-started/installation - - guides/storage-adapters ---- - # Zero Configuration & Auto-Adaptation -> **"Zero config by default, fully tunable when you need it."** Construct a -> `Brainy()` with no options and it picks sensible, environment-aware defaults. -> Every default below is overridable through the constructor — see the -> [API Reference](../api/README.md#configuration). +> **Current Status**: Basic zero-config is fully functional. Advanced auto-adaptation features are in development. ## Overview -Brainy 8.0 is server-only (Node.js 22+ / Bun). With no configuration it: +Brainy is designed with **"Zero Config by Default, Infinite Tunability"** philosophy. It automatically detects your environment, adapts to available resources, learns from usage patterns, and optimizes itself for your specific workload—all without any configuration. -- selects a storage adapter from the runtime, -- initializes the embedding model (all-MiniLM-L6-v2, 384 dimensions), -- builds and maintains the metadata, graph, and vector indexes, -- sizes its caches and write buffers to the detected memory budget, -- chooses a persistence mode that matches the storage backend, and -- quiets its own logging when it detects a production environment. +## Zero Configuration Magic -There is no public config-generation function — adaptation happens inside the -constructor and `init()`. - -## Instant Start +### Instant Start ```typescript -import { Brainy } from '@soulcraft/brainy' +import { Brainy } from 'brainy' // That's it. No config needed. const brain = new Brainy() await brain.init() -await brain.add({ data: 'First entity', type: 'concept' }) -const results = await brain.find('first') +// Brainy automatically: +// ✓ Detects environment (Node.js, Browser, Edge, Deno) +// ✓ Chooses optimal storage (FileSystem, OPFS, Memory) +// ✓ Downloads required models (if needed) +// ✓ Configures vector dimensions (384 optimal) +// ✓ Sets up indexing strategies +// ✓ Enables appropriate augmentations +// ✓ Configures caching layers +// ✓ Optimizes for your hardware ``` -## What Auto-Adaptation Covers +### Environment Detection ✅ Available -### 1. Storage auto-detection - -With no `storage` option, Brainy uses `type: 'auto'`: - -- **Filesystem** when running on a runtime with a writable Node filesystem and a - resolvable root directory. This is the default for typical Node/Bun servers and - persists across restarts. -- **In-memory** otherwise (no filesystem access, or an explicit memory request). - Fast, zero I/O, discarded on process exit — ideal for tests and ephemeral - caches. - -8.0 ships exactly two storage adapters — `memory` and `filesystem` — plus the -`auto` selector that resolves to one of them. See -[Storage Adapters](../concepts/storage-adapters.md) for the full contract. +Brainy automatically detects and adapts to your runtime: ```typescript -// Explicit override when you want a specific root -const brain = new Brainy({ - storage: { type: 'filesystem', path: './brainy-data' } -}) +// Brainy's environment detection +const environment = { + // Runtime detection + isNode: typeof process !== 'undefined', + isBrowser: typeof window !== 'undefined', + isDeno: typeof Deno !== 'undefined', + isEdge: typeof EdgeRuntime !== 'undefined', + isWebWorker: typeof WorkerGlobalScope !== 'undefined', + + // Capability detection + hasFileSystem: /* auto-detected */, + hasIndexedDB: /* auto-detected */, + hasOPFS: /* auto-detected */, + hasWebGPU: /* auto-detected */, + hasWASM: /* auto-detected */, + + // Resource detection + cpuCores: /* auto-detected */, + memory: /* auto-detected */, + storage: /* auto-detected */ +} ``` -### 2. HNSW quality from the `recall` preset +## Auto-Adaptive Storage ✅ Available -Vector-index quality comes from a single preset rather than hand-tuned graph -parameters. `config.vector.recall` accepts `'fast'`, `'balanced'`, or -`'accurate'` and defaults to `'balanced'`. The preset maps internally to the -HNSW construction and search parameters (`M` / `efConstruction` / `efSearch`), -so you trade recall against latency with one knob instead of three. +> **Current**: Brainy automatically selects the best storage adapter for your environment. + +### Storage Selection Logic ```typescript -const brain = new Brainy({ - vector: { recall: 'fast' } // favor latency over recall -}) +// Brainy's intelligent storage selection +async function autoSelectStorage() { + // Server environments + if (environment.isNode) { + if (await hasWritePermission('./data')) { + return 'filesystem' // Best for servers + } else if (process.env.S3_BUCKET) { + return 's3' // Cloud deployment + } else { + return 'memory' // Fallback for restricted environments + } + } + + // Browser environments + if (environment.isBrowser) { + if (await navigator.storage.estimate() > 1GB) { + return 'opfs' // Best for modern browsers + } else if (indexedDB) { + return 'indexeddb' // Fallback for older browsers + } else { + return 'memory' // In-memory for restricted contexts + } + } + + // Edge environments + if (environment.isEdge) { + return 'kv' // Use edge KV stores (Cloudflare, Vercel) + } +} ``` -The default JS index is `JsHnswVectorIndex`. An optional native acceleration -provider (the `@soulcraft/cor` package) can replace it with a -higher-performing implementation; the public knobs stay the same. Quantization -and other index-internal acceleration are the native provider's concern, not a -Brainy configuration option. +### Storage Migration -### 3. Persistence mode follows the backend - -`config.vector.persistMode` accepts `'immediate'` or `'deferred'`. Left unset, -Brainy chooses for you: - -- **Immediate** on filesystem storage, so the index file stays in lock-step with - the data and survives a crash. -- **Deferred** on in-memory storage, where there is nothing durable to sync to, - so writes are batched for throughput. +Brainy seamlessly migrates between storage types: ```typescript -const brain = new Brainy({ - vector: { persistMode: 'deferred' } // batch persistence for write-heavy loads +// Start with memory storage (development) +const brain = new Brainy() // Auto-selects memory + +// Later, migrate to production storage +await brain.migrate({ + to: 'filesystem', + path: './production-data' }) +// All data seamlessly transferred ``` -### 4. Memory-aware cache and buffer sizing +## Learning & Optimization 🚧 Coming Soon -Brainy reads the container's memory budget — `CLOUD_RUN_MEMORY`, `MEMORY_LIMIT`, -or the cgroup memory limit when running in a container — and sizes its read -caches and write buffers to fit. On a small instance it stays conservative; on a -large one it uses more of the available headroom. Query-result limits are capped -against the same budget (roughly 25 KB per result) to keep a single oversized -query from exhausting memory. +> **Note**: These features are planned for Q2 2025. Currently, Brainy uses static optimizations. -You can pin the cache explicitly: +### Query Pattern Learning 🚧 Planned + +Brainy learns from your query patterns and optimizes accordingly: ```typescript -const brain = new Brainy({ - cache: { maxSize: 10000, ttl: 3_600_000 } -}) +// Brainy observes query patterns +class QueryPatternLearner { + analyze(queries: Query[]) { + return { + // Frequency analysis + mostCommonFields: this.getTopFields(queries), + avgResultSize: this.getAvgSize(queries), + temporalPatterns: this.getTimePatterns(queries), + + // Relationship analysis + commonTraversals: this.getGraphPatterns(queries), + typicalDepth: this.getAvgDepth(queries), + + // Performance analysis + slowQueries: this.getSlowQueries(queries), + cacheability: this.getCacheability(queries) + } + } +} + +// Automatic optimizations based on learning: +// - Creates indexes for frequently queried fields +// - Pre-computes common graph traversals +// - Adjusts cache sizes based on working set +// - Optimizes vector search parameters ``` -### 5. Logging quiets in production +### Auto-Indexing 🚧 Planned -Brainy detects production-style environments (for example `NODE_ENV` set to a -non-development value) and reduces its own log verbosity automatically. This is -logging-only behavior — it does not change indexing, storage, or query results. +Brainy automatically creates indexes based on usage: + +```typescript +// No manual index configuration needed +await brain.find({ where: { category: "tech" } }) // First query +// Brainy notices 'category' field usage + +await brain.find({ where: { category: "science" } }) // Second query +// Pattern detected - auto-creates category index + +await brain.find({ where: { category: "tech" } }) // Third query +// Now using index - 100x faster! +``` + +### Adaptive Caching 🚧 Planned + +Cache strategies adapt to your access patterns: + +```typescript +class AdaptiveCache { + async adapt(metrics: AccessMetrics) { + if (metrics.hitRate < 0.3) { + // Low hit rate - switch strategy + this.strategy = 'lfu' // Least Frequently Used + } else if (metrics.workingSet > this.size) { + // Working set too large - increase size + this.size = Math.min(metrics.workingSet * 1.5, maxMemory) + } else if (metrics.temporalLocality > 0.8) { + // High temporal locality - use time-based eviction + this.strategy = 'ttl' + this.ttl = metrics.avgAccessInterval * 2 + } + } +} +``` + +## Performance Auto-Scaling 🚧 Coming Soon + +### Dynamic Batch Sizing + +Brainy adjusts batch sizes based on system load: + +```typescript +class DynamicBatcher { + calculateOptimalBatch() { + const cpuUsage = process.cpuUsage() + const memoryUsage = process.memoryUsage() + + if (cpuUsage < 30 && memoryUsage < 50) { + return 1000 // System idle - large batches + } else if (cpuUsage < 60 && memoryUsage < 70) { + return 100 // Moderate load - medium batches + } else { + return 10 // High load - small batches + } + } +} + +// Automatically applied during bulk operations +for (const item of millionItems) { + await brain.add(item) // Internally batched optimally +} +``` + +### Memory Management + +Automatic memory pressure handling: + +```typescript +class MemoryManager { + async handlePressure() { + const usage = process.memoryUsage() + const available = os.freemem() + + if (available < 100 * 1024 * 1024) { // Less than 100MB free + // Emergency mode + await this.flushCaches() + await this.compactIndexes() + await this.offloadToDisk() + } else if (usage.heapUsed / usage.heapTotal > 0.9) { + // Preventive mode + await this.reduceCacheSizes() + await this.pauseBackgroundTasks() + } + } +} +``` + +### Connection Pooling + +Automatic connection management for storage backends: + +```typescript +class ConnectionPool { + async getOptimalPoolSize() { + // Adapts based on workload + const metrics = await this.getMetrics() + + if (metrics.waitTime > 100) { + // Queries waiting - increase pool + this.size = Math.min(this.size * 1.5, this.maxSize) + } else if (metrics.idleConnections > this.size * 0.5) { + // Too many idle - decrease pool + this.size = Math.max(this.size * 0.7, this.minSize) + } + + return this.size + } +} +``` + +## Model Auto-Selection + +### Embedding Model Selection + +Brainy chooses the best embedding model for your use case: + +```typescript +async function autoSelectModel(data: Sample[]) { + const analysis = { + languages: detectLanguages(data), + domainSpecific: detectDomain(data), + averageLength: getAvgLength(data), + requiresMultilingual: languages.length > 1 + } + + if (analysis.requiresMultilingual) { + return 'multilingual-e5-base' // Handles 100+ languages + } else if (analysis.domainSpecific === 'code') { + return 'codebert-base' // Optimized for code + } else if (analysis.averageLength > 512) { + return 'all-mpnet-base-v2' // Better for long text + } else { + return 'all-MiniLM-L6-v2' // Fast and efficient default + } +} +``` + +### Model Downloading + +Models are automatically downloaded when needed: + +```typescript +// First use - model auto-downloads +const brain = new Brainy() +await brain.init() // Downloads model if not cached + +// Intelligent model caching +const modelCache = { + location: process.env.MODEL_CACHE || '~/.brainy/models', + maxSize: 5 * 1024 * 1024 * 1024, // 5GB max + strategy: 'lru', // Least recently used eviction + + // CDN selection based on location + cdn: await selectFastestCDN([ + 'https://cdn.brainy.io', + 'https://brainy.b-cdn.net', + 'https://models.huggingface.co' + ]) +} +``` + +## Workload Detection + +### Pattern Recognition + +Brainy identifies your workload type and optimizes: + +```typescript +enum WorkloadType { + OLTP = 'oltp', // Many small transactions + OLAP = 'olap', // Analytical queries + STREAMING = 'streaming', // Real-time ingestion + BATCH = 'batch', // Bulk processing + HYBRID = 'hybrid' // Mixed workload +} + +class WorkloadDetector { + detect(metrics: OperationMetrics): WorkloadType { + if (metrics.writesPerSecond > 1000 && metrics.avgWriteSize < 1024) { + return WorkloadType.STREAMING + } else if (metrics.avgQueryComplexity > 0.8 && metrics.avgResultSize > 10000) { + return WorkloadType.OLAP + } else if (metrics.batchOperations > metrics.singleOperations) { + return WorkloadType.BATCH + } else if (metrics.writeReadRatio > 0.3 && metrics.writeReadRatio < 0.7) { + return WorkloadType.HYBRID + } else { + return WorkloadType.OLTP + } + } +} +``` + +### Optimization Strategies + +Different optimizations for different workloads: + +```typescript +class WorkloadOptimizer { + optimize(workload: WorkloadType) { + switch (workload) { + case WorkloadType.STREAMING: + return { + entityRegistry: true, // Deduplication + batchSize: 1000, + walEnabled: true, + cacheSize: 'small', + indexStrategy: 'lazy' + } + + case WorkloadType.OLAP: + return { + entityRegistry: false, + batchSize: 10000, + walEnabled: false, + cacheSize: 'large', + indexStrategy: 'eager', + parallelQueries: true + } + + case WorkloadType.BATCH: + return { + entityRegistry: false, + batchSize: 50000, + walEnabled: false, + cacheSize: 'minimal', + indexStrategy: 'deferred' + } + + default: + return this.defaultConfig + } + } +} +``` + +## Hardware Adaptation 🚧 Coming Soon + +> **Note**: GPU acceleration and hardware optimization planned for Q3 2025. + +### CPU Optimization + +Adapts to available CPU resources: + +```typescript +class CPUAdapter { + async optimize() { + const cores = os.cpus().length + const type = os.cpus()[0].model + + // Parallel processing based on cores + this.parallelism = Math.max(1, cores - 1) // Leave one core free + + // SIMD detection for vector operations + if (type.includes('Intel') || type.includes('AMD')) { + this.enableSIMD = await checkSIMDSupport() + } + + // Thread pool sizing + this.threadPoolSize = cores * 2 // Optimal for I/O bound + + // Vector search optimization + if (cores >= 8) { + this.hnswConstruction = 200 // Higher quality index + this.hnswSearch = 100 // More accurate search + } else { + this.hnswConstruction = 100 // Balanced + this.hnswSearch = 50 // Faster search + } + } +} +``` + +### Memory Adaptation + +Intelligent memory allocation: + +```typescript +class MemoryAdapter { + async configure() { + const totalMemory = os.totalmem() + const availableMemory = os.freemem() + + // Allocate based on available memory + const allocation = { + cache: Math.min(availableMemory * 0.25, 2 * GB), + vectors: Math.min(availableMemory * 0.30, 4 * GB), + indexes: Math.min(availableMemory * 0.20, 2 * GB), + working: Math.min(availableMemory * 0.25, 2 * GB) + } + + // Adjust for low memory systems + if (totalMemory < 4 * GB) { + allocation.cache *= 0.5 + allocation.vectors *= 0.7 + this.enableSwapping = true + } + + return allocation + } +} +``` + +### GPU Acceleration + +Automatic GPU detection and utilization: + +```typescript +class GPUAdapter { + async detect() { + // WebGPU in browsers + if (navigator?.gpu) { + const adapter = await navigator.gpu.requestAdapter() + return { + available: true, + type: 'webgpu', + memory: adapter.limits.maxBufferSize, + compute: adapter.limits.maxComputeWorkgroupsPerDimension + } + } + + // CUDA in Node.js + if (process.platform === 'linux' || process.platform === 'win32') { + const hasCuda = await checkCudaSupport() + if (hasCuda) { + return { + available: true, + type: 'cuda', + memory: await getCudaMemory(), + compute: await getCudaCores() + } + } + } + + return { available: false } + } + + async optimize(gpu: GPUInfo) { + if (gpu.available) { + // Offload vector operations to GPU + this.vectorOps = 'gpu' + this.embeddingGeneration = 'gpu' + this.matrixMultiplication = 'gpu' + + // Larger batch sizes for GPU + this.batchSize = gpu.memory > 8 * GB ? 10000 : 1000 + } + } +} +``` + +## Network Adaptation + +### Bandwidth Detection + +Optimizes for available network bandwidth: + +```typescript +class NetworkAdapter { + async measureBandwidth() { + const testSize = 1 * MB + const start = Date.now() + await this.transfer(testSize) + const duration = Date.now() - start + + const bandwidth = (testSize / duration) * 1000 // bytes/sec + + if (bandwidth < 1 * MB) { + // Low bandwidth - optimize + this.compression = 'aggressive' + this.batchTransfers = true + this.cacheRemote = true + } else if (bandwidth > 100 * MB) { + // High bandwidth + this.compression = 'minimal' + this.parallelTransfers = true + } + } +} +``` + +### Latency Optimization + +Adapts to network latency: + +```typescript +class LatencyOptimizer { + async optimize() { + const latency = await this.measureLatency() + + if (latency > 100) { // High latency + // Batch operations + this.minBatchSize = 100 + + // Aggressive prefetching + this.prefetchDepth = 3 + + // Local caching + this.cacheStrategy = 'aggressive' + + // Connection pooling + this.connectionPool = Math.min(latency / 10, 50) + } + } +} +``` + +## Cloud Provider Detection 🚧 Coming Soon + +> **Note**: Cloud provider auto-detection planned for Q3 2025. + +### Automatic Cloud Optimization + +Detects and optimizes for cloud providers: + +```typescript +class CloudDetector { + async detect() { + // AWS Detection + if (process.env.AWS_REGION || await canReachMetadata('169.254.169.254')) { + return { + provider: 'aws', + instance: await getEC2InstanceType(), + region: process.env.AWS_REGION, + services: { + storage: 's3', + cache: 'elasticache', + compute: 'lambda' + } + } + } + + // Google Cloud Detection + if (process.env.GOOGLE_CLOUD_PROJECT || await canReachMetadata('metadata.google.internal')) { + return { + provider: 'gcp', + instance: await getGCEInstanceType(), + region: process.env.GOOGLE_CLOUD_REGION, + services: { + storage: 'gcs', + cache: 'memorystore', + compute: 'cloud-run' + } + } + } + + // Vercel Edge Detection + if (process.env.VERCEL) { + return { + provider: 'vercel', + region: process.env.VERCEL_REGION, + services: { + storage: 'vercel-kv', + cache: 'edge-config', + compute: 'edge-runtime' + } + } + } + } +} +``` + +## Development vs Production + +### Automatic Environment Detection + +```typescript +class EnvironmentDetector { + detect() { + const indicators = { + // Development indicators + isDevelopment: + process.env.NODE_ENV === 'development' || + process.env.DEBUG || + process.argv.includes('--dev') || + isLocalhost() || + hasDevTools(), + + // Test indicators + isTest: + process.env.NODE_ENV === 'test' || + process.env.CI || + isTestRunner(), + + // Production indicators + isProduction: + process.env.NODE_ENV === 'production' || + process.env.VERCEL || + process.env.NETLIFY || + !isLocalhost() + } + + return indicators + } +} + +// Different defaults for different environments +const config = environment.isProduction ? { + storage: 'filesystem', + wal: true, + monitoring: true, + compression: true, + caching: 'aggressive' +} : { + storage: 'memory', + wal: false, + monitoring: false, + compression: false, + caching: 'minimal' +} +``` + +## Error Recovery + +### Automatic Fallbacks + +Brainy automatically recovers from errors: + +```typescript +class AutoRecovery { + async handleStorageFailure() { + try { + await this.primaryStorage.write(data) + } catch (error) { + console.warn('Primary storage failed, trying fallback') + + // Try secondary storage + if (this.secondaryStorage) { + await this.secondaryStorage.write(data) + } else { + // Fall back to memory + await this.memoryStorage.write(data) + + // Schedule retry + this.scheduleRetry(data) + } + } + } + + async handleModelFailure() { + try { + return await this.primaryModel.embed(text) + } catch (error) { + // Fall back to simpler model + return await this.fallbackModel.embed(text) + } + } +} +``` ## Configuration Override -Zero-config is the default, not a ceiling. Every adaptive decision above has an -explicit constructor option: +While zero-config is default, you can override when needed: ```typescript +// Explicit configuration when needed const brain = new Brainy({ - storage: { type: 'filesystem', path: '/var/lib/brainy' }, - vector: { - recall: 'accurate', - persistMode: 'immediate' + // Override auto-detection + storage: { + type: 'filesystem', + path: '/custom/path' }, - cache: { maxSize: 50000, ttl: 600_000 } + + // Override auto-optimization + optimization: { + autoIndex: false, + autoCache: false, + autoBatch: false + }, + + // Override auto-scaling + scaling: { + maxMemory: 2 * GB, + maxConnections: 100, + maxBatchSize: 1000 + } }) - -await brain.init() ``` -See the [API Reference](../api/README.md#configuration) for the complete option -list. +## Monitoring Auto-Adaptation + +Brainy provides visibility into its auto-adaptation: + +```typescript +brain.on('adaptation', (event) => { + console.log(`Brainy adapted: ${event.type}`) + console.log(`Reason: ${event.reason}`) + console.log(`Before: ${JSON.stringify(event.before)}`) + console.log(`After: ${JSON.stringify(event.after)}`) +}) + +// Example events: +// - Index created for frequently queried field +// - Cache strategy changed due to low hit rate +// - Batch size increased due to high throughput +// - Storage migrated due to space constraints +// - Model switched due to multilingual content +``` + +## Conclusion + +Brainy's zero-configuration and auto-adaptation capabilities mean you can focus on your application logic while Brainy handles: + +- Environment detection and optimization +- Storage selection and migration +- Performance tuning and scaling +- Resource management +- Error recovery +- Workload optimization + +Just create a Brainy instance and start using it. Brainy will learn, adapt, and optimize itself for your specific use case—no configuration required. ## See Also - [Architecture Overview](./overview.md) -- [Storage Adapters](../concepts/storage-adapters.md) -- [Scaling Guide](../SCALING.md) -- [API Reference](../api/README.md) +- [Storage Architecture](./storage.md) +- [Performance Guide](../guides/performance.md) +- [Augmentations System](./augmentations.md) \ No newline at end of file diff --git a/docs/augmentations/COMPLETE-REFERENCE.md b/docs/augmentations/COMPLETE-REFERENCE.md new file mode 100644 index 00000000..73ef6173 --- /dev/null +++ b/docs/augmentations/COMPLETE-REFERENCE.md @@ -0,0 +1,451 @@ +# 🔌 Brainy v4.0.0 Augmentations Complete Reference + +> **All augmentations that power Brainy's extensibility - with locations, usage, and examples** +> +> **⚠️ v4.0.0 Update**: Updated for metadata structure changes and billion-scale optimizations + +## Quick Start + +```typescript +import { Brainy } from '@soulcraft/brainy' + +const brain = new Brainy({ + // Augmentations auto-configure based on environment + storage: 'auto', // Storage augmentation + cache: true, // Cache augmentation + index: true // Index augmentation +}) + +await brain.init() // Augmentations initialize automatically +``` + +## v4.0.0 Augmentation Architecture + +### Key Improvements for Billion-Scale Performance + +1. **Metadata/Vector Separation**: Augmentations now work with separated metadata and vectors + - Metadata stored separately from vector data + - 99.2% memory reduction for type tracking + - Two-file storage pattern for optimal I/O + +2. **Type System Enforcement**: All metadata requires type fields + - `NounMetadata` requires `noun: NounType` + - `VerbMetadata` requires `verb: VerbType` + - Type inference system available as public API + +3. **Storage Adapter Pattern**: Internal vs public method distinction + - `_methods`: Return pure structures (HNSWNoun, HNSWVerb) + - Public methods: Return WithMetadata types + - MetadataEnforcer Proxy ensures proper access + +### What This Means for Augmentation Users + +**✅ If you use built-in augmentations**: No changes needed! They're all updated for v4.0.0. + +**⚠️ If you created custom storage augmentations**: Update your storage adapter to: +- Wrap metadata with required `noun`/`verb` fields +- Follow the internal/public method pattern +- Use two-file storage approach + +**⚠️ If you access relationship data**: Change `verb.type` to `verb.verb` + +## Core Concepts + +### What are Augmentations? +Augmentations are modular extensions that add functionality to Brainy without cluttering the core API. They follow a unified interface and can be: +- **Auto-enabled**: Based on configuration (cache, index, storage) +- **Manually registered**: For custom functionality +- **Chained**: Multiple augmentations work together seamlessly +- **Billion-scale ready**: Optimized for datasets with billions of nouns and verbs + +### Augmentation Lifecycle +1. **Registration**: Augmentations register before init() +2. **Initialization**: Two-phase init (storage first, then others) +3. **Execution**: Hook into operations (before/after/both) +4. **Shutdown**: Clean teardown on brain.shutdown() + +--- + +## Storage Augmentations (8 total) + +### MemoryStorageAugmentation +**Location**: `src/augmentations/storageAugmentations.ts` +**Auto-enabled**: When `storage: 'memory'` or in test environments +**Purpose**: In-memory storage for testing and temporary data +```typescript +const brain = new Brainy({ storage: 'memory' }) +``` + +### FileSystemStorageAugmentation +**Location**: `src/augmentations/storageAugmentations.ts` +**Auto-enabled**: When `storage: 'filesystem'` or Node.js detected +**Purpose**: Persistent file-based storage for Node.js applications +```typescript +const brain = new Brainy({ + storage: { type: 'filesystem', path: './data' } +}) +``` + +### OPFSStorageAugmentation +**Location**: `src/augmentations/storageAugmentations.ts` +**Auto-enabled**: When `storage: 'opfs'` or browser with OPFS support +**Purpose**: Browser-based persistent storage using Origin Private File System +```typescript +const brain = new Brainy({ storage: 'opfs' }) +``` + +### S3StorageAugmentation +**Location**: `src/augmentations/storageAugmentations.ts` +**Manual**: Requires AWS credentials +**Purpose**: AWS S3-compatible cloud storage +```typescript +const brain = new Brainy({ + storage: { + type: 's3', + bucket: 'my-bucket', + region: 'us-east-1', + credentials: { accessKeyId, secretAccessKey } + } +}) +``` + +### R2StorageAugmentation +**Location**: `src/augmentations/storageAugmentations.ts` +**Manual**: Requires Cloudflare credentials +**Purpose**: Cloudflare R2 storage (S3-compatible) +```typescript +const brain = new Brainy({ + storage: { + type: 'r2', + accountId: 'xxx', + bucket: 'my-bucket', + credentials: { accessKeyId, secretAccessKey } + } +}) +``` + +### GCSStorageAugmentation +**Location**: `src/augmentations/storageAugmentations.ts` +**Manual**: Requires Google Cloud credentials +**Purpose**: Google Cloud Storage +```typescript +const brain = new Brainy({ + storage: { + type: 'gcs', + bucket: 'my-bucket', + projectId: 'my-project' + } +}) +``` + +### StorageAugmentation (base) +**Location**: `src/augmentations/storageAugmentation.ts` +**Purpose**: Base class for custom storage implementations + +### DynamicStorageAugmentation +**Location**: `src/augmentations/storageAugmentation.ts` +**Purpose**: Runtime storage adapter switching + +--- + +## Performance Augmentations (7 total) + +### CacheAugmentation +**Location**: `src/augmentations/cacheAugmentation.ts` +**Auto-enabled**: When `cache: true` (default) +**Purpose**: LRU cache for search results and frequent queries +```typescript +brain.clearCache() // Exposed via API +brain.getCacheStats() // Cache hit/miss statistics +``` + +### IndexAugmentation +**Location**: `src/augmentations/indexAugmentation.ts` +**Auto-enabled**: When `index: true` (default) +**Purpose**: Metadata indexing for O(1) field lookups +```typescript +brain.rebuildMetadataIndex() // Exposed via API +// Enables fast where queries: +brain.find({ where: { category: 'tech' } }) +``` + +### MetricsAugmentation +**Location**: `src/augmentations/metricsAugmentation.ts` +**Auto-enabled**: Always active +**Purpose**: Performance metrics and statistics collection +```typescript +brain.getStats() // Comprehensive metrics +``` + +### MonitoringAugmentation +**Location**: `src/augmentations/monitoringAugmentation.ts` +**Manual**: Register for detailed monitoring +**Purpose**: Real-time performance monitoring and alerts + +### BatchProcessingAugmentation +**Location**: `src/augmentations/batchProcessingAugmentation.ts` +**Auto-enabled**: For batch operations +**Purpose**: Optimizes bulk add/update/delete operations +```typescript +brain.addNouns([...]) // Automatically batched +``` + +### RequestDeduplicatorAugmentation +**Location**: `src/augmentations/requestDeduplicatorAugmentation.ts` +**Auto-enabled**: Always active +**Purpose**: Prevents duplicate concurrent operations + +### ConnectionPoolAugmentation +**Location**: `src/augmentations/connectionPoolAugmentation.ts` +**Auto-enabled**: For network storage +**Purpose**: Connection pooling for cloud storage adapters + +--- + +## Data Integrity Augmentations (3 total) + +**Auto-enabled**: When `wal: true` +**Purpose**: Write-ahead logging for crash recovery +```typescript +const brain = new Brainy({ wal: true }) +// Automatic recovery on restart after crash +``` + +### EntityRegistryAugmentation +**Location**: `src/augmentations/entityRegistryAugmentation.ts` +**Auto-enabled**: For streaming operations +**Purpose**: High-speed deduplication for real-time data +```typescript +// Prevents duplicate entities in streaming scenarios +brain.add(data) // Automatically deduplicated +``` + +### AutoRegisterEntitiesAugmentation +**Location**: `src/augmentations/entityRegistryAugmentation.ts` +**Manual**: For automatic entity discovery +**Purpose**: Auto-discovers and registers entities from data + +--- + +## Intelligence Augmentations (2 total) + +### NeuralImportAugmentation +**Location**: `src/augmentations/neuralImport.ts` +**Manual**: Via `brain.neuralImport()` +**Purpose**: AI-powered smart data import +```typescript +const result = await brain.neuralImport(data, { + confidenceThreshold: 0.7, + autoApply: true +}) +// Automatically detects entities and relationships +``` + +### IntelligentVerbScoringAugmentation +**Location**: `src/augmentations/intelligentVerbScoringAugmentation.ts` +**Auto-enabled**: When verbs are used +**Purpose**: ML-based relationship strength scoring +```typescript +brain.verbScoring.train(feedback) +brain.verbScoring.getScore(verbId) +``` + +--- + +## Communication Augmentations (4 total) + +### APIServerAugmentation +**Location**: `src/augmentations/apiServerAugmentation.ts` +**Manual**: For server deployments +**Purpose**: REST/WebSocket/MCP API server +```typescript +const augmentation = new APIServerAugmentation() +await brain.registerAugmentation(augmentation) +// Exposes full Brainy API over network +``` + +### WebSocketConduitAugmentation +**Location**: `src/augmentations/conduitAugmentations.ts` +**Manual**: For Brainy-to-Brainy sync +**Purpose**: Real-time sync between Brainy instances +```typescript +const conduit = new WebSocketConduitAugmentation() +await conduit.establishConnection('ws://other-brain') +``` + +### ServerSearchConduitAugmentation +**Location**: `src/augmentations/serverSearchAugmentations.ts` +**Manual**: For client-server search +**Purpose**: Search remote Brainy instance, cache locally + +### ServerSearchActivationAugmentation +**Location**: `src/augmentations/serverSearchAugmentations.ts` +**Manual**: Works with ServerSearchConduit +**Purpose**: Triggers and manages server search operations + +--- + +## External Integration (2 total) + +### SynapseAugmentation (base) +**Location**: `src/augmentations/synapseAugmentation.ts` +**Purpose**: Base class for external platform integrations +```typescript +// Example: NotionSynapse, SlackSynapse, etc. +class NotionSynapse extends SynapseAugmentation { + async fetchData() { /* Notion API calls */ } + async pushData() { /* Sync to Notion */ } +} +``` + +### ExampleFileSystemSynapse +**Location**: `src/augmentations/synapseAugmentation.ts` +**Purpose**: Example implementation for file system sync + +--- + +## Augmentation Configuration + +### Auto-Configuration +```typescript +const brain = new Brainy({ + // These auto-register augmentations: + storage: 'auto', // Storage augmentation + cache: true, // Cache augmentation + index: true, // Index augmentation + metrics: true // Metrics augmentation +}) +``` + +### Manual Registration +```typescript +const brain = new Brainy() + +// Register before init() +const customAug = new MyCustomAugmentation() +await brain.registerAugmentation(customAug) + +await brain.init() +``` + +### Creating Custom Augmentations +```typescript +import { BaseAugmentation } from '@soulcraft/brainy' + +class MyAugmentation extends BaseAugmentation { + readonly name = 'my-augmentation' + readonly timing = 'after' // before | after | both + readonly operations = ['addNoun', 'search'] // Which ops to hook + readonly priority = 10 // Execution order (lower = earlier) + + protected async onInit(): Promise { + // Initialize your augmentation + } + + async execute( + operation: string, + params: any, + context?: AugmentationContext + ): Promise { + // Your augmentation logic + if (operation === 'addNoun') { + console.log('Noun added:', params) + } + } + + protected async onShutdown(): Promise { + // Cleanup + } +} +``` + +--- + +## Augmentation Timing & Priority + +### Timing Options +- **`before`**: Runs before the operation (can modify params) +- **`after`**: Runs after the operation (can see results) +- **`both`**: Runs before AND after + +### Priority (lower = earlier) +1. Storage augmentations (priority: 0) +2. Cache/Index augmentations (priority: 5-10) +3. Monitoring/Metrics (priority: 15-20) +4. Conduits/Synapses (priority: 20-30) + +--- + +## Key Integration Points + +### Where Augmentations Hook In + +**Brainy Constructor**: +- Storage augmentations register based on config +- Cache/Index augmentations auto-register if enabled + +**brain.init()**: +- Two-phase initialization (storage first, then others) +- Augmentations can access brain instance via context + +**Operations** (addNoun, search, etc.): +- Augmentations execute based on timing and operations filter +- Can modify params (before) or see results (after) + +**brain.shutdown()**: +- All augmentations cleaned up in reverse order + +--- + +## Performance Impact + +Most augmentations have minimal overhead: +- **Cache**: ~1ms per search (saves 10-100ms on hits) +- **Index**: ~1ms per operation (saves 100ms+ on queries) +- **Metrics**: <1ms per operation +- **Storage**: Varies by adapter (memory: 0ms, S3: 50-200ms) + +--- + +## Best Practices + +1. **Let auto-configuration work**: Most apps need zero manual config +2. **Storage first**: Always configure storage before other augmentations +3. **Use built-in augmentations**: They're optimized and battle-tested +4. **Custom augmentations**: Extend BaseAugmentation for consistency +5. **Respect timing**: Use 'before' to modify, 'after' to observe +6. **Mind priority**: Lower numbers execute first + +--- + +## Troubleshooting + +### Augmentation not working? +```typescript +// Check if registered +brain.listAugmentations() + +// Check if enabled +brain.isAugmentationEnabled('cache') + +// Enable/disable at runtime +brain.enableAugmentation('cache') +brain.disableAugmentation('cache') +``` + +### Performance issues? +```typescript +// Check augmentation overhead +const stats = brain.getStats() +console.log(stats.augmentations) + +// Disable non-critical augmentations +brain.disableAugmentation('monitoring') +``` + +--- + + +--- + +*Augmentations make Brainy infinitely extensible while keeping the core API clean and simple!* \ No newline at end of file diff --git a/docs/augmentations/CONFIGURATION.md b/docs/augmentations/CONFIGURATION.md new file mode 100644 index 00000000..3a7bd112 --- /dev/null +++ b/docs/augmentations/CONFIGURATION.md @@ -0,0 +1,620 @@ +# Augmentation Configuration System + +**Version**: 2.0.0 +**Status**: Production Ready + +## Overview + +The Brainy Augmentation Configuration System provides a VSCode-style extension architecture with multiple configuration sources, schema validation, and tool discovery. This system maintains Brainy's zero-config philosophy while enabling sophisticated enterprise configuration management. + +## Table of Contents +- [Quick Start](#quick-start) +- [Configuration Sources](#configuration-sources) +- [Creating Configurable Augmentations](#creating-configurable-augmentations) +- [Configuration Discovery](#configuration-discovery) +- [Runtime Configuration](#runtime-configuration) +- [Environment Variables](#environment-variables) +- [Configuration Files](#configuration-files) +- [CLI Commands](#cli-commands) +- [Tool Integration](#tool-integration) +- [Migration Guide](#migration-guide) + +## Quick Start + +### Using an Augmentation with Configuration + +```typescript +import { Brainy } from '@soulcraft/brainy' + +// Zero-config (uses defaults) +const brain = new Brainy() + +// With custom configuration + immediateWrites: true, + checkpointInterval: 300000 // 5 minutes +})) +``` + +### Configuring via Environment Variables + +```bash +export BRAINY_AUG_CACHE_TTL=600000 +``` + +### Configuring via Files + +Create a `.brainyrc` file in your project root: + +```json +{ + "augmentations": { + "wal": { + "enabled": true, + "immediateWrites": true, + "maxSize": 20971520 + }, + "cache": { + "ttl": 600000, + "maxSize": 2000 + } + } +} +``` + +## Configuration Sources + +Configuration is resolved in the following priority order (highest to lowest): + +1. **Runtime Updates** - Dynamic configuration changes via API +2. **Constructor Parameters** - Code-time configuration +3. **Environment Variables** - `BRAINY_AUG__` +4. **Configuration Files** - `.brainyrc`, `brainy.config.json` +5. **Schema Defaults** - Default values from manifest + +### Resolution Example + +```typescript +// Schema default +{ maxSize: 10485760 } + +// File configuration (.brainyrc) +{ maxSize: 20971520 } + +// Environment variable + +// Constructor parameter + +// Final resolved value: 41943040 (constructor wins) +``` + +## Creating Configurable Augmentations + +### Step 1: Extend ConfigurableAugmentation + +```typescript +import { ConfigurableAugmentation, AugmentationManifest } from '@soulcraft/brainy' + +export class MyAugmentation extends ConfigurableAugmentation { + name = 'my-augmentation' + timing = 'around' as const + metadata = 'none' as const + operations = ['search', 'add'] + priority = 50 + + constructor(config?: MyConfig) { + super(config) // Handles configuration resolution + } + + // Required: Provide manifest for discovery + getManifest(): AugmentationManifest { + return { + id: 'my-augmentation', + name: 'My Augmentation', + version: '1.0.0', + description: 'Does something amazing', + category: 'performance', + configSchema: { + type: 'object', + properties: { + enabled: { + type: 'boolean', + default: true, + description: 'Enable this augmentation' + }, + threshold: { + type: 'number', + default: 100, + minimum: 1, + maximum: 1000, + description: 'Processing threshold' + } + } + } + } + } + + // Optional: Handle runtime configuration changes + protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise { + if (newConfig.threshold !== oldConfig.threshold) { + // React to threshold change + this.updateThreshold(newConfig.threshold) + } + } + + async execute(operation: string, params: any, next: () => Promise): Promise { + if (!this.config.enabled) { + return next() + } + + // Your augmentation logic here + return next() + } +} +``` + +### Step 2: Define Configuration Interface + +```typescript +interface MyConfig { + enabled?: boolean + threshold?: number + mode?: 'fast' | 'balanced' | 'thorough' +} +``` + +### Step 3: Add JSON Schema in Manifest + +```typescript +configSchema: { + type: 'object', + properties: { + enabled: { + type: 'boolean', + default: true, + description: 'Enable augmentation' + }, + threshold: { + type: 'number', + default: 100, + minimum: 1, + maximum: 1000, + description: 'Processing threshold' + }, + mode: { + type: 'string', + default: 'balanced', + enum: ['fast', 'balanced', 'thorough'], + description: 'Processing mode' + } + }, + required: [], + additionalProperties: false +} +``` + +## Configuration Discovery + +The Discovery API allows tools to discover and configure augmentations dynamically: + +```typescript +import { AugmentationDiscovery } from '@soulcraft/brainy' + +const discovery = new AugmentationDiscovery(brain.augmentations) + +// Discover all augmentations with manifests +const listings = await discovery.discover({ + includeConfig: true, + includeSchema: true +}) + +// Get configuration schema +const schema = await discovery.getConfigSchema('wal') + +// Validate configuration +const validation = await discovery.validateConfig('wal', { + enabled: true, + maxSize: 'invalid' // Will fail validation +}) + +// Update configuration at runtime +await discovery.updateConfig('wal', { + checkpointInterval: 120000 +}) +``` + +## Runtime Configuration + +### Update Configuration Dynamically + +```typescript +// Get augmentation +const wal = brain.augmentations.get('wal') + +// Update configuration +await wal.updateConfig({ + checkpointInterval: 300000 +}) + +// Get current configuration +const config = wal.getConfig() +``` + +### React to Configuration Changes + +```typescript +class MyAugmentation extends ConfigurableAugmentation { + protected async onConfigChange(newConfig: any, oldConfig: any): Promise { + // Stop old processes + if (oldConfig.enabled && !newConfig.enabled) { + await this.stop() + } + + // Start new processes + if (!oldConfig.enabled && newConfig.enabled) { + await this.start() + } + + // Update settings + if (newConfig.interval !== oldConfig.interval) { + this.rescheduleTimer(newConfig.interval) + } + } +} +``` + +## Environment Variables + +### Naming Convention + +```bash +BRAINY_AUG__=value +``` + +### Examples + +```bash + +# Cache augmentation +BRAINY_AUG_CACHE_ENABLED=true +BRAINY_AUG_CACHE_MAX_SIZE=2000 +BRAINY_AUG_CACHE_TTL=600000 + +# Complex values (JSON) +BRAINY_AUG_MYAUG_FILTERS='["*.js","*.ts"]' +BRAINY_AUG_MYAUG_OPTIONS='{"deep":true,"follow":false}' +``` + +### Docker Example + +```dockerfile +ENV BRAINY_AUG_CACHE_TTL=600000 +``` + +## Configuration Files + +### File Locations (Priority Order) + +1. `.brainyrc` (current directory) +2. `.brainyrc.json` (current directory) +3. `brainy.config.json` (current directory) +4. `~/.brainy/config.json` (user home) +5. `~/.brainyrc` (user home) + +### File Format + +```json +{ + "augmentations": { + "wal": { + "enabled": true, + "immediateWrites": true, + "maxSize": 20971520, + "checkpointInterval": 300000 + }, + "cache": { + "enabled": true, + "maxSize": 2000, + "ttl": 600000 + }, + "metrics": { + "enabled": false + } + } +} +``` + +### Per-Environment Configuration + +```json +{ + "augmentations": { + "wal": { + "development": { + "enabled": true, + "immediateWrites": true, + "maxSize": 5242880 + }, + "production": { + "enabled": true, + "immediateWrites": false, + "maxSize": 104857600, + "checkpointInterval": 60000 + } + } + } +} +``` + +## CLI Commands + +### List Augmentations with Configuration + +```bash +# Show all augmentations with config status +brainy augment list --detailed + +# Show configuration for specific augmentation +brainy augment config wal + +# Set configuration value +brainy augment config wal --set immediateWrites=true + +# Show environment variable names +brainy augment config wal --env + +# Export configuration schema +brainy augment schema wal > wal-schema.json + +# Validate configuration file +brainy augment validate --file config.json +``` + +### Interactive Configuration + +```bash +# Interactive configuration wizard +brainy augment configure wal + +? Operation mode? + ❯ Performance (immediate writes) + Durability (synchronous writes) + Custom +? Maximum log size? (10MB) 20MB +? Checkpoint interval? (1 minute) 5 minutes + +Configuration saved to .brainyrc +``` + +## Tool Integration + +### Brain-Cloud Explorer UI + +```typescript +// Auto-generate configuration form from schema +const ConfigurationUI = ({ augmentationId }) => { + const [manifest, setManifest] = useState(null) + const [config, setConfig] = useState({}) + + useEffect(() => { + // Fetch manifest with schema + fetch(`/api/augmentations/${augmentationId}/manifest`) + .then(res => res.json()) + .then(setManifest) + + // Get current configuration + discovery.getConfig(augmentationId) + .then(setConfig) + }, [augmentationId]) + + const handleSave = async (newConfig) => { + // Validate configuration + const validation = await fetch(`/api/augmentations/${augmentationId}/validate`, { + method: 'POST', + body: JSON.stringify(newConfig) + }).then(res => res.json()) + + if (validation.valid) { + // Apply configuration + await discovery.updateConfig(augmentationId, newConfig) + } + } + + // Render form based on schema + return +} +``` + +### VS Code Extension + +```json +// package.json contribution points +{ + "contributes": { + "configuration": { + "title": "Brainy Augmentations", + "properties": { + "brainy.augmentations.wal.enabled": { + "type": "boolean", + "default": true, + }, + "brainy.augmentations.wal.maxSize": { + "type": "number", + "default": 10485760, + } + } + } + } +} +``` + +## Migration Guide + +### Migrating from BaseAugmentation + +**Before:** +```typescript +export class MyAugmentation extends BaseAugmentation { + constructor(config: MyConfig = {}) { + super() + this.config = { + enabled: config.enabled ?? true, + threshold: config.threshold ?? 100 + } + } + + // No manifest + // No config discovery + // No runtime updates +} +``` + +**After:** +```typescript +export class MyAugmentation extends ConfigurableAugmentation { + constructor(config?: MyConfig) { + super(config) // Config resolution handled automatically + } + + getManifest(): AugmentationManifest { + return { + id: 'my-augmentation', + name: 'My Augmentation', + version: '1.0.0', + description: 'Does something amazing', + category: 'performance', + configSchema: { + type: 'object', + properties: { + enabled: { type: 'boolean', default: true }, + threshold: { type: 'number', default: 100 } + } + } + } + } + + // Optional: Handle config changes + protected async onConfigChange(newConfig: MyConfig, oldConfig: MyConfig): Promise { + // React to changes + } +} +``` + +### Backwards Compatibility + +The system maintains full backwards compatibility: + +1. **BaseAugmentation still works** - Existing augmentations continue to function +2. **Constructor config still works** - Existing configuration patterns preserved +3. **Zero-config still works** - Defaults are applied automatically +4. **Progressive enhancement** - Add features as needed + +## Best Practices + +### 1. Always Provide Defaults + +```typescript +configSchema: { + properties: { + enabled: { + type: 'boolean', + default: true, // Always provide defaults + description: 'Enable this feature' + } + } +} +``` + +### 2. Use Descriptive Configuration Keys + +```typescript +// Good +checkpointInterval: 60000 + +// Bad +ci: 60000 +``` + +### 3. Validate Configuration + +```typescript +protected async onConfigChange(newConfig: any, oldConfig: any): Promise { + // Validate before applying + if (newConfig.maxSize < 1048576) { + throw new Error('maxSize must be at least 1MB') + } + + // Apply changes + this.maxSize = newConfig.maxSize +} +``` + +### 4. Document Environment Variables + +```typescript +/** + * Environment Variables: + * - BRAINY_AUG_MYAUG_ENABLED: Enable augmentation (boolean) + * - BRAINY_AUG_MYAUG_THRESHOLD: Processing threshold (number) + * - BRAINY_AUG_MYAUG_MODE: Processing mode (fast|balanced|thorough) + */ +``` + +### 5. Provide Configuration Examples + +```typescript +configExamples: [ + { + name: 'Production', + description: 'Optimized for production use', + config: { + enabled: true, + mode: 'thorough', + threshold: 500 + } + }, + { + name: 'Development', + description: 'Lightweight for development', + config: { + enabled: true, + mode: 'fast', + threshold: 10 + } + } +] +``` + +## Troubleshooting + +### Configuration Not Loading + +1. Check file locations and names +2. Verify JSON syntax in config files +3. Check environment variable names (case-sensitive) +4. Use `brainy augment config --debug` to see resolution + +### Validation Errors + +1. Check schema requirements +2. Verify data types match schema +3. Check minimum/maximum constraints +4. Use discovery API to validate before applying + +### Runtime Updates Not Working + +1. Ensure augmentation extends ConfigurableAugmentation +2. Implement onConfigChange if needed +3. Check for validation errors +4. Verify augmentation is initialized + +## API Reference + +See the [Discovery API Documentation](./discovery-api.md) for complete API details. + +## Examples + +See the [examples directory](../../examples/augmentation-config/) for complete working examples. \ No newline at end of file diff --git a/docs/augmentations/DEVELOPER-GUIDE.md b/docs/augmentations/DEVELOPER-GUIDE.md new file mode 100644 index 00000000..20097ebf --- /dev/null +++ b/docs/augmentations/DEVELOPER-GUIDE.md @@ -0,0 +1,526 @@ +# 🛠️ Brainy Augmentation Developer Guide + +> **How to create, test, and use augmentations in Brainy v4.0.0** +> +> **⚠️ v4.0.0 Update**: This guide has been updated with breaking changes for metadata structure and type system improvements. + +## v4.0.0 Migration Guide + +### What Changed? + +1. **Metadata Structure**: All metadata now requires type fields (`noun` or `verb`) +2. **Property Rename**: `verb.type` → `verb.verb` for relationships +3. **Two-File Storage**: Vectors and metadata stored separately for performance +4. **Return Types**: Storage methods distinguish between internal (pure) and public (WithMetadata) returns + +### Migration Checklist + +- [ ] Update metadata creation to include required `noun` field +- [ ] Change `verb.type` to `verb.verb` in all relationship code +- [ ] Update storage adapter methods to follow internal/public pattern +- [ ] Ensure metadata access uses correct structure + +### Quick Migration Example + +```typescript +// ❌ v3.x +const verb = { + type: 'relatedTo', + sourceId: 'a', + targetId: 'b' +} +if (verb.type === 'relatedTo') { ... } + +// ✅ v4.0.0 +const verb = { + verb: 'relatedTo', + sourceId: 'a', + targetId: 'b' +} +const metadata: VerbMetadata = { + verb: 'relatedTo', + sourceId: 'a', + targetId: 'b' +} +if (verb.verb === 'relatedTo') { ... } +``` + +## Quick Start: Your First Augmentation + +```typescript +import { BaseAugmentation, BrainyAugmentation, AugmentationContext } from '@soulcraft/brainy' + +export class MyFirstAugmentation extends BaseAugmentation { + readonly name = 'my-first-augmentation' + readonly timing = 'after' as const // When to run: before | after | both + readonly operations = ['add'] as const // Which operations to hook + readonly priority = 10 // Execution order (lower = first) + + protected async onInit(): Promise { + // Initialize your augmentation + console.log('MyFirstAugmentation initialized!') + } + + async execute( + operation: string, + params: any, + context?: AugmentationContext + ): Promise { + // Your augmentation logic + if (operation === 'add') { + console.log('Noun added:', params.noun) + + // v4.0.0: Access metadata correctly + if (params.noun?.metadata) { + console.log('Noun type:', params.noun.metadata.noun) // Required field + } + + // You can access the brain instance + const stats = await context?.brain.getStats() + console.log('Total nouns:', stats.totalNouns) + } + } + + protected async onShutdown(): Promise { + // Cleanup + console.log('MyFirstAugmentation shutting down') + } +} +``` + +## Using Your Augmentation + +```typescript +import { Brainy } from '@soulcraft/brainy' +import { MyFirstAugmentation } from './my-first-augmentation' + +const brain = new Brainy() + +// Register before init() +brain.augmentations.register(new MyFirstAugmentation()) + +await brain.init() + +// Now your augmentation runs automatically! +await brain.add('Hello World') +// Console: "Noun added: { id: '...', vector: [...], metadata: {} }" +``` + +--- + +## Augmentation Lifecycle + +### 1. Registration Phase +```typescript +const aug = new MyAugmentation() +brain.augmentations.register(aug) // Before brain.init()! +``` + +### 2. Initialization Phase +```typescript +await brain.init() // Calls aug.initialize() internally +// Your onInit() method runs here +``` + +### 3. Execution Phase +```typescript +await brain.add('data') // Your execute() method runs +``` + +### 4. Shutdown Phase +```typescript +await brain.shutdown() // Your onShutdown() method runs +``` + +--- + +## Timing Options + +### `before` - Modify Input +```typescript +class ValidationAugmentation extends BaseAugmentation { + readonly timing = 'before' as const + + async execute(operation: string, params: any): Promise { + if (operation === 'add') { + // Validate and/or modify params + if (!params.content) { + throw new Error('Content required') + } + // Return modified params + return { ...params, validated: true } + } + } +} +``` + +### `after` - React to Results +```typescript +class LoggingAugmentation extends BaseAugmentation { + readonly timing = 'after' as const + + async execute(operation: string, params: any): Promise { + if (operation === 'search') { + console.log(`Search for "${params.query}" returned ${params.result.length} results`) + } + // Don't return anything - just observe + } +} +``` + +### `both` - Before AND After +```typescript +class TimingAugmentation extends BaseAugmentation { + readonly timing = 'both' as const + private startTime?: number + + async execute(operation: string, params: any, context?: AugmentationContext): Promise { + if (!this.startTime) { + // Before execution + this.startTime = Date.now() + } else { + // After execution + const duration = Date.now() - this.startTime + console.log(`${operation} took ${duration}ms`) + this.startTime = undefined + } + } +} +``` + +--- + +## Operation Hooks + +### Core Operations You Can Hook +```typescript +readonly operations = [ + 'add', // Adding data + 'update', // Updating data + 'delete', // Deleting data + 'get', // Retrieving data + 'search', // Searching + 'find', // Triple Intelligence queries + 'relate', // Adding relationships + 'unrelate', // Removing relationships + 'clear', // Clearing data + 'all' // Hook ALL operations +] as const +``` + +### Example: Multi-Operation Hook +```typescript +class AuditAugmentation extends BaseAugmentation { + readonly operations = ['add', 'update', 'delete'] as const + + async execute(operation: string, params: any): Promise { + // Log all data modifications + await this.logToAuditTrail(operation, params) + } +} +``` + +--- + +## Accessing Brain Context + +```typescript +class ContextAwareAugmentation extends BaseAugmentation { + async execute( + operation: string, + params: any, + context?: AugmentationContext + ): Promise { + // Access the brain instance + const brain = context?.brain + if (!brain) return + + // Use any brain method + const stats = await brain.getStats() + const size = await brain.size() + const results = await brain.search('query') + + // Access other augmentations + const cache = brain.augmentations.get('cache') + if (cache) { + await cache.clear() + } + } +} +``` + +--- + +## Real-World Examples + +### 1. Backup Augmentation +```typescript +class BackupAugmentation extends BaseAugmentation { + readonly name = 'backup' + readonly timing = 'after' as const + readonly operations = ['add', 'update', 'delete'] as const + readonly priority = 5 + + private changes = 0 + private readonly backupThreshold = 100 + + async execute(operation: string, params: any, context?: AugmentationContext): Promise { + this.changes++ + + if (this.changes >= this.backupThreshold) { + await this.performBackup(context?.brain) + this.changes = 0 + } + } + + private async performBackup(brain?: any): Promise { + if (!brain) return + const backup = await brain.backup() + await this.saveToCloud(backup) + console.log('Automatic backup completed') + } +} +``` + +### 2. Rate Limiting Augmentation +```typescript +class RateLimitAugmentation extends BaseAugmentation { + readonly name = 'rate-limit' + readonly timing = 'before' as const + readonly operations = ['search', 'find'] as const + readonly priority = 100 // High priority - run first + + private requests = new Map() + private readonly limit = 100 // 100 requests + private readonly window = 60000 // per minute + + async execute(operation: string, params: any): Promise { + const now = Date.now() + const key = params.userId || 'anonymous' + + // Get request timestamps + const timestamps = this.requests.get(key) || [] + + // Remove old timestamps + const recent = timestamps.filter(t => now - t < this.window) + + // Check limit + if (recent.length >= this.limit) { + throw new Error('Rate limit exceeded') + } + + // Add current request + recent.push(now) + this.requests.set(key, recent) + } +} +``` + +### 3. Encryption Augmentation +```typescript +class EncryptionAugmentation extends BaseAugmentation { + readonly name = 'encryption' + readonly timing = 'both' as const + readonly operations = ['add', 'get'] as const + readonly priority = 90 // Run early + + async execute(operation: string, params: any): Promise { + if (operation === 'add') { + // Encrypt before storing + if (params.metadata?.sensitive) { + params.content = await this.encrypt(params.content) + params.encrypted = true + } + return params + } + + if (operation === 'get' && params.result?.encrypted) { + // Decrypt after retrieval + params.result.content = await this.decrypt(params.result.content) + delete params.result.encrypted + return params.result + } + } +} +``` + +--- + +## Testing Your Augmentation + +```typescript +import { describe, it, expect } from 'vitest' +import { Brainy } from '@soulcraft/brainy' +import { MyAugmentation } from './my-augmentation' + +describe('MyAugmentation', () => { + it('should hook into addNoun', async () => { + const brain = new Brainy({ storage: 'memory' }) + const aug = new MyAugmentation() + + // Spy on the execute method + const executeSpy = vi.spyOn(aug, 'execute') + + brain.augmentations.register(aug) + await brain.init() + + // Trigger the augmentation + await brain.add('test data') + + // Verify it was called + expect(executeSpy).toHaveBeenCalledWith( + 'add', + expect.objectContaining({ content: 'test data' }), + expect.any(Object) + ) + }) +}) +``` + +--- + +## Best Practices + +### 1. Use Proper Timing +- `before`: Validation, modification, rate limiting +- `after`: Logging, metrics, side effects +- `both`: Timing, tracing, wrapping + +### 2. Set Appropriate Priority +```typescript +// Priority guidelines +100: Critical (auth, rate limiting) +50: Important (validation, transformation) +10: Normal (logging, metrics) +1: Optional (debugging, tracing) +``` + +### 3. Handle Errors Gracefully +```typescript +async execute(operation: string, params: any): Promise { + try { + await this.riskyOperation() + } catch (error) { + // Log but don't break the main operation + console.error(`Augmentation error in ${this.name}:`, error) + // Optionally report to monitoring + this.reportError(error) + } +} +``` + +### 4. Be Performance Conscious +```typescript +class CachedAugmentation extends BaseAugmentation { + private cache = new Map() + + async execute(operation: string, params: any): Promise { + const key = this.getCacheKey(params) + + // Check cache first + if (this.cache.has(key)) { + return this.cache.get(key) + } + + // Expensive operation + const result = await this.expensiveOperation(params) + this.cache.set(key, result) + + return result + } +} +``` + +### 5. Clean Up Resources +```typescript +protected async onShutdown(): Promise { + // Close connections + await this.connection?.close() + + // Clear intervals + clearInterval(this.interval) + + // Flush buffers + await this.flush() + + // Clear caches + this.cache.clear() +} +``` + +--- + +## Publishing Your Augmentation (Future) + +### Package Structure +``` +my-augmentation/ +├── src/ +│ └── index.ts # Your augmentation +├── dist/ # Built output +├── tests/ +│ └── augmentation.test.ts +├── package.json +├── tsconfig.json +└── README.md +``` + +### package.json +```json +{ + "name": "@mycompany/brainy-custom-augmentation", + "version": "1.0.0", + "main": "dist/index.js", + "types": "dist/index.d.ts", + "keywords": ["brainy-augmentation"], + "peerDependencies": { + "@soulcraft/brainy": ">=2.0.0" + }, + "brainy": { + "type": "augmentation", + "class": "CustomAugmentation", + "timing": "after", + "operations": ["add"], + "priority": 10 + } +} +``` + +### Future: Brain Cloud Registry +```bash +# Coming in 2.1+ +npm run build +npm test +brainy publish # Publishes to brain-cloud registry +``` + +--- + +## FAQ + +### Q: Can I modify the operation result? +**A**: Yes, if `timing: 'before'`, return modified params. If `timing: 'after'`, you can see but not modify results. + +### Q: Can augmentations communicate? +**A**: Yes, through the context: `context.brain.augmentations.get('other-augmentation')` + +### Q: What if my augmentation fails? +**A**: Handle errors internally. Don't break the main operation unless critical. + +### Q: Can I use async operations? +**A**: Yes, everything is async-friendly. + +### Q: How do I access storage directly? +**A**: Through context: `context.brain.storage` (but prefer using brain methods) + +--- + +## Get Help + +- **GitHub**: [github.com/soulcraft/brainy](https://github.com/soulcraft/brainy) +- **Discord**: [discord.gg/brainy](https://discord.gg/brainy) +- **Examples**: See `/examples/augmentations/` in the repo + +--- + +*Start building your augmentation today! The marketplace is coming in 2.1 🚀* \ No newline at end of file diff --git a/docs/augmentations/README.md b/docs/augmentations/README.md new file mode 100644 index 00000000..05e9d41e --- /dev/null +++ b/docs/augmentations/README.md @@ -0,0 +1,204 @@ +# Brainy Augmentations + +Augmentations are the core extensibility mechanism in Brainy. They allow you to modify, enhance, and extend Brainy's behavior without changing the core code. + +## Core Principle: One Interface, Infinite Possibilities + +Every augmentation implements the same simple `BrainyAugmentation` interface: + +```typescript +interface BrainyAugmentation { + name: string + timing: 'before' | 'after' | 'around' | 'replace' + operations: string[] + priority: number + initialize(context): Promise + execute(operation, params, next): Promise +} +``` + +This single interface can handle EVERYTHING - from adding AI capabilities to exposing APIs to replacing storage backends. + +## Available Augmentations + +### 🧠 Data Processing +Augmentations that enhance how data is processed and stored. + +| Augmentation | Description | Timing | Status | +|-------------|-------------|--------|--------| +| **NeuralImportAugmentation** | AI-powered entity and relationship extraction | `before` | ✅ Production | +| **EntityRegistryAugmentation** | High-performance entity deduplication | `before` | ✅ Production | +| **BatchProcessingAugmentation** | Optimizes bulk operations | `around` | ✅ Production | +| **IntelligentVerbScoringAugmentation** | Learns relationship importance over time | `after` | ✅ Production | + +### 🔌 External Connections (Synapses) +Connect Brainy to external services and data sources. + +| Augmentation | Description | Timing | Status | +|-------------|-------------|--------|--------| +| **NotionSynapse** | Sync with Notion databases | `after` | 📝 Example | +| **SalesforceSynapse** | Connect to Salesforce CRM | `after` | 📝 Example | +| **SlackSynapse** | Import Slack conversations | `after` | 📝 Example | +| **GoogleDriveSynapse** | Sync Google Drive documents | `after` | 📝 Example | + +### 🌐 API Exposure +Expose Brainy through various protocols. + +| Augmentation | Description | Timing | Status | +|-------------|-------------|--------|--------| +| **APIServerAugmentation** | REST, WebSocket, and MCP server | `after` | ✅ Production | +| **GraphQLAugmentation** | GraphQL API endpoint | `after` | 🚧 Planned | +| **gRPCAugmentation** | gRPC service | `after` | 🚧 Planned | + +### 💾 Storage Backends +Replace or enhance the storage layer. + +| Augmentation | Description | Timing | Status | +|-------------|-------------|--------|--------| +| **S3StorageAugmentation** | Use S3 as storage backend | `replace` | 📝 Example | +| **RedisAugmentation** | Redis caching layer | `around` | 📝 Example | +| **PostgresAugmentation** | PostgreSQL persistence | `replace` | 📝 Example | + +### 🔄 Real-time & Sync +Handle real-time updates and synchronization. + +| Augmentation | Description | Timing | Status | +|-------------|-------------|--------|--------| +| **WebSocketConduitAugmentation** | WebSocket client connections | `after` | ⚠️ Legacy | +| **ServerSearchAugmentation** | Connect to remote Brainy servers | `after` | ⚠️ Legacy | +| **TeamCoordinationAugmentation** | Multi-agent synchronization | `after` | 📝 Example | + +### 🛡️ Infrastructure +Core infrastructure and reliability features. + +| Augmentation | Description | Timing | Status | +|-------------|-------------|--------|--------| +| **ConnectionPoolAugmentation** | Optimize cloud storage connections | `before` | ✅ Production | +| **RequestDeduplicatorAugmentation** | Prevent duplicate concurrent requests | `before` | ✅ Production | +| **TransactionAugmentation** | ACID transaction support | `around` | 🚧 Planned | +| **CacheAugmentation** | Multi-level caching | `around` | ✅ Production | + +### 📊 Monitoring & Analytics +Track and analyze Brainy's behavior. + +| Augmentation | Description | Timing | Status | +|-------------|-------------|--------|--------| +| **MetricsAugmentation** | Prometheus metrics | `after` | 📝 Example | +| **LoggingAugmentation** | Structured logging | `after` | 📝 Example | +| **TracingAugmentation** | Distributed tracing | `around` | 🚧 Planned | + +### 🤖 AI & Chat +AI-powered interfaces and chat capabilities. + +| Augmentation | Description | Timing | Status | +|-------------|-------------|--------|--------| +| **ChatInterfaceAugmentation** | Natural language interface | `before` | 📝 Example | +| **MCPAgentMemoryAugmentation** | AI agent memory via MCP | `after` | 📝 Example | +| **LLMQueryAugmentation** | LLM-enhanced queries | `before` | 📝 Example | + +### 📈 Visualization +Visual representations of data. + +| Augmentation | Description | Timing | Status | +|-------------|-------------|--------|--------| +| **GraphVisualizationAugmentation** | Real-time graph visualization | `after` | 📝 Example | +| **DashboardAugmentation** | Web-based dashboard | `after` | 🚧 Planned | + +## Status Legend + +- ✅ **Production**: Fully implemented and tested +- 📝 **Example**: Example implementation available +- 🚧 **Planned**: On the roadmap +- ⚠️ **Legacy**: Being replaced by newer augmentations + +## Using Augmentations + +### Zero-Config Approach + +```typescript +const brain = new Brainy() + +// Just register augmentations - they work automatically! +brain.augmentations.register(new EntityRegistryAugmentation()) +brain.augmentations.register(new APIServerAugmentation()) + +await brain.init() +``` + +### With Configuration + +```typescript +const brain = new Brainy() + +brain.augmentations.register( + new APIServerAugmentation({ + port: 8080, + auth: { required: true } + }) +) + +await brain.init() +``` + +## Creating Custom Augmentations + +See [Creating Custom Augmentations](./creating-augmentations.md) for a complete guide. + +Quick example: + +```typescript +class MyAugmentation extends BaseAugmentation { + readonly name = 'my-augmentation' + readonly timing = 'after' + readonly operations = ['add', 'search'] + readonly priority = 50 + + async execute(operation: string, params: any, next: () => Promise): Promise { + console.log(`Before ${operation}`) + const result = await next() + console.log(`After ${operation}`) + return result + } +} +``` + +## Augmentation Timing + +### `before` +Executes before the main operation. Used for: +- Input validation +- Data transformation +- Authentication checks + +### `after` +Executes after the main operation. Used for: +- Broadcasting updates +- Syncing to external services +- Logging and metrics + +### `around` +Wraps the main operation. Used for: +- Transactions +- Caching +- Error handling + +### `replace` +Completely replaces the main operation. Used for: +- Alternative storage backends +- Mock implementations +- Proxy operations + +## Priority System + +Higher numbers execute first: +- **100**: Critical system operations +- **50**: Performance optimizations +- **10**: Enhancement features +- **1**: Optional features + +## Related Documentation + +- [API Server Augmentation](./api-server.md) - Complete API server documentation +- [Creating Augmentations](./creating-augmentations.md) - How to build your own +- [Augmentation Examples](../AUGMENTATION-EXAMPLES.md) - Real-world examples +- [Architecture Overview](../COMPLETE-ARCHITECTURE-VISION.md) - System architecture \ No newline at end of file diff --git a/docs/augmentations/api-server.md b/docs/augmentations/api-server.md new file mode 100644 index 00000000..9546a41e --- /dev/null +++ b/docs/augmentations/api-server.md @@ -0,0 +1,403 @@ +# API Server Augmentation + +## Overview + +The `APIServerAugmentation` is a powerful augmentation that exposes your Brainy instance through REST, WebSocket, and MCP (Model Context Protocol) APIs. It transforms Brainy into a full-featured API server with zero configuration required. + +## Features + +### 🌐 REST API +Complete CRUD operations and advanced queries through HTTP endpoints. + +### 🔌 WebSocket Server +Real-time bidirectional communication with automatic operation broadcasting. + +### 🧠 MCP Integration +Built-in Model Context Protocol support for AI agent communication. + +### 📊 Operation Broadcasting +Automatically broadcasts all Brainy operations to subscribed WebSocket clients. + +### 🔒 Optional Security +Built-in authentication and rate limiting when needed. + +## Installation + +The APIServerAugmentation is included in Brainy core. No additional installation required. + +For Node.js environments, you may want to install optional dependencies: +```bash +npm install express cors ws +``` + +## Zero-Config Usage + +```typescript +import { Brainy } from 'brainy' +import { APIServerAugmentation } from 'brainy/augmentations' + +const brain = new Brainy() + +// Register the API server augmentation +brain.augmentations.register(new APIServerAugmentation()) + +await brain.init() + +// Server is now running at http://localhost:3000 +console.log('API Server ready!') +console.log('REST: http://localhost:3000/api/*') +console.log('WebSocket: ws://localhost:3000/ws') +console.log('MCP: http://localhost:3000/api/mcp') +``` + +## Configuration Options + +While zero-config works great, you can customize the server: + +```typescript +const apiServer = new APIServerAugmentation({ + enabled: true, // Enable/disable the server + port: 3000, // HTTP port + host: '0.0.0.0', // Bind address + + cors: { + origin: '*', // CORS allowed origins + credentials: true // Allow credentials + }, + + auth: { + required: false, // Require authentication + apiKeys: [], // Valid API keys + bearerTokens: [] // Valid bearer tokens + }, + + rateLimit: { + windowMs: 60000, // Rate limit window (ms) + max: 100 // Max requests per window + } +}) +``` + +## REST API Endpoints + +### Health Check +```http +GET /health +``` +Returns server status and basic metrics. + +### Search +```http +POST /api/search +Content-Type: application/json + +{ + "query": "search text", + "limit": 10, + "options": {} +} +``` + +### Add Data +```http +POST /api/add +Content-Type: application/json + +{ + "content": "data to add", + "metadata": { + "key": "value" + } +} +``` + +### Get by ID +```http +GET /api/get/:id +``` + +### Delete +```http +DELETE /api/delete/:id +``` + +### Create Relationship +```http +POST /api/relate +Content-Type: application/json + +{ + "source": "id1", + "target": "id2", + "verb": "relates_to", + "metadata": {} +} +``` + +### Complex Queries +```http +POST /api/find +Content-Type: application/json + +{ + "where": { "type": "document" }, + "like": "machine learning", + "limit": 10 +} +``` + +### Clustering +```http +POST /api/cluster +Content-Type: application/json + +{ + "algorithm": "kmeans", + "options": { + "k": 5 + } +} +``` + +### Statistics +```http +GET /api/stats +``` + +### Operation History +```http +GET /api/history +``` + +## WebSocket API + +### Connection +```javascript +const ws = new WebSocket('ws://localhost:3000/ws') + +ws.onopen = () => { + console.log('Connected to Brainy WebSocket') +} + +ws.onmessage = (event) => { + const msg = JSON.parse(event.data) + console.log('Received:', msg) +} +``` + +### Subscribe to Operations +```javascript +ws.send(JSON.stringify({ + type: 'subscribe', + operations: ['all'] // or specific: ['add', 'search', 'delete'] +})) +``` + +### Search via WebSocket +```javascript +ws.send(JSON.stringify({ + type: 'search', + query: 'your search', + limit: 10, + requestId: 'unique-id' +})) +``` + +### Add Data via WebSocket +```javascript +ws.send(JSON.stringify({ + type: 'add', + content: 'data to add', + metadata: {}, + requestId: 'unique-id' +})) +``` + +### Operation Broadcasts +When subscribed, you'll receive real-time updates: +```javascript +{ + "type": "operation", + "operation": "add", + "params": { /* sanitized parameters */ }, + "timestamp": 1234567890, + "duration": 15 +} +``` + +## MCP (Model Context Protocol) + +The MCP endpoint allows AI agents to interact with Brainy: + +```http +POST /api/mcp +Content-Type: application/json + +{ + "method": "search", + "params": { + "query": "find documents about AI" + } +} +``` + +## Authentication + +When authentication is enabled: + +### API Key +```http +GET /api/stats +X-API-Key: your-api-key +``` + +### Bearer Token +```http +GET /api/stats +Authorization: Bearer your-token +``` + +## Environment Support + +### Node.js ✅ +Full support with Express, WebSocket, and all features. + +### Deno 🚧 +Planned support using Deno.serve() or oak framework. + +### Browser/Service Worker 🚧 +Planned support for intercepting fetch() calls locally. + +## How It Works + +The APIServerAugmentation hooks into Brainy's augmentation pipeline: + +1. **Timing**: Executes `after` operations complete +2. **Operations**: Monitors `all` operations +3. **Broadcasting**: Sends operation details to subscribed clients +4. **History**: Maintains operation history (last 1000 operations) + +## Example: Multi-Client Sync + +```typescript +// Server +const brain = new Brainy() +brain.augmentations.register(new APIServerAugmentation()) +await brain.init() + +// Client 1 - WebSocket subscriber +const ws1 = new WebSocket('ws://localhost:3000/ws') +ws1.onopen = () => { + ws1.send(JSON.stringify({ + type: 'subscribe', + operations: ['add', 'delete'] + })) +} +ws1.onmessage = (e) => { + console.log('Client 1 received update:', JSON.parse(e.data)) +} + +// Client 2 - REST API user +fetch('http://localhost:3000/api/add', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + content: 'New data', + metadata: { source: 'client2' } + }) +}) +// Client 1 automatically receives notification! +``` + +## Performance Considerations + +- **Operation History**: Limited to last 1000 operations +- **WebSocket Heartbeat**: Every 30 seconds +- **Client Timeout**: 60 seconds of inactivity +- **Parameter Sanitization**: Sensitive fields removed, large content truncated +- **Rate Limiting**: In-memory tracking (use Redis in production) + +## Security Notes + +1. **Default Configuration**: No auth, open CORS - suitable for development +2. **Production**: Enable auth, configure CORS, use HTTPS +3. **Sensitive Data**: Parameters are sanitized before broadcasting +4. **Rate Limiting**: Basic in-memory implementation included + +## Comparison with Previous Implementations + +The APIServerAugmentation unifies and replaces: +- `BrainyMCPBroadcast` - Node-specific WebSocket/HTTP server +- `WebSocketConduitAugmentation` - WebSocket client functionality +- `ServerSearchAugmentations` - Remote Brainy connections + +Benefits of the unified approach: +- Single augmentation for all API needs +- Consistent interface across protocols +- Automatic operation broadcasting +- Environment-aware implementation +- Zero-configuration philosophy + +## Advanced Usage + +### Custom Operation Filtering + +```typescript +class FilteredAPIServer extends APIServerAugmentation { + shouldExecute(operation: string, params: any): boolean { + // Don't broadcast sensitive operations + if (operation === 'delete' && params.sensitive) { + return false + } + return true + } +} +``` + +### Integration with Other Augmentations + +```typescript +const brain = new Brainy() + +// Stack augmentations for complete system +brain.augmentations.register(new EntityRegistryAugmentation()) // Dedup +brain.augmentations.register(new APIServerAugmentation()) // API + +await brain.init() +// All augmentations work together seamlessly! +``` + +## Troubleshooting + +### Server won't start +- Check if port is already in use +- Verify Node.js dependencies are installed: `npm install express cors ws` +- Check console for error messages + +### WebSocket connections drop +- Ensure heartbeat responses are handled +- Check for proxy/firewall issues +- Verify CORS configuration + +### Authentication not working +- Ensure `auth.required` is set to `true` +- Verify API keys or bearer tokens are correctly configured +- Check request headers are properly formatted + +## Future Enhancements + +- [ ] Deno server implementation +- [ ] Service Worker implementation +- [ ] GraphQL endpoint +- [ ] gRPC support +- [ ] Built-in SSL/TLS +- [ ] Redis-based rate limiting +- [ ] Prometheus metrics endpoint +- [ ] OpenAPI/Swagger documentation + +## Related Documentation + +- [Augmentation System Overview](../AUGMENTATION-SYSTEM.md) +- [BrainyAugmentation Interface](./brainy-augmentation.md) +- [MCP Integration](../mcp/README.md) +- [Zero-Config Philosophy](../ZERO-CONFIG.md) \ No newline at end of file diff --git a/docs/concepts/consistency-model.md b/docs/concepts/consistency-model.md deleted file mode 100644 index bad996c5..00000000 --- a/docs/concepts/consistency-model.md +++ /dev/null @@ -1,386 +0,0 @@ ---- -title: Consistency Model -slug: concepts/consistency-model -public: true -category: concepts -template: concept -order: 4 -description: The exact guarantees behind Brainy's Db API — snapshot isolation, atomic transactions, two levels of compare-and-swap, time travel, retention, snapshots, crash recovery, and the reserved-field contract. -next: - - guides/snapshots-and-time-travel - - guides/optimistic-concurrency ---- - -# Consistency Model - -Brainy 8.0's consistency story rests on one mechanism: **generational MVCC** -— multi-version concurrency control over immutable, generation-stamped -records. It is exposed through a single value type, the **`Db`**: an -immutable, point-in-time view of the whole store that you query like the -live brain. - -```typescript -const db = brain.now() // pin the current state — O(1), no I/O - -await brain.transact([ - { op: 'update', id: invoiceId, metadata: { status: 'paid' } } -]) - -await db.get(invoiceId) // still 'pending' — pinned, forever -await brain.get(invoiceId) // 'paid' — live -await db.release() // unpin when done -``` - -This page states the guarantees precisely — what is promised, what it costs, -and where the honest limits are. The design record is -[ADR-001](../ADR-001-generational-mvcc.md); every guarantee below is proven -by a dedicated test in `tests/integration/db-mvcc.test.ts`. - -## The generation clock - -A **monotonic generation counter** is the store's logical clock: - -- It advances **once per committed `transact()` batch** and once per - single-operation write (`add`/`update`/`remove`/`relate`/…). -- `brain.generation()` reads it; it is persisted in the data directory and - **never reissued** — not across restarts, and not across `restore()` - (the counter is floored at its pre-restore value). - -Every `Db` is pinned at one generation. `db.generation` and `db.timestamp` -identify the view; `newerDb.since(olderDb)` returns exactly the entity and -relationship ids that committed transactions touched between two views. - -## Snapshot isolation for reads - -**Guarantee:** a `Db` reads exactly the state at its pinned generation, no -matter what commits afterwards — including deletes. There are no torn reads, -no partially applied batches, and no drift over time. - -- `brain.now()` pins the current generation in O(1). -- `brain.transact()` returns a `Db` pinned at the freshly committed - generation. -- `brain.asOf(generation | Date | snapshotPath)` pins past state. - -While nothing has committed past the pin, reads delegate to the live fast -paths — pinning is free until history actually moves. Once later -transactions commit, the view keeps serving the **full query surface** at -its generation (see "Reading the past" below). - -Writers are never blocked by readers and readers never block writers: a -pinned view stays valid because nothing overwrites the immutable records it -resolves from (the LMDB reader-pin model). - -## Transaction atomicity - -`brain.transact(ops)` executes a declarative batch — `add`, `update`, -`remove`, `relate`, `unrelate` — **atomically as exactly one generation**: - -```typescript -const db = await brain.transact([ - { op: 'add', id: orderId, type: NounType.Document, subtype: 'order', data: 'Order #1042' }, - { op: 'add', id: itemId, type: NounType.Thing, subtype: 'line-item', data: 'Widget x3' }, - { op: 'relate', from: orderId, to: itemId, type: VerbType.Contains, subtype: 'order-line' } -], { meta: { author: 'order-service', requestId: 'req-9f2' } }) - -db.receipt.ids // resolved id per operation, in input order -``` - -Either every operation applies, or none do and the store is byte-identical -to its pre-transaction state. Operation semantics mirror the corresponding -single-operation methods — validation, subtype enforcement, relationship -deduplication, delete cascades — and later operations may reference ids -created earlier in the same batch. - -**The commit point is one atomic rename.** The durability protocol: - -1. Before-images of every touched id are staged into an immutable - generation directory and **fsynced**. -2. The batch executes through the transaction manager (which has its own - operation-level rollback for non-crash failures). -3. The store manifest is replaced via atomic temp-file rename and fsynced. - **The rename is the commit** — a generation is committed if and only if - the manifest says so. - -**Crash recovery:** on the next open, any staged generation above the -manifest watermark is an uncommitted transaction; its before-images are -restored (idempotently — recovery can itself crash and rerun) and derived -indexes never observe the rolled-back state. A crash anywhere before the -rename rolls back to the exact pre-transaction bytes; a crash after it keeps -the transaction. - -Transaction metadata (`meta`) is reified Datomic-style: recorded in an -append-only transaction log readable via `brain.transactionLog()` — audit -fields live in the database, not in commit messages. - -## Two levels of compare-and-swap - -Concurrent `transact()` calls commit serially (snapshot-isolated batches). -For lost-update protection across a read–modify–write cycle, Brainy offers -CAS at two granularities: - -| Granularity | Mechanism | Conflict error | Use when | -|---|---|---|---| -| **Per entity** | `_rev` + `{ op: 'update', ifRev }` (also on `brain.update()`) | `RevisionConflictError` | "This entity must not have changed since I read it." | -| **Whole store** | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` | "*Nothing* may have committed since I read." | - -```typescript -const view = brain.now() -const order = await view.get(orderId) - -try { - await brain.transact( - [{ op: 'update', id: orderId, metadata: { total: recompute(order) }, ifRev: order._rev }], - { ifAtGeneration: view.generation } - ) -} catch (err) { - if (err instanceof GenerationConflictError) { - // Something committed since the pin — re-read and retry. - } -} finally { - await view.release() -} -``` - -An `ifRev` conflict on any operation rejects the **whole batch**; an -`ifAtGeneration` conflict is detected before anything is staged. Both leave -the store untouched and the generation counter unchanged. See -[Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md) -for the per-entity pattern in depth. - -## Reading the past - -`brain.asOf()` accepts a generation number, a `Date` (resolved through the -transaction log to the newest generation committed at or before it), or a -snapshot directory path. Historical views serve the **full query surface** -— `get()`, `find()` in every mode, semantic search, graph traversal, -cursors, aggregation — through two complementary paths: - -- **Record path** (free): `get()`, metadata-level `find()`, and - filter-based `related()` resolve directly through the immutable record - layer. Ids untouched since the pin still ride the live fast paths. -- **Index path** (paid once): index-accelerated queries — semantic/vector - search, graph traversal, cursors, aggregation — are served by an - **at-generation index materialization** built lazily on first use: - Brainy reconstructs in-memory indexes over the exact record set at that - generation. This costs O(n at the pinned generation) time and memory, - **once per `Db`**, cached until `release()`. That is the open-core price - of historical index queries, stated plainly. - -A native index provider implementing the optional -`VersionedIndexProvider` plugin capability serves the same historical reads -from its retained index segments **without any rebuild** — the materializer -is the correctness baseline, the provider is the accelerator. Semantics are -identical on both paths. - -### History granularity — the honest limit - -Generation *records* are written per `transact()` batch only. -Single-operation writes (`add`/`update`/`remove`/`relate`/… outside -`transact()`) advance the generation counter — so watermarks and CAS stay -sound — but do **not** stage before-images: they remain visible through -earlier pins and are not reported by `db.since()`. Code that needs pinned -isolation across its own writes uses `transact()`. This is the documented -8.0 contract, not an accident. - -## Speculative writes: `db.with()` - -`db.with(ops)` returns a new `Db` whose reads see the operations applied -**in memory, on top of the view** — Datomic's `with`. Nothing touches disk, -the generation counter, or index providers: - -```typescript -const current = brain.now() -const whatIf = await current.with([ - { op: 'update', id: employeeId, metadata: { team: 'platform' } } -]) - -await whatIf.find({ where: { team: 'platform' } }) // sees the change -await brain.get(employeeId) // unchanged — nothing committed -``` - -**The one boundary:** overlay entities carry no embeddings (`with()` never -invokes the embedder), so index-accelerated queries and `persist()` on a -speculative view throw `SpeculativeOverlayError` rather than returning -silently incomplete results. `get()`, metadata-filter `find()`, and -filter-based `related()` work fully on overlays. To get the full surface, -commit the same operations with `brain.transact()`. - -## Retention and compaction - -Historical records cost disk space, so retention is explicit: - -- Every live `Db` holds a refcounted **pin**; a record-set is never - reclaimed while any pin could need it — pinned reads stay correct across - compaction, always. -- The **`retention`** knob governs auto-compaction (on every `flush()`/ - `close()`): unset → ADAPTIVE (disk/RAM-pressure, zero-config) · `'all'` → - unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit caps. - `brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims - manually on the same caps, and records the **horizon** — `asOf()` below it - throws `GenerationCompactedError`, explicitly, never partial data. -- To keep a state readable forever, `persist()` it first: snapshots are - self-contained and unaffected by compaction of the source store. - -Release `Db` values you do not keep (including the ones `transact()` -returns). A `FinalizationRegistry` backstop releases leaked pins at garbage -collection, but explicit `release()` is what makes compaction -deterministic. - -## Durability: snapshots and restore - -`db.persist(path)` cuts a **self-contained snapshot** under the store's -commit mutex, so no commit or compaction can interleave. On filesystem -storage it is built from **hard links**: because every data file is -immutable-by-rename, linking is safe — the snapshot is created without -copying entity data, shares disk space with the source, and later writes to -the source can never alter it (rewrites swap inodes; the snapshot keeps the -old bytes). Cross-device targets fall back to byte copies; in-memory stores -serialize to the same directory layout, producing a real, durable store. - -Two rules keep snapshots honest: - -- `persist()` requires the view to still be the store's **latest** - generation (a snapshot captures current bytes); a view that history has - moved past throws `GenerationConflictError` instead of persisting the - wrong state. -- `brain.restore(path, { confirm: true })` replaces the store's entire - state from a snapshot via byte copy (never links — the snapshot stays - independent), rebuilds all indexes, and floors the generation counter so - observed generation numbers are never reissued. Live pins do not survive - a restore — release them first (a warning is logged when any exist). - -`Brainy.load(path)` (or `brain.asOf(path)`) opens a snapshot as a -self-contained **read-only** store with the full query surface, including -vector search. - -## Reserved fields - -Some field names belong to Brainy, not to your metadata. They live at **top -level** on every entity and relationship, have dedicated write paths, and may -never appear inside a `metadata` bag: - -| Entities (nouns) | Relationships (verbs) | Canonical write path | -|---|---|---| -| `noun` | `verb` | the `type` param of `add()` / `relate()` | -| `subtype` | `subtype` | the `subtype` param | -| `visibility` | `visibility` | the `visibility` param (`'public'` \| `'internal'`) | -| `confidence` | `confidence` | the `confidence` param | -| `weight` | `weight` | the `weight` param | -| `service` | `service` | the `service` param (fixed at create time) | -| `data` | `data` | the `data` param | -| `createdBy` | `createdBy` | the `createdBy` param of `add()` (system-managed on verbs) | -| `createdAt`, `updatedAt`, `_rev` | `createdAt`, `updatedAt`, `_rev` | system-managed (`ifRev` for CAS) | - -The canonical machine-readable lists are exported as -`RESERVED_ENTITY_FIELDS` and `RESERVED_RELATION_FIELDS` (defined in -`src/types/reservedFields.ts`, the single source of truth). Three layers -enforce the contract: - -1. **Compile time** — every `metadata` param (`add`, `update`, `relate`, - `updateRelation`, and the matching `transact()` operations) rejects a - literal reserved key as a TypeScript error. -2. **Write time** — untyped (JavaScript) callers that pass one anyway are - normalized: user-settable fields (`confidence`, `weight`, `subtype`, and - `service`/`createdBy` at create time) are remapped to their dedicated - param — **top-level wins** when both are supplied — and system-managed - fields are dropped with a one-shot warning naming the correct write path. - `update({ metadata: { confidence: 0.9 } })` therefore behaves exactly - like `update({ confidence: 0.9 })`. -3. **Read time** — every read path (`get`, `find`, `search`, - `related`, batch reads, and historical `asOf()` materialization) - surfaces reserved fields **only at top level**: `entity.metadata` and - `relation.metadata` contain only your custom fields, always. - -```typescript -const id = await brain.add({ - type: 'document', subtype: 'invoice', - data: 'Invoice #42', confidence: 0.95, // reserved → top-level params - metadata: { customer: 'acme', total: 129.5 } // custom fields only -}) - -const entity = await brain.get(id) -entity.confidence // 0.95 — top level -entity.metadata // { customer: 'acme', total: 129.5 } -``` - -### Visibility — `public` / `internal` / `system` - -`visibility` is a reserved tier that controls whether an entity or relationship -surfaces on Brainy's **default** user-facing reads. The absence of the field is -exactly equivalent to `'public'`. - -| Tier | Counted in `getNounCount()` / `stats()`? | Returned by default `find()` / `related()`? | Opt-in | -|---|---|---|---| -| `'public'` (default, or field absent) | yes | yes | — | -| `'internal'` | no | no | `find({ includeInternal: true })` / `related({ includeInternal: true })` | -| `'system'` | no | no | `find({ includeSystem: true })` / `related({ includeSystem: true })` | - -- **`'public'`** — normal data. Counted and returned everywhere. Stored lean: - the field is omitted on disk for public records, so existing data needs no - migration. -- **`'internal'`** — your app's own bookkeeping (audit trails, derived caches, - scratch entities) that should not pollute default queries, counts, or - `stats()`, yet must stay retrievable on demand. Set it via the `visibility` - param; read it back with the `includeInternal` opt-in. -- **`'system'`** — Brainy's own plumbing (for example the Virtual File System - root entity). Hidden everywhere by default — even when `includeInternal` is - set — and surfaced only with the explicit `includeSystem` opt-in. The - `'system'` tier is **not** part of the public `add()` / `relate()` param type - (`'public' | 'internal'`); only internal Brainy code assigns it. - -The opt-ins are applied as a **hard candidate filter** — hidden entities are -removed before `limit` / `offset` are applied, so a default `find({ limit: 10 })` -always returns ten *visible* results when that many exist, never a short page. - -```typescript -// App-internal scratch entity: present, retrievable, but out of the way. -await brain.add({ type: 'task', data: 'reindex job', visibility: 'internal' }) - -await brain.getNounCount() // unchanged — internal not counted -await brain.find({ type: 'task' }) // [] — hidden by default -await brain.find({ type: 'task', includeInternal: true }) // includes it - -// A brand-new brain reports zero user entities even though the VFS root exists: -const fresh = new Brainy() -await fresh.init() -await fresh.getNounCount() // 0 — the root is visibility:'system' -``` - -> **Note (8.0):** the structural `Contains` edges the VFS creates between -> directories and files are left at the default (`public`) visibility for now — -> only the VFS *root entity* is `'system'`. Marking those edges system requires -> companion changes to VFS traversal and is out of scope for this change. - -## What is not guaranteed - -Stated plainly, so nothing surprises you in production: - -- **Single-writer.** Brainy is a single-writer, many-reader database - ([multi-process model](./multi-process.md)). Transactions are atomic - within one writer process — there is no distributed or cross-process - transaction coordination. -- **History granularity.** Every write is its own immutable generation — - `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`. - A pin always freezes against later writes, and every write is addressable - via `asOf()`. (`transact()` groups several operations into ONE atomic - generation; durability of single-op history is batched via async - group-commit — a hard crash can lose only the last un-flushed window's - *history*, never live data.) -- **Compacted history is gone.** `asOf()` below the compaction horizon - fails explicitly; persist what you must keep. -- **Counter persistence is coalesced for single-operation writes.** Durable - artifacts (records, manifests, snapshots) always persist the counter at - their own commit points, so a crash inside the coalescing window can lose - only counter values nothing durable ever referenced. -- **Speculative overlays are metadata-only readers.** Index-accelerated - queries on `with()` views throw rather than guess. - -## Where to go next - -- [Snapshots & Time Travel](../guides/snapshots-and-time-travel.md) — the - recipes: backup, restore, time-travel debugging, what-if analysis, audit - trails. -- [Optimistic concurrency with `_rev`](../guides/optimistic-concurrency.md) - — the per-entity CAS pattern. -- [ADR-001: Generational MVCC](../ADR-001-generational-mvcc.md) — the full - design record, including the persisted layout and the proof table. diff --git a/docs/concepts/generation-fact-log.md b/docs/concepts/generation-fact-log.md deleted file mode 100644 index f9b3e974..00000000 --- a/docs/concepts/generation-fact-log.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -title: The Generation Fact Log -slug: concepts/generation-fact-log -public: true -category: concepts -template: concept -order: 6 -description: Every committed write also appends a self-verifying "fact" — an after-image commit record — to an append-only log. What facts are, the crash-safety model, the scanFacts() streaming surface, family stamps, and how index providers consume the log for sequential heals. -next: - - concepts/consistency-model - - guides/snapshots-and-time-travel ---- - -# The Generation Fact Log - -Since 8.4.0, every committed generation also appends a **fact** — a compact record of what each -touched entity or relationship *became* — to an append-only, checksummed log under -`_generations/facts/`. Where the generational history answers *"what did things look like -before?"* (before-images, powering `asOf()` and rollback), the fact log answers *"what happened, -in order?"* — one sequential, self-verifying stream of the store's present being written. - -Nothing about querying changes. The fact log exists for three consumers: - -1. **Index heals and rebuilds** — one sequential read in commit order replaces a per-entity - directory walk over millions of files. -2. **Incremental catch-up** — a derived index that knows which generation it reflects reads *just - the gap*, instead of rebuilding from scratch. -3. **Replay and audit tooling** — anything that wants the store's committed timeline as a stream. - -## What a fact is - -One fact per committed generation: - -- **`generation`** and **`timestamp`** — which commit, when. -- **`ops`** — every write in that commit: `{ kind: 'noun' | 'verb', id, record }` where `record` - holds the entity's full after-image (both stored legs), or **`null` for a tombstone** — a - removal carries no body, by design. -- **`meta`** — the transaction metadata `transact()` was submitted with, when present. -- **`blobHashes`** — content-blob references, for exact reclamation accounting. - -Facts accumulate **from the first write after upgrading** — pre-existing history is not -retroactively converted, and consumers fall back to the enumeration walk when no log exists. - -## Crash safety, in one paragraph - -Facts are appended and fsynced **inside the same durability window as the commit itself**, before -the commit point — so after a crash, the log can only ever be *ahead* of committed truth, never -behind it with a hole. On open, the store reconciles the log back to the committed watermark: -torn tails are detected by per-record checksums and cut; whole records beyond the watermark are -truncated. The invariant every reader can rely on: **an absent generation was never committed; a -present fact was.** `transact()` facts are durable the moment `transact()` returns; single-op -facts share the same group-commit flush as the rest of their generation, so a hard kill loses the -fact and the generation *together* — never a torn state. - -## Reading the log - -```typescript -const scan = brain.scanFacts({ fromGeneration: 1 }) -if (scan) { - // Telemetry up front — progress bars get a denominator from second zero. - console.log(scan.headGeneration, scan.segmentCount, scan.approxFactCount) - - for await (const batch of scan.batches()) { - // Each batch: { facts, firstGeneration, lastGeneration, factCount, byteSize, segmentId } - for (const fact of batch.facts) { - for (const op of fact.ops) { - if (op.record === null) { - // a tombstone: op.id was removed in this generation - } - } - } - } - - console.log(scan.summary()) // { factsYielded, segmentsRead } — the cross-check -} -``` - -- `scanFacts()` returns `null` when the store hosts no fact log (older store, or a storage adapter - without binary append support) — fall back to enumerating entities. -- Scans run against a **snapshot**: facts appended after the scan opens never bleed in, each fact - is yielded exactly once, and a detected gap aborts loudly — never a silent skip. -- `brain.factSegmentPaths()` returns the immutable, *sealed* segment files for zero-copy consumers - (the append-mutable tail is excluded — read it through `scanFacts()`). - -## Family stamps: how a projection proves it's current - -Anything derived from the store — an index, the entity file tree itself — carries a **family -stamp**: a small JSON record of *which committed generation the projection reflects* -(`sourceGeneration`) plus the invariants that verify it whole (exact per-file byte sizes for -bounded families; rollup invariants like entity counts for unbounded ones). At open, coherence is -a **comparison**, not a walk: - -- stamp equals the committed watermark and invariants hold → serve; -- stamp is behind → the projection reads just the gap from the fact log; -- invariants fail → loud, named divergence — `brain.repairIndex()` rebuilds from canonical and - re-stamps. - -The verifier is exported (`verifyFamilyStamp`) so every projection — TypeScript or native — runs -literally the same check. - -## For plugin authors: the storage capability - -Index providers receive the storage adapter, not the brain — so the host wires the log onto it. -Feature-detect and prefer the stream; fall back to enumeration: - -```typescript -const scan = storage.scanFacts?.({ fromGeneration: stamp.sourceGeneration + 1 }) -if (scan) { - // sequential catch-up from the log -} else { - // enumeration walk (older store or adapter) -} -const committed = storage.committedGeneration?.() // the watermark stamps compare against -``` - -Providers must never construct their own reader over the log's files — the open path belongs to -the single writer (it reconciles the log at open); the capability is the sanctioned seam. diff --git a/docs/concepts/multi-process.md b/docs/concepts/multi-process.md deleted file mode 100644 index 8fda315f..00000000 --- a/docs/concepts/multi-process.md +++ /dev/null @@ -1,153 +0,0 @@ ---- -title: Multi-Process Model -slug: concepts/multi-process -public: true -category: concepts -template: concept -order: 5 -description: How Brainy coordinates a single writer with any number of readers on a filesystem data directory — and how to safely inspect a live store. -next: - - guides/inspection ---- - -# Multi-Process Model - -Brainy is a **single-writer, many-reader** database when backed by filesystem -storage. This page explains the model, the guarantees, and the safe ways to -inspect a live store from a second process. - -## The rule - -For one data directory: - -- **One writer** at a time. The writer acquires an exclusive lock on the - directory at `init()` and releases it on `close()`. -- **Any number of readers**, concurrent with each other and with the writer. - Readers open via `Brainy.openReadOnly()` — they never touch the writer - lock. - -Any attempt to open a second writer on the same directory throws: - -``` -BrainyError: Another writer holds this Brainy directory. - PID: 1774431 on host app-host-1 - Started: 2026-05-15T14:22:11Z - Heartbeat: 2026-05-15T14:22:34Z - Version: 7.21.0 - Directory: /data/brain -``` - -This is intentional. Two writers sharing a directory would silently corrupt -in-memory indexes and produce wrong query results — the worst possible default -for an operations tool. - -## Why a lock? - -Brainy keeps its primary indexes (HNSW, metadata, graph adjacency) in memory. -On disk, those indexes are persisted incrementally as writes flush. A second -process opening the same directory: - -- Loads the *persisted* state into a fresh in-memory copy. -- Has no awareness of writes the first process buffered but hasn't flushed. -- Will overwrite the persisted state on its own next flush, racing the first - process and corrupting whichever wins. - -The fix is the lock: refuse to open a second writer. SQLite has done the same -since the late 1990s (`SQLITE_BUSY`). - -## What about Cor? - -Brainy + Cor compose cleanly under this model: - -- Cor stores its column-index segments inside the same `rootDir` (under - `indexes/_column_index/{field}/`). -- Segments (`*.cidx` files) are **immutable** once written. Cor mmaps them - read-only. -- The `MANIFEST.json` per field is updated via atomic rename — readers see - either the old or new manifest, never a torn file. - -A reader process can safely mmap Cor segments alongside a live writer -without coordination. The single Brainy writer lock at -`/locks/_writer.lock` covers Cor too, because Cor segment -writes happen on the writer's side. - -## Stale-lock detection - -If a writer crashes or is forcibly killed, its lock file is left behind. To -avoid a permanently-jammed directory, Brainy treats a lock as stale when: - -1. The recorded `hostname` equals the current host (cross-host PID checks - are unsafe), AND -2. The recorded `pid` is no longer alive (`process.kill(pid, 0)` returns - `ESRCH`), OR the `lastHeartbeat` field is older than 60 seconds. - -A live writer rewrites `lastHeartbeat` every 10 seconds, so a hung writer -that's missed several heartbeats is treated as dead. Stale locks are -overwritten with a warning. - -If stale detection cannot prove the existing lock is dead — for example, a -crashed writer on a different host writing to a shared filesystem — pass -`{ force: true }` to override. A warning is logged either way. - -## Heartbeat and shutdown - -The heartbeat interval rewrites the lock file every 10 seconds. The timer -is unref'd, so it does not keep the event loop alive on its own. - -On normal shutdown the writer releases the lock in `close()`. The shutdown -hooks Brainy registers for `SIGTERM`, `SIGINT`, and `beforeExit` also -release the lock so a container restart doesn't strand the directory. - -## How to inspect a live writer - -Use `Brainy.openReadOnly()`. It does not acquire the writer lock, so it -coexists with whatever the writer is doing: - -```typescript -const reader = await Brainy.openReadOnly({ - storage: { type: 'filesystem', path: '/data/brain' } -}) - -const stats = await reader.stats() -const bookings = await reader.find({ where: { entityType: 'booking' } }) - -await reader.close() -``` - -What the reader sees reflects the writer's most recent **flush** to disk. If -you need fresher state, ask the writer to flush before opening: - -```typescript -const reader = await Brainy.openReadOnly({ - storage: { type: 'filesystem', path: '/data/brain' } -}) - -const acked = await reader.requestFlush({ timeoutMs: 5000 }) -if (!acked) { - console.warn('Writer did not respond; results reflect last natural flush.') -} - -const fresh = await reader.find({ where: { entityType: 'booking' } }) -``` - -The CLI `brainy inspect` subcommands all do this for you by default -(`--no-fresh` to opt out). - -## What's not enforced (yet) - -- **Non-filesystem backends** are out of scope in 8.0, which ships only the - filesystem and memory adapters. A custom `BaseStorage` subclass that is not - filesystem-backed does not enforce multi-process locking by default: two - processes can both succeed at `init()` in writer mode and clobber each - other's writes. A best-effort warning is logged in writer mode against a - non-filesystem backend. -- **Long-running readers** do not automatically pick up new Cor segments - the writer publishes. One-shot inspector calls re-open the store and see - fresh segments; a reader that stays open for hours sees its column store - as-of the time it opened. - -## Reading material - -- `Brainy.openReadOnly()` — [API reference](../api/brainy.md) -- `brainy inspect` — [inspection guide](../guides/inspection.md) -- Cor columnar storage — see `node_modules/@soulcraft/cor/README.md` diff --git a/docs/concepts/storage-adapters.md b/docs/concepts/storage-adapters.md deleted file mode 100644 index af6d068f..00000000 --- a/docs/concepts/storage-adapters.md +++ /dev/null @@ -1,189 +0,0 @@ ---- -title: Storage Adapter Inheritance Contract -slug: concepts/storage-adapters -public: true -category: concepts -template: concept -order: 6 -description: How storage adapters extend Brainy's BaseStorage and FileSystemStorage to inherit multi-process safety, what plugin authors need to override, and the contract Brainy promises to keep stable. -next: - - concepts/multi-process - - guides/inspection ---- - -# Storage Adapter Inheritance Contract - -Brainy's storage layer is designed for plugins to extend cleanly. A plugin -that subclasses `BaseStorage` or `FileSystemStorage` inherits new behavior -Brainy adds over time without code changes — provided the import resolution -brings in the right Brainy version at runtime. - -This page documents what plugin authors can rely on, what they need to -override, and the install-time failure modes that the defensive -`hasStorageMethod()` guard exists to handle. - -## The class hierarchy - -``` -BaseStorageAdapter (counts, batch ops, multi-tenancy hooks) - ↑ -BaseStorage (type-statistics, lifecycle helpers, generation - hooks, default no-op multi-process methods) - ↑ -FileSystemStorage (real filesystem I/O, writer-lock implementation, - flush-request watcher, atomic writes) - ↑ - (Cor's MmapFileSystemStorage, etc.) -``` - -When you `extend FileSystemStorage`, your adapter inherits every method on -the chain — including the ones Brainy adds in a later release — for free. -JavaScript's prototype chain resolves method lookups dynamically; nothing -about the inheritance is baked in at class-definition time. - -## What you inherit for free - -A plugin whose storage class extends `FileSystemStorage` automatically gets -the full multi-process safety surface: - -| Method | What it does | Override? | -|---|---|---| -| `acquireWriterLock(opts)` | Write `_writer.lock`, start heartbeat, throw on conflict | No | -| `releaseWriterLock()` | Clean up lock file + heartbeat timer | No | -| `readWriterLock()` | Read lock file as `WriterLockInfo` | No | -| `startFlushRequestWatcher(cb)` | Poll `_flush_requests/` and invoke `cb` | No | -| `stopFlushRequestWatcher()` | Stop the polling timer | No | -| `requestFlushOverFilesystem(timeoutMs)` | Drop a `.req`, await `.ack` | No | -| `supportsMultiProcessLocking()` | Return `false` (default) | **Yes — override to `true`** | - -The only required override is the capability flag. Returning `true` from -`supportsMultiProcessLocking()` is the signal Brainy uses to decide whether -to call `acquireWriterLock()` at init. - -```typescript -import { FileSystemStorage } from '@soulcraft/brainy' - -export class MmapFileSystemStorage extends FileSystemStorage { - public supportsMultiProcessLocking(): boolean { - return true - } - // ... your mmap-specific overrides ... -} -``` - -That's the full ceremony for inheriting multi-process safety. - -## When NOT to extend FileSystemStorage - -If your storage is **not filesystem-backed** (a custom -network backend), extend `BaseStorage` directly: - -```typescript -import { BaseStorage } from '@soulcraft/brainy' - -export class MyCloudStorage extends BaseStorage { - // BaseStorage's default no-op implementations of the multi-process - // methods stay in effect. `supportsMultiProcessLocking()` returns false - // by default — keep it that way unless you've implemented an object- - // versioned lease or similar cross-process synchronization for your - // backend. -} -``` - -Brainy treats cloud backends as not-multi-process-safe by default and logs a -one-line warning at init. That's the correct behavior until cloud locking -ships (currently out of scope — see -[`concepts/multi-process`](./multi-process.md)). - -## What `hasStorageMethod()` actually guards against - -The defensive check at every new-storage-method call site (`brainy.ts`, -`hasStorageMethod(name)`) does **not** exist to handle "plugin bundles a -stale BaseStorage." Plugins ship a dist that preserves the dynamic ESM -import (verify in your plugin's `dist/`: `import { FileSystemStorage } from -'@soulcraft/brainy'` is not rewritten to a vendored copy). The prototype -chain at runtime resolves to whatever Brainy version your consumer has -installed. - -`hasStorageMethod()` protects against **build/install artifacts** that break -the prototype chain at the consumer-app level: - -- **Stale `node_modules`** — a lingering install from before the consumer - upgraded Brainy. The package.json says `@soulcraft/brainy@7.22.0` but - `node_modules/@soulcraft/brainy` is still 7.20.x. -- **Lockfile drift** — `bun.lockb` / `package-lock.json` pins a brainy - version older than the package.json range, and `bun install` honors the - lockfile. -- **Docker layer cache** — the image reuses a `node_modules` from an - earlier build that predates the brainy bump. -- **Bundler quirks** — some bundlers (esbuild, webpack) flatten the - prototype chain at build time and lose later prototype mutations. Brainy - doesn't mutate prototypes at runtime, but bundler behavior can still - cause method lookups to fail in non-Node environments. - -In any of those, calling `storage.acquireWriterLock(...)` unconditionally -throws `TypeError: storage.acquireWriterLock is not a function`. The guard -turns that into a logged warning + graceful no-op so the app still boots, -and the warning names the adapter class plus a remediation hint: - -``` -[brainy] Storage adapter `MmapFileSystemStorage` is missing the multi-process -methods on its prototype chain. Writer locking and the flush-request RPC are -disabled for this directory. Likely fix: clean install (`rm -rf node_modules -bun.lockb && bun install`) or rebuild your container image to refresh -`@soulcraft/brainy` to ≥7.21. See docs/concepts/storage-adapters.md. -``` - -## Authoring a new storage adapter — minimum checklist - -1. **Extend the right base class.** - - Filesystem-backed → `FileSystemStorage`. - - Cloud / network / custom → `BaseStorage`. - -2. **Override the capability flag.** If filesystem-backed: - ```typescript - public supportsMultiProcessLocking(): boolean { return true } - ``` - -3. **Don't `super.X()`-wrap the multi-process methods**. They're inherited; - leaving them inherited means `hasStorageMethod()` finds them on the - prototype chain. Re-declaring them as `super.X()` wrappers makes the - helper resolve to your wrapper, which can fool the guard if your - constructor runs before the super class initializes. - -4. **Do override `init()` / `flush()` / `close()`** as needed. Always call - `super.init()` / `super.flush()` / `super.close()` first so the - filesystem prep, writer-lock acquisition, and lock release happen in the - expected order. - -5. **Verify the inheritance.** A one-line smoke test in your plugin's - `__tests__/`: - ```typescript - const s = new MyStorage(rootDir) - await s.init() - assert(typeof s.acquireWriterLock === 'function') - assert(s.supportsMultiProcessLocking() === true) - ``` - If `acquireWriterLock` is undefined the prototype chain is broken at - install time — fix install, not your plugin. - -6. **Pin your peer dep generously.** `"peerDependencies": { - "@soulcraft/brainy": "^7.21.0" }` accepts any compatible 7.x. Don't pin - to an exact patch unless you're tracking a known regression. - -## Future direction - -The 7 multi-process methods are currently defaults on `BaseStorage`. A -future refactor may extract them into a `MultiProcessSafeStorage` -interface/mixin for cleaner separation — only adapters that opt in would -expose them. This would require a minor bump and is tracked as an internal -follow-up; consumers don't need to anticipate the change. - -## Reading material - -- [`concepts/multi-process`](./multi-process.md) — the writer-lock model, - heartbeat semantics, what the lock protects. -- [`guides/inspection`](../guides/inspection.md) — `brainy inspect` and the - read-only mode. -- `node_modules/@soulcraft/brainy/dist/storage/baseStorage.d.ts` — the - authoritative type signatures for every method this page references. diff --git a/docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md b/docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md new file mode 100644 index 00000000..3a72a966 --- /dev/null +++ b/docs/deployment/CLOUD_DEPLOYMENT_GUIDE.md @@ -0,0 +1,756 @@ +# Brainy 3.0 Cloud Deployment Guide + +This guide provides production-ready deployment configurations for Brainy using S3CompatibleStorage (preferred) or FileSystemStorage across major cloud platforms. All examples are verified against the actual Brainy 3.0 codebase. + +## Overview + +The API Server augmentation provides a universal handler that works with standard Request/Response objects, making Brainy deployable on any JavaScript runtime. + +## Storage Adapter + +**S3CompatibleStorage** is the preferred storage adapter for cloud deployments. It works with: +- Amazon S3 +- Cloudflare R2 +- Google Cloud Storage +- Azure Blob Storage +- Any S3-compatible service + +## Deployment Examples + +### AWS Lambda + +```javascript +// handler.js +const { Brainy } = require('@soulcraft/brainy') +const { S3CompatibleStorage } = require('@soulcraft/brainy/storage') + +let brain +let handler + +exports.handler = async (event) => { + if (!brain) { + const storage = new S3CompatibleStorage({ + endpoint: 's3.amazonaws.com', + region: process.env.AWS_REGION, + bucket: process.env.BRAINY_BUCKET, + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, + prefix: 'brainy-data/' + }) + + brain = new Brainy({ + storage, + augmentations: [{ + name: 'api-server', + config: { + enabled: true, + auth: { + required: true, + apiKeys: [process.env.API_KEY] + } + } + }] + }) + + await brain.init() + + // Get the universal handler from the augmentation + const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') + handler = apiAugmentation.createUniversalHandler() + } + + // Convert Lambda event to Request + const url = `https://${event.requestContext.domainName}${event.rawPath}` + const request = new Request(url, { + method: event.requestContext.http.method, + headers: event.headers, + body: event.body + }) + + // Use the universal handler + const response = await handler(request) + + return { + statusCode: response.status, + headers: Object.fromEntries(response.headers), + body: await response.text() + } +} +``` + +### Google Cloud Functions + +```javascript +// index.js +const { Brainy } = require('@soulcraft/brainy') +const { S3CompatibleStorage } = require('@soulcraft/brainy/storage') + +let brain +let handler + +exports.brainyAPI = async (req, res) => { + if (!brain) { + const storage = new S3CompatibleStorage({ + endpoint: 'storage.googleapis.com', + bucket: process.env.GCS_BUCKET, + accessKeyId: process.env.GCS_ACCESS_KEY, + secretAccessKey: process.env.GCS_SECRET_KEY, + prefix: 'brainy/', + forcePathStyle: false, + region: 'US' + }) + + brain = new Brainy({ storage }) + await brain.init() + + const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') + handler = apiAugmentation.createUniversalHandler() + } + + // Convert Express req/res to Request/Response + const request = new Request(`https://${req.hostname}${req.originalUrl}`, { + method: req.method, + headers: req.headers, + body: JSON.stringify(req.body) + }) + + const response = await handler(request) + + res.status(response.status) + res.set(Object.fromEntries(response.headers)) + res.send(await response.text()) +} +``` + +### Google Cloud Run with Cloud Storage + +Google Cloud Run is ideal for containerized deployments with automatic scaling. This example uses Google Cloud Storage via the S3-compatible API. + +```javascript +// server.js +import { Brainy } from '@soulcraft/brainy' +import { S3CompatibleStorage } from '@soulcraft/brainy/storage' +import express from 'express' + +const app = express() +app.use(express.json()) + +const PORT = process.env.PORT || 8080 + +let brain +let handler + +async function initBrainy() { + // Google Cloud Storage is S3-compatible + const storage = new S3CompatibleStorage({ + endpoint: 'storage.googleapis.com', + bucket: process.env.GCS_BUCKET || 'brainy-data', + accessKeyId: process.env.GCS_ACCESS_KEY, + secretAccessKey: process.env.GCS_SECRET_KEY, + prefix: 'brainy/', + forcePathStyle: false, + region: process.env.GCS_REGION || 'US' + }) + + brain = new Brainy({ + storage, + augmentations: [{ + name: 'api-server', + config: { + enabled: true, + port: PORT, + cors: { + origin: process.env.CORS_ORIGIN || '*' + }, + auth: { + required: process.env.AUTH_REQUIRED === 'true', + apiKeys: process.env.API_KEY ? [process.env.API_KEY] : [] + } + } + }] + }) + + await brain.init() + + const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') + handler = apiAugmentation.createUniversalHandler() +} + +// Initialize on startup +await initBrainy() + +// Health check endpoint +app.get('/health', (req, res) => { + res.json({ status: 'healthy', service: 'brainy-api' }) +}) + +// Universal handler for all API routes +app.use('*', async (req, res) => { + const request = new Request(`https://${req.hostname}${req.originalUrl}`, { + method: req.method, + headers: req.headers, + body: req.method !== 'GET' ? JSON.stringify(req.body) : undefined + }) + + const response = await handler(request) + + res.status(response.status) + res.set(Object.fromEntries(response.headers)) + res.send(await response.text()) +}) + +app.listen(PORT, () => { + console.log(`Brainy API running on port ${PORT}`) +}) +``` + +```dockerfile +# Dockerfile +FROM node:20-alpine AS builder +WORKDIR /app +COPY package*.json ./ +RUN npm ci --only=production + +FROM node:20-alpine +WORKDIR /app +COPY --from=builder /app/node_modules ./node_modules +COPY . . +EXPOSE 8080 +CMD ["node", "server.js"] +``` + +```yaml +# cloudbuild.yaml +steps: + # Build the container image + - name: 'gcr.io/cloud-builders/docker' + args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA', '.'] + + # Push to Container Registry + - name: 'gcr.io/cloud-builders/docker' + args: ['push', 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA'] + + # Deploy to Cloud Run + - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk' + entrypoint: gcloud + args: + - 'run' + - 'deploy' + - 'brainy-api' + - '--image' + - 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA' + - '--region' + - 'us-central1' + - '--platform' + - 'managed' + - '--allow-unauthenticated' + - '--set-env-vars' + - 'GCS_BUCKET=brainy-storage,GCS_REGION=US' + - '--set-secrets' + - 'GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest' + - '--memory' + - '2Gi' + - '--cpu' + - '2' + - '--max-instances' + - '100' + - '--min-instances' + - '0' + +images: + - 'gcr.io/$PROJECT_ID/brainy-api:$COMMIT_SHA' +``` + +**Deploy with gcloud CLI:** +```bash +# Build and submit to Cloud Build +gcloud builds submit --config cloudbuild.yaml + +# Or deploy directly +gcloud run deploy brainy-api \ + --source . \ + --platform managed \ + --region us-central1 \ + --allow-unauthenticated \ + --set-env-vars GCS_BUCKET=brainy-storage \ + --set-secrets "GCS_ACCESS_KEY=gcs-access-key:latest,GCS_SECRET_KEY=gcs-secret-key:latest" +``` + +**Create GCS Bucket with S3-compatible access:** +```bash +# Create bucket +gsutil mb -p PROJECT_ID -c STANDARD -l US gs://brainy-storage/ + +# Enable interoperability +gsutil iam ch serviceAccount:SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com:objectAdmin gs://brainy-storage + +# Generate HMAC keys for S3-compatible access +gsutil hmac create SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com + +# Store the access key and secret in Secret Manager +echo -n "YOUR_ACCESS_KEY" | gcloud secrets create gcs-access-key --data-file=- +echo -n "YOUR_SECRET_KEY" | gcloud secrets create gcs-secret-key --data-file=- +``` + +### Microsoft Azure Functions + +```javascript +// index.js +module.exports = async function (context, req) { + const { Brainy } = require('@soulcraft/brainy') + const { S3CompatibleStorage } = require('@soulcraft/brainy/storage') + + const storage = new S3CompatibleStorage({ + endpoint: `${process.env.AZURE_STORAGE_ACCOUNT}.blob.core.windows.net`, + bucket: 'brainy-data', + accessKeyId: process.env.AZURE_STORAGE_ACCOUNT, + secretAccessKey: process.env.AZURE_STORAGE_KEY, + prefix: 'entities/', + forcePathStyle: false + }) + + const brain = new Brainy({ storage }) + await brain.init() + + const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') + const handler = apiAugmentation.createUniversalHandler() + + const request = new Request(`https://${context.req.headers.host}${context.req.url}`, { + method: context.req.method, + headers: context.req.headers, + body: JSON.stringify(context.req.body) + }) + + const response = await handler(request) + + context.res = { + status: response.status, + headers: Object.fromEntries(response.headers), + body: await response.text() + } +} +``` + +### Cloudflare Workers + +```javascript +// worker.js +import { Brainy } from '@soulcraft/brainy' +import { R2Storage } from '@soulcraft/brainy/storage' // Alias for S3CompatibleStorage + +let handler + +export default { + async fetch(request, env, ctx) { + if (!handler) { + const storage = new R2Storage({ + endpoint: `${env.ACCOUNT_ID}.r2.cloudflarestorage.com`, + bucket: 'brainy-data', + accessKeyId: env.R2_ACCESS_KEY_ID, + secretAccessKey: env.R2_SECRET_ACCESS_KEY, + region: 'auto', + forcePathStyle: true + }) + + const brain = new Brainy({ + storage, + augmentations: [{ + name: 'api-server', + config: { + enabled: true, + cors: { origin: '*' } + } + }] + }) + + await brain.init() + + const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') + handler = apiAugmentation.createUniversalHandler() + } + + // The handler works directly with Request/Response! + return handler(request) + } +} +``` + +```toml +# wrangler.toml +name = "brainy-api" +main = "worker.js" +compatibility_date = "2024-01-01" + +[[r2_buckets]] +binding = "R2" +bucket_name = "brainy-data" + +[vars] +ACCOUNT_ID = "your-account-id" + +[env.production.vars] +R2_ACCESS_KEY_ID = "your-access-key" +R2_SECRET_ACCESS_KEY = "your-secret-key" +``` + +### Vercel Edge Functions + +```javascript +// api/brainy.js +import { Brainy } from '@soulcraft/brainy' +import { S3CompatibleStorage } from '@soulcraft/brainy/storage' + +let handler + +export const config = { + runtime: 'edge', +} + +export default async (request) => { + if (!handler) { + const storage = new S3CompatibleStorage({ + endpoint: 's3.amazonaws.com', + region: 'us-east-1', + bucket: process.env.S3_BUCKET, + accessKeyId: process.env.AWS_ACCESS_KEY_ID, + secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY + }) + + const brain = new Brainy({ + storage, + augmentations: [{ + name: 'api-server', + config: { enabled: true } + }] + }) + + await brain.init() + + const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') + handler = apiAugmentation.createUniversalHandler() + } + + return handler(request) +} +``` + +```json +// vercel.json +{ + "functions": { + "api/brainy.js": { + "maxDuration": 30, + "memory": 1024 + } + }, + "rewrites": [ + { + "source": "/api/:path*", + "destination": "/api/brainy" + } + ] +} +``` + +### Railway + +```javascript +// server.js +import { Brainy } from '@soulcraft/brainy' +import { S3CompatibleStorage } from '@soulcraft/brainy/storage' +import express from 'express' + +const app = express() +const PORT = process.env.PORT || 3000 + +let brain +let handler + +async function init() { + const storage = new S3CompatibleStorage({ + endpoint: process.env.S3_ENDPOINT || 's3.amazonaws.com', + region: process.env.S3_REGION || 'us-east-1', + bucket: process.env.S3_BUCKET, + accessKeyId: process.env.S3_ACCESS_KEY, + secretAccessKey: process.env.S3_SECRET_KEY, + prefix: 'brainy/' + }) + + brain = new Brainy({ + storage, + augmentations: [{ + name: 'api-server', + config: { + enabled: true, + port: PORT + } + }] + }) + + await brain.init() + + const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') + handler = apiAugmentation.createUniversalHandler() +} + +await init() + +// Universal handler for all routes +app.use('*', async (req, res) => { + const request = new Request(`http://localhost${req.originalUrl}`, { + method: req.method, + headers: req.headers, + body: JSON.stringify(req.body) + }) + + const response = await handler(request) + + res.status(response.status) + res.set(Object.fromEntries(response.headers)) + res.send(await response.text()) +}) + +app.listen(PORT, () => { + console.log(`Brainy API running on port ${PORT}`) +}) +``` + +```toml +# railway.toml +[build] +builder = "nixpacks" +buildCommand = "npm ci" + +[deploy] +startCommand = "node server.js" +restartPolicyType = "always" +restartPolicyMaxRetries = 3 +``` + +### Render + +```javascript +// server.js (same as Railway example above) +// Use S3CompatibleStorage with your preferred object storage provider +``` + +```yaml +# render.yaml +services: + - type: web + name: brainy-api + runtime: node + buildCommand: npm install + startCommand: node server.js + envVars: + - key: S3_BUCKET + value: brainy-data + - key: S3_ENDPOINT + value: s3.amazonaws.com + - key: S3_REGION + value: us-east-1 + - key: S3_ACCESS_KEY + sync: false + - key: S3_SECRET_KEY + sync: false + - key: API_KEY + generateValue: true + healthCheckPath: /health + autoDeploy: true +``` + +### Deno Deploy + +```typescript +// main.ts +import { Brainy } from "npm:@soulcraft/brainy" +import { S3CompatibleStorage } from "npm:@soulcraft/brainy/storage" + +const storage = new S3CompatibleStorage({ + endpoint: Deno.env.get("S3_ENDPOINT") || "s3.amazonaws.com", + bucket: Deno.env.get("S3_BUCKET")!, + accessKeyId: Deno.env.get("S3_ACCESS_KEY")!, + secretAccessKey: Deno.env.get("S3_SECRET_KEY")!, + region: "auto" +}) + +const brain = new Brainy({ + storage, + augmentations: [{ + name: 'api-server', + config: { enabled: true } + }] +}) + +await brain.init() + +const apiAugmentation = brain.augmentationRegistry.getAugmentation('api-server') +const handler = apiAugmentation.createUniversalHandler() + +Deno.serve(handler) +``` + +## API Endpoints + +The API Server augmentation provides these REST endpoints: + +- `POST /api/brainy/add` - Add entity +- `GET /api/brainy/get?id=xxx` - Get entity by ID +- `PUT /api/brainy/update` - Update entity +- `DELETE /api/brainy/delete?id=xxx` - Delete entity +- `POST /api/brainy/find` - Search/find entities +- `POST /api/brainy/relate` - Create relationship +- `GET /api/brainy/insights` - Get statistics and insights +- `GET /health` - Health check + +## WebSocket Support + +The API Server augmentation includes WebSocket support for real-time updates through the `setupUniversalWebSocket()` method. + +## MCP Support + +Model Context Protocol (MCP) endpoints are available at `/mcp/*` for AI tool integration. + +## Environment Variables + +```bash +# Storage Configuration (S3Compatible) +S3_ENDPOINT=s3.amazonaws.com +S3_REGION=us-east-1 +S3_BUCKET=brainy-data +S3_ACCESS_KEY=xxx +S3_SECRET_KEY=xxx + +# API Configuration +API_KEY=your-secret-key +PORT=3000 + +# CORS +CORS_ORIGIN=* + +# Rate Limiting +RATE_LIMIT_WINDOW=60000 +RATE_LIMIT_MAX=100 +``` + +## Client Usage + +```javascript +// REST API Client +const response = await fetch('https://your-api.com/api/brainy/add', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Bearer YOUR_API_KEY' + }, + body: JSON.stringify({ + data: 'Your content here', + metadata: { type: 'document' } + }) +}) + +const { id } = await response.json() + +// Search +const searchResponse = await fetch('https://your-api.com/api/brainy/find', { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + 'Authorization': 'Bearer YOUR_API_KEY' + }, + body: JSON.stringify({ + query: 'neural networks' + }) +}) + +const results = await searchResponse.json() + +// WebSocket Client +const ws = new WebSocket('wss://your-api.com/ws') +ws.onopen = () => { + ws.send(JSON.stringify({ + type: 'subscribe', + pattern: 'technology' + })) +} +ws.onmessage = (event) => { + const update = JSON.parse(event.data) + console.log('Real-time update:', update) +} +``` + +## Storage Adapter Configuration + +S3CompatibleStorage constructor parameters (verified from source): + +```javascript +{ + endpoint: string, // Required (e.g., 's3.amazonaws.com') + bucket: string, // Required + accessKeyId: string, // Required + secretAccessKey: string, // Required + region?: string, // Optional (default: 'us-east-1') + prefix?: string, // Optional (e.g., 'brainy/') + forcePathStyle?: boolean, // Optional (needed for some S3-compatible services) +} +``` + +## Important Notes + +1. All examples use the **real** Brainy 3.0 APIs +2. The `createUniversalHandler()` method is provided by the API Server augmentation +3. S3CompatibleStorage works with any S3-compatible service +4. Always call `brain.init()` before using Brainy +5. The handler can be cached across requests for better performance +6. R2Storage is an alias for S3CompatibleStorage (for Cloudflare R2) + +## Security Best Practices + +1. **Always use environment variables for sensitive data** (API keys, secrets) +2. **Enable authentication** in the API Server augmentation config +3. **Use HTTPS/TLS** for all production deployments +4. **Implement rate limiting** to prevent abuse +5. **Configure CORS** appropriately for your use case + +## Performance Tips + +1. **Cache the brain instance** - Initialize once and reuse across requests +2. **Use S3CompatibleStorage** for cloud deployments (better scalability) +3. **Enable the cache augmentation** for frequently accessed data +5. **Configure appropriate memory limits** for your runtime + +## Troubleshooting + +### Common Issues + +1. **"Brainy not initialized"** - Make sure to call `await brain.init()` before use +2. **"augmentationRegistry.getAugmentation is not a function"** - The augmentation wasn't loaded properly +3. **S3 Access Denied** - Check your IAM permissions and credentials +4. **CORS errors** - Configure the CORS settings in the API Server augmentation + +### Debug Mode + +Enable debug logging by setting: +```javascript +const brain = new Brainy({ + storage, + debug: true, + augmentations: [{ + name: 'api-server', + config: { + enabled: true, + verbose: true + } + }] +}) +``` + +## Support + +- Documentation: https://github.com/soulcraft/brainy/docs +- Issues: https://github.com/soulcraft/brainy/issues +- NPM Package: https://www.npmjs.com/package/@soulcraft/brainy + +--- + +This guide contains only verified, working code from the actual Brainy 3.0 codebase. No mock implementations, no stub methods, only production-ready code. \ No newline at end of file diff --git a/docs/deployment/aws-deployment.md b/docs/deployment/aws-deployment.md new file mode 100644 index 00000000..671fee74 --- /dev/null +++ b/docs/deployment/aws-deployment.md @@ -0,0 +1,416 @@ +# AWS Deployment Guide for Brainy + +## Overview +Deploy Brainy on AWS with automatic scaling, high availability, and zero-config dynamic adaptation. Brainy automatically adapts to your AWS environment without manual configuration. + +## Quick Start (Zero-Config) + +### Option 1: AWS Lambda (Serverless) + +```bash +# Install Brainy +npm install @soulcraft/brainy + +# Create handler.js +cat > handler.js << 'EOF' +const { Brainy } = require('@soulcraft/brainy') + +let brain + +exports.handler = async (event) => { + // Brainy auto-detects Lambda environment and configures accordingly + if (!brain) { + brain = new Brainy() // Zero config - auto-adapts to Lambda + await brain.init() + } + + const { method, ...params } = JSON.parse(event.body) + + switch(method) { + case 'add': + const id = await brain.add(params) + return { statusCode: 200, body: JSON.stringify({ id }) } + case 'find': + const results = await brain.find(params) + return { statusCode: 200, body: JSON.stringify({ results }) } + default: + return { statusCode: 400, body: 'Unknown method' } + } +} +EOF + +# Deploy with AWS SAM +sam init --runtime nodejs20.x --name brainy-app +sam deploy --guided +``` + +### Option 2: ECS Fargate (Container) + +```bash +# Build and push Docker image +aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $ECR_URI +docker build -t brainy . +docker tag brainy:latest $ECR_URI/brainy:latest +docker push $ECR_URI/brainy:latest + +# Deploy with minimal ECS task definition +cat > task-definition.json << 'EOF' +{ + "family": "brainy", + "networkMode": "awsvpc", + "requiresCompatibilities": ["FARGATE"], + "cpu": "256", + "memory": "512", + "containerDefinitions": [{ + "name": "brainy", + "image": "$ECR_URI/brainy:latest", + "environment": [ + {"name": "NODE_ENV", "value": "production"} + ], + "logConfiguration": { + "logDriver": "awslogs", + "options": { + "awslogs-group": "/ecs/brainy", + "awslogs-region": "us-east-1", + "awslogs-stream-prefix": "ecs" + } + } + }] +} +EOF + +# Brainy auto-detects ECS environment and uses S3 for storage +aws ecs register-task-definition --cli-input-json file://task-definition.json +aws ecs create-service --cluster default --service-name brainy --task-definition brainy --desired-count 2 +``` + +### Option 3: EC2 Auto-Scaling + +```bash +# User data script for EC2 instances +#!/bin/bash +curl -fsSL https://rpm.nodesource.com/setup_20.x | sudo bash - +sudo yum install -y nodejs git + +# Clone and setup (or use your deployment method) +git clone https://github.com/yourorg/brainy-app.git /app +cd /app +npm install --production + +# Create systemd service +cat > /etc/systemd/system/brainy.service << 'EOF' +[Unit] +Description=Brainy +After=network.target + +[Service] +Type=simple +User=ec2-user +WorkingDirectory=/app +ExecStart=/usr/bin/node index.js +Restart=on-failure +Environment=NODE_ENV=production + +[Install] +WantedBy=multi-user.target +EOF + +systemctl start brainy +systemctl enable brainy +``` + +## Zero-Config Storage (Automatic) + +Brainy automatically detects and uses the best available storage: + +```javascript +// No configuration needed - Brainy auto-detects: +const brain = new Brainy() + +// Auto-detection priority: +// 1. S3 (if IAM role has permissions) +// 2. EFS (if mounted at /mnt/efs) +// 3. EBS volume (if available) +// 4. Instance storage (fallback) +``` + +### Manual S3 Configuration (Optional) + +```javascript +const brain = new Brainy({ + storage: { + type: 's3', + options: { + bucket: process.env.S3_BUCKET || 'auto', // 'auto' creates bucket + region: process.env.AWS_REGION || 'auto', // 'auto' detects region + // IAM role provides credentials automatically + } + } +}) +``` + +## Scaling Strategies + +### 1. Horizontal Scaling (Recommended) + +```yaml +# Auto-scaling policy +Resources: + AutoScalingTarget: + Type: AWS::ApplicationAutoScaling::ScalableTarget + Properties: + ServiceNamespace: ecs + ResourceId: service/default/brainy + ScalableDimension: ecs:service:DesiredCount + MinCapacity: 2 + MaxCapacity: 100 + + AutoScalingPolicy: + Type: AWS::ApplicationAutoScaling::ScalingPolicy + Properties: + PolicyType: TargetTrackingScaling + TargetTrackingScalingPolicyConfiguration: + TargetValue: 70.0 + PredefinedMetricSpecification: + PredefinedMetricType: ECSServiceAverageCPUUtilization +``` + +### 2. Vertical Scaling + +Brainy automatically adapts to available memory: +- **256MB**: Minimal mode, optimized caching +- **512MB**: Standard mode, balanced performance +- **1GB+**: Full mode, maximum performance + +## High Availability Setup + +### Multi-AZ Deployment + +```javascript +// Brainy automatically handles multi-AZ with S3 +const brain = new Brainy({ + distributed: { + enabled: true, // Auto-enables with S3 storage + coordinationMethod: 'auto' // Uses S3 for coordination + } +}) +``` + +### Load Balancing + +```bash +# Application Load Balancer with health checks +aws elbv2 create-load-balancer \ + --name brainy-alb \ + --subnets subnet-xxx subnet-yyy \ + --security-groups sg-xxx + +aws elbv2 create-target-group \ + --name brainy-targets \ + --protocol HTTP \ + --port 3000 \ + --vpc-id vpc-xxx \ + --health-check-path /health \ + --health-check-interval-seconds 30 +``` + +## Monitoring & Observability + +### CloudWatch Integration + +Brainy automatically sends metrics when running on AWS: + +```javascript +// Automatic CloudWatch metrics (no config needed) +// - Request count +// - Response time +// - Error rate +// - Storage usage +// - Memory usage +``` + +### Custom Metrics + +```javascript +const brain = new Brainy({ + monitoring: { + enabled: true, + customMetrics: { + namespace: 'Brainy/Production', + dimensions: [ + { Name: 'Environment', Value: 'production' }, + { Name: 'Service', Value: 'api' } + ] + } + } +}) +``` + +## Security Best Practices + +### 1. IAM Role (Recommended) + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:PutObject", + "s3:DeleteObject", + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::brainy-*/*", + "arn:aws:s3:::brainy-*" + ] + } + ] +} +``` + +### 2. VPC Configuration + +```bash +# Private subnets with NAT Gateway +aws ec2 create-vpc --cidr-block 10.0.0.0/16 +aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.1.0/24 --availability-zone us-east-1a +aws ec2 create-subnet --vpc-id vpc-xxx --cidr-block 10.0.2.0/24 --availability-zone us-east-1b +``` + +### 3. Encryption + +```javascript +// Automatic encryption with S3 +const brain = new Brainy({ + storage: { + type: 's3', + options: { + encryption: 'auto' // Uses S3 SSE-S3 by default + } + } +}) +``` + +## Cost Optimization + +### 1. Spot Instances (70% savings) + +```bash +aws ec2 request-spot-fleet --spot-fleet-request-config '{ + "IamFleetRole": "arn:aws:iam::xxx:role/fleet-role", + "TargetCapacity": 2, + "SpotPrice": "0.05", + "LaunchSpecifications": [{ + "ImageId": "ami-xxx", + "InstanceType": "t3.medium", + "UserData": "BASE64_ENCODED_STARTUP_SCRIPT" + }] +}' +``` + +### 2. S3 Intelligent-Tiering + +```javascript +// Brainy automatically uses S3 Intelligent-Tiering +const brain = new Brainy({ + storage: { + type: 's3', + options: { + storageClass: 'INTELLIGENT_TIERING' // Automatic cost optimization + } + } +}) +``` + +### 3. Lambda Reserved Concurrency + +```bash +aws lambda put-function-concurrency \ + --function-name brainy-handler \ + --reserved-concurrent-executions 10 +``` + +## Deployment Automation + +### GitHub Actions CI/CD + +```yaml +name: Deploy to AWS +on: + push: + branches: [main] + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + + - name: Configure AWS credentials + uses: aws-actions/configure-aws-credentials@v1 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + + - name: Deploy to ECS + run: | + docker build -t brainy . + docker tag brainy:latest $ECR_URI/brainy:latest + docker push $ECR_URI/brainy:latest + aws ecs update-service --cluster default --service brainy --force-new-deployment +``` + +## Troubleshooting + +### Common Issues + +1. **Storage Auto-Detection Fails** + ```javascript + // Explicitly specify storage + const brain = new Brainy({ + storage: { type: 's3', options: { bucket: 'my-bucket' } } + }) + ``` + +2. **Memory Issues** + ```javascript + // Optimize for low memory + const brain = new Brainy({ + cache: { maxSize: 100 }, // Reduce cache size + index: { M: 8 } // Reduce HNSW connections + }) + ``` + +3. **Cold Starts (Lambda)** + ```javascript + // Warm-up configuration + exports.warmup = async () => { + if (!brain) { + brain = new Brainy({ warmup: true }) + await brain.init() + } + } + ``` + +## Production Checklist + +- [ ] IAM roles configured with minimal permissions +- [ ] VPC with private subnets +- [ ] Auto-scaling configured +- [ ] CloudWatch alarms set up +- [ ] Backup strategy (S3 versioning enabled) +- [ ] SSL/TLS certificates configured +- [ ] Rate limiting enabled +- [ ] Health checks configured +- [ ] Monitoring dashboard created +- [ ] Cost alerts configured + +## Support + +- Documentation: https://brainy.soulcraft.ai/docs +- Issues: https://github.com/soulcraft/brainy/issues +- Community: https://discord.gg/brainy \ No newline at end of file diff --git a/docs/deployment/gcp-deployment.md b/docs/deployment/gcp-deployment.md new file mode 100644 index 00000000..7e812eff --- /dev/null +++ b/docs/deployment/gcp-deployment.md @@ -0,0 +1,547 @@ +# Google Cloud Platform Deployment Guide for Brainy + +## Overview +Deploy Brainy on GCP with automatic scaling, global distribution, and zero-config dynamic adaptation. Brainy automatically detects and optimizes for GCP services. + +## Quick Start (Zero-Config) + +### Option 1: Cloud Run (Serverless Containers) + +```bash +# Build and deploy with one command +gcloud run deploy brainy \ + --source . \ + --platform managed \ + --region us-central1 \ + --allow-unauthenticated + +# Brainy auto-detects Cloud Run and configures: +# - Memory-optimized caching +# - GCS for storage (if available) +# - Cloud SQL for metadata (if available) +``` + +### Option 2: Cloud Functions (Serverless) + +```javascript +// index.js +const { Brainy } = require('@soulcraft/brainy') + +let brain + +exports.brainyHandler = async (req, res) => { + // Zero-config - auto-adapts to Cloud Functions + if (!brain) { + brain = new Brainy() // Detects GCP environment automatically + await brain.init() + } + + const { method, ...params } = req.body + + try { + let result + switch(method) { + case 'add': + result = await brain.add(params) + break + case 'find': + result = await brain.find(params) + break + case 'relate': + result = await brain.relate(params) + break + default: + return res.status(400).json({ error: 'Unknown method' }) + } + res.json({ result }) + } catch (error) { + res.status(500).json({ error: error.message }) + } +} +``` + +Deploy: +```bash +gcloud functions deploy brainy \ + --runtime nodejs20 \ + --trigger-http \ + --entry-point brainyHandler \ + --memory 512MB \ + --timeout 60s +``` + +### Option 3: Google Kubernetes Engine (GKE) + +```bash +# Create autopilot cluster (fully managed, zero-config) +gcloud container clusters create-auto brainy-cluster \ + --region us-central1 + +# Deploy using Cloud Build +gcloud builds submit --tag gcr.io/$PROJECT_ID/brainy + +# Apply Kubernetes manifest +kubectl apply -f - < { + // Send custom metrics to Cloud Monitoring + await monitoring.createTimeSeries({ + name: monitoring.projectPath(projectId), + timeSeries: [{ + metric: { + type: `custom.googleapis.com/brainy/${metric.name}`, + labels: metric.labels + }, + points: [{ + interval: { endTime: { seconds: Date.now() / 1000 } }, + value: { doubleValue: metric.value } + }] + }] + }) + } +}) +``` + +### Cloud Trace Integration + +```javascript +const brain = new Brainy({ + tracing: { + enabled: true, + sampleRate: 0.1 // Sample 10% of requests + } +}) +``` + +## Security Best Practices + +### 1. Workload Identity (GKE) + +```yaml +# Enable Workload Identity +apiVersion: v1 +kind: ServiceAccount +metadata: + name: brainy-sa + annotations: + iam.gke.io/gcp-service-account: brainy@PROJECT_ID.iam.gserviceaccount.com +``` + +### 2. Binary Authorization + +```yaml +# Ensure only signed container images +apiVersion: binaryauthorization.grafeas.io/v1beta1 +kind: Policy +metadata: + name: brainy-policy +spec: + defaultAdmissionRule: + requireAttestationsBy: + - projects/PROJECT_ID/attestors/prod-attestor +``` + +### 3. VPC Service Controls + +```bash +# Create VPC Service Perimeter +gcloud access-context-manager perimeters create brainy_perimeter \ + --resources=projects/PROJECT_NUMBER \ + --restricted-services=storage.googleapis.com \ + --title="Brainy Security Perimeter" +``` + +## Cost Optimization + +### 1. Preemptible VMs (80% savings) + +```yaml +# GKE node pool with preemptible VMs +apiVersion: container.cnrm.cloud.google.com/v1beta1 +kind: ContainerNodePool +metadata: + name: brainy-preemptible-pool +spec: + clusterRef: + name: brainy-cluster + config: + preemptible: true + machineType: n2-standard-2 + autoscaling: + minNodeCount: 1 + maxNodeCount: 10 +``` + +### 2. Cloud CDN for Static Assets + +```bash +# Enable Cloud CDN for frequently accessed data +gcloud compute backend-buckets create brainy-assets \ + --gcs-bucket-name=brainy-static + +gcloud compute backend-buckets update brainy-assets \ + --enable-cdn \ + --cache-mode=CACHE_ALL_STATIC +``` + +### 3. Committed Use Discounts + +```bash +# Purchase committed use for predictable workloads +gcloud compute commitments create brainy-commitment \ + --plan=TWELVE_MONTH \ + --resources=vcpu=100,memory=400 +``` + +## Deployment Automation + +### Cloud Build CI/CD + +```yaml +# cloudbuild.yaml +steps: + # Build container + - name: 'gcr.io/cloud-builders/docker' + args: ['build', '-t', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA', '.'] + + # Push to registry + - name: 'gcr.io/cloud-builders/docker' + args: ['push', 'gcr.io/$PROJECT_ID/brainy:$SHORT_SHA'] + + # Deploy to Cloud Run + - name: 'gcr.io/cloud-builders/gcloud' + args: + - 'run' + - 'deploy' + - 'brainy' + - '--image=gcr.io/$PROJECT_ID/brainy:$SHORT_SHA' + - '--region=us-central1' + - '--platform=managed' + +# Trigger on push to main +trigger: + branch: + name: main +``` + +### Terraform Infrastructure + +```hcl +# main.tf +resource "google_cloud_run_service" "brainy" { + name = "brainy" + location = "us-central1" + + template { + spec { + containers { + image = "gcr.io/${var.project_id}/brainy" + + resources { + limits = { + cpu = "2" + memory = "2Gi" + } + } + + env { + name = "NODE_ENV" + value = "production" + } + } + } + } + + traffic { + percent = 100 + latest_revision = true + } +} + +resource "google_cloud_run_service_iam_member" "public" { + service = google_cloud_run_service.brainy.name + location = google_cloud_run_service.brainy.location + role = "roles/run.invoker" + member = "allUsers" +} +``` + +## Performance Optimization + +### 1. Memory Store (Redis Compatible) + +```javascript +// Brainy can use Memorystore for caching +const brain = new Brainy({ + cache: { + type: 'redis', + options: { + host: process.env.REDIS_HOST || 'auto-detect', + port: 6379 + } + } +}) +``` + +### 2. Cloud Spanner for Global Consistency + +```javascript +const brain = new Brainy({ + metadata: { + type: 'spanner', + options: { + instance: 'brainy-instance', + database: 'brainy-db' + } + } +}) +``` + +## Troubleshooting + +### Common Issues + +1. **Quota Exceeded** + ```bash + # Check quotas + gcloud compute project-info describe --project=$PROJECT_ID + + # Request increase + gcloud compute project-info add-metadata \ + --metadata google-compute-default-region=us-central1 + ``` + +2. **Cold Starts** + ```javascript + // Keep minimum instances warm + const brain = new Brainy({ + warmup: { + enabled: true, + interval: 60000 // Ping every minute + } + }) + ``` + +3. **Memory Pressure** + ```javascript + // Optimize for GCP memory constraints + const brain = new Brainy({ + memory: { + mode: 'aggressive', // Aggressive garbage collection + maxHeap: 0.8 // Use 80% of available memory + } + }) + ``` + +## Production Checklist + +- [ ] Enable Workload Identity for secure access +- [ ] Configure Cloud Armor for DDoS protection +- [ ] Set up Cloud KMS for encryption keys +- [ ] Enable VPC Service Controls +- [ ] Configure Cloud IAP for authentication +- [ ] Set up Cloud Monitoring dashboards +- [ ] Configure Error Reporting +- [ ] Enable Cloud Trace +- [ ] Set up budget alerts +- [ ] Configure backup and disaster recovery + +## Support + +- Documentation: https://brainy.soulcraft.ai/docs +- Issues: https://github.com/soulcraft/brainy/issues +- GCP Marketplace: https://console.cloud.google.com/marketplace/product/brainy \ No newline at end of file diff --git a/docs/deployment/kubernetes-deployment.md b/docs/deployment/kubernetes-deployment.md new file mode 100644 index 00000000..66e076af --- /dev/null +++ b/docs/deployment/kubernetes-deployment.md @@ -0,0 +1,727 @@ +# Kubernetes Deployment Guide for Brainy + +## Overview +Deploy Brainy on Kubernetes with automatic scaling, high availability, and zero-config dynamic adaptation. Works with any Kubernetes distribution (vanilla, EKS, GKE, AKS, OpenShift, etc.). + +## Quick Start (Zero-Config) + +### Basic Deployment + +```yaml +# brainy-deployment.yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: brainy + labels: + app: brainy +spec: + replicas: 3 + selector: + matchLabels: + app: brainy + template: + metadata: + labels: + app: brainy + spec: + containers: + - name: brainy + image: soulcraft/brainy:latest + ports: + - containerPort: 3000 + env: + - name: NODE_ENV + value: production + # Brainy auto-detects Kubernetes and configures accordingly + resources: + requests: + memory: "256Mi" + cpu: "100m" + limits: + memory: "1Gi" + cpu: "500m" +--- +apiVersion: v1 +kind: Service +metadata: + name: brainy-service +spec: + selector: + app: brainy + ports: + - protocol: TCP + port: 80 + targetPort: 3000 + type: LoadBalancer +``` + +Deploy: +```bash +kubectl apply -f brainy-deployment.yaml +``` + +## Production-Grade Setup + +### 1. StatefulSet with Persistent Storage + +```yaml +apiVersion: v1 +kind: StorageClass +metadata: + name: brainy-storage +provisioner: kubernetes.io/aws-ebs # Or your cloud provider +parameters: + type: gp3 + iopsPerGB: "10" +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: brainy +spec: + serviceName: brainy-headless + replicas: 3 + selector: + matchLabels: + app: brainy + template: + metadata: + labels: + app: brainy + spec: + containers: + - name: brainy + image: soulcraft/brainy:latest + ports: + - containerPort: 3000 + env: + - name: NODE_ENV + value: production + - name: BRAINY_STORAGE_TYPE + value: filesystem + - name: BRAINY_STORAGE_PATH + value: /data + volumeMounts: + - name: data + mountPath: /data + livenessProbe: + httpGet: + path: /health + port: 3000 + initialDelaySeconds: 30 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /ready + port: 3000 + initialDelaySeconds: 5 + periodSeconds: 5 + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + storageClassName: brainy-storage + resources: + requests: + storage: 10Gi +``` + +### 2. Horizontal Pod Autoscaler + +```yaml +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: brainy-hpa +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: brainy + minReplicas: 2 + maxReplicas: 100 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 + - type: Resource + resource: + name: memory + target: + type: Utilization + averageUtilization: 80 + behavior: + scaleUp: + stabilizationWindowSeconds: 60 + policies: + - type: Percent + value: 100 + periodSeconds: 60 + scaleDown: + stabilizationWindowSeconds: 300 + policies: + - type: Percent + value: 10 + periodSeconds: 60 +``` + +### 3. Ingress Configuration + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: brainy-ingress + annotations: + kubernetes.io/ingress.class: nginx + cert-manager.io/cluster-issuer: letsencrypt-prod + nginx.ingress.kubernetes.io/rate-limit: "100" + nginx.ingress.kubernetes.io/proxy-body-size: "50m" +spec: + tls: + - hosts: + - api.brainy.example.com + secretName: brainy-tls + rules: + - host: api.brainy.example.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: brainy-service + port: + number: 80 +``` + +## Zero-Config Storage Options + +### Option 1: S3-Compatible Storage (Recommended) + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: brainy-s3-credentials +type: Opaque +data: + access-key: + secret-key: +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: brainy +spec: + template: + spec: + containers: + - name: brainy + env: + - name: BRAINY_STORAGE_TYPE + value: s3 + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: brainy-s3-credentials + key: access-key + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: brainy-s3-credentials + key: secret-key + - name: S3_BUCKET + value: brainy-data + - name: AWS_REGION + value: us-east-1 +``` + +### Option 2: MinIO (Self-Hosted S3) + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: minio +spec: + replicas: 1 + selector: + matchLabels: + app: minio + template: + metadata: + labels: + app: minio + spec: + containers: + - name: minio + image: minio/minio:latest + args: + - server + - /data + env: + - name: MINIO_ROOT_USER + value: brainy + - name: MINIO_ROOT_PASSWORD + value: brainy123456 + volumeMounts: + - name: data + mountPath: /data + volumes: + - name: data + persistentVolumeClaim: + claimName: minio-pvc +--- +apiVersion: v1 +kind: Service +metadata: + name: minio-service +spec: + selector: + app: minio + ports: + - port: 9000 + targetPort: 9000 +``` + +### Option 3: Shared NFS Storage + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: brainy-nfs-pv +spec: + capacity: + storage: 100Gi + accessModes: + - ReadWriteMany + nfs: + server: nfs-server.example.com + path: /export/brainy +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: brainy-nfs-pvc +spec: + accessModes: + - ReadWriteMany + resources: + requests: + storage: 100Gi +``` + +## High Availability Configuration + +### 1. Pod Anti-Affinity + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: brainy +spec: + template: + spec: + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app + operator: In + values: + - brainy + topologyKey: kubernetes.io/hostname +``` + +### 2. Pod Disruption Budget + +```yaml +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: brainy-pdb +spec: + minAvailable: 2 + selector: + matchLabels: + app: brainy +``` + +### 3. Multi-Zone Deployment + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: brainy +spec: + template: + spec: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + app: brainy +``` + +## Monitoring & Observability + +### 1. Prometheus Metrics + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: brainy-metrics + annotations: + prometheus.io/scrape: "true" + prometheus.io/port: "9090" + prometheus.io/path: "/metrics" +spec: + selector: + app: brainy + ports: + - name: metrics + port: 9090 + targetPort: 9090 +``` + +### 2. Grafana Dashboard + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: brainy-dashboard +data: + dashboard.json: | + { + "dashboard": { + "title": "Brainy Metrics", + "panels": [ + { + "title": "Request Rate", + "targets": [ + { + "expr": "rate(brainy_requests_total[5m])" + } + ] + }, + { + "title": "Response Time", + "targets": [ + { + "expr": "histogram_quantile(0.95, brainy_response_time)" + } + ] + } + ] + } + } +``` + +### 3. Logging with Fluentd + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: fluentd-config +data: + fluent.conf: | + + @type tail + path /var/log/containers/brainy*.log + pos_file /var/log/fluentd-brainy.log.pos + tag brainy.* + + @type json + + + + + @type elasticsearch + host elasticsearch.logging.svc.cluster.local + port 9200 + logstash_format true + logstash_prefix brainy + +``` + +## Security Best Practices + +### 1. Network Policies + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: brainy-netpol +spec: + podSelector: + matchLabels: + app: brainy + policyTypes: + - Ingress + - Egress + ingress: + - from: + - namespaceSelector: + matchLabels: + name: ingress-nginx + ports: + - protocol: TCP + port: 3000 + egress: + - to: + - namespaceSelector: {} + ports: + - protocol: TCP + port: 443 # HTTPS + - protocol: TCP + port: 9000 # MinIO/S3 +``` + +### 2. Pod Security Policy + +```yaml +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: brainy-psp +spec: + privileged: false + allowPrivilegeEscalation: false + requiredDropCapabilities: + - ALL + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'persistentVolumeClaim' + runAsUser: + rule: 'MustRunAsNonRoot' + seLinux: + rule: 'RunAsAny' + fsGroup: + rule: 'RunAsAny' +``` + +### 3. RBAC Configuration + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: brainy-sa +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: brainy-role +rules: +- apiGroups: [""] + resources: ["configmaps"] + verbs: ["get", "list", "watch"] +- apiGroups: [""] + resources: ["secrets"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: brainy-rolebinding +subjects: +- kind: ServiceAccount + name: brainy-sa +roleRef: + kind: Role + name: brainy-role + apiGroup: rbac.authorization.k8s.io +``` + +## GitOps with ArgoCD + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: brainy + namespace: argocd +spec: + project: default + source: + repoURL: https://github.com/yourorg/brainy-k8s + targetRevision: HEAD + path: manifests + destination: + server: https://kubernetes.default.svc + namespace: brainy + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=true +``` + +## Helm Chart Installation + +```bash +# Add Brainy Helm repository +helm repo add brainy https://charts.brainy.io +helm repo update + +# Install with custom values +cat > values.yaml << EOF +replicaCount: 3 +image: + repository: soulcraft/brainy + tag: latest + pullPolicy: IfNotPresent + +service: + type: LoadBalancer + port: 80 + +ingress: + enabled: true + className: nginx + hosts: + - host: api.brainy.example.com + paths: + - path: / + pathType: Prefix + +resources: + limits: + cpu: 1000m + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi + +autoscaling: + enabled: true + minReplicas: 2 + maxReplicas: 100 + targetCPUUtilizationPercentage: 70 + +storage: + type: s3 + s3: + bucket: brainy-data + region: us-east-1 +EOF + +helm install brainy brainy/brainy -f values.yaml +``` + +## Cost Optimization + +### 1. Spot/Preemptible Nodes + +```yaml +apiVersion: v1 +kind: NodePool +metadata: + name: brainy-spot-pool +spec: + nodeSelector: + node.kubernetes.io/lifecycle: spot + taints: + - key: spot + value: "true" + effect: NoSchedule + tolerations: + - key: spot + operator: Equal + value: "true" + effect: NoSchedule +``` + +### 2. Vertical Pod Autoscaler + +```yaml +apiVersion: autoscaling.k8s.io/v1 +kind: VerticalPodAutoscaler +metadata: + name: brainy-vpa +spec: + targetRef: + apiVersion: apps/v1 + kind: Deployment + name: brainy + updatePolicy: + updateMode: "Auto" + resourcePolicy: + containerPolicies: + - containerName: brainy + minAllowed: + cpu: 100m + memory: 128Mi + maxAllowed: + cpu: 2 + memory: 2Gi +``` + +## Troubleshooting + +### Common Issues + +1. **Pod CrashLoopBackOff** + ```bash + kubectl logs -f pod/brainy-xxx + kubectl describe pod brainy-xxx + ``` + +2. **Storage Issues** + ```bash + kubectl get pv,pvc + kubectl describe pvc brainy-data + ``` + +3. **Network Connectivity** + ```bash + kubectl exec -it pod/brainy-xxx -- curl http://brainy-service/health + kubectl get endpoints brainy-service + ``` + +4. **Memory Pressure** + ```bash + kubectl top pods -l app=brainy + kubectl describe node + ``` + +## Production Checklist + +- [ ] High availability with multiple replicas +- [ ] Pod disruption budgets configured +- [ ] Resource limits and requests set +- [ ] Horizontal and vertical autoscaling enabled +- [ ] Persistent storage configured +- [ ] Network policies in place +- [ ] RBAC properly configured +- [ ] Monitoring and alerting setup +- [ ] Backup and disaster recovery plan +- [ ] Security scanning enabled +- [ ] GitOps deployment pipeline + +## Support + +- Documentation: https://brainy.soulcraft.ai/docs +- Helm Charts: https://github.com/soulcraft/brainy-charts +- Issues: https://github.com/soulcraft/brainy/issues +- Slack: https://brainy-community.slack.com \ No newline at end of file diff --git a/docs/eli5.md b/docs/eli5.md deleted file mode 100644 index e0bb9a19..00000000 --- a/docs/eli5.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -title: What is Brainy? -slug: getting-started/what-is-brainy -public: true -category: getting-started -template: guide -order: 0 -description: Plain-language guide covering what Brainy does, how it compares to other tools, and what you can build with it. No jargon, no code — just clear analogies. -next: - - getting-started/installation - - getting-started/quick-start ---- - -# Brainy and Cor — Explained Simply - -*A plain-language guide for anyone who wants to understand what this thing actually does.* - ---- - -## What is Brainy? - -Imagine you have the world's smartest librarian. - -You walk up and say *"I'm looking for something about climate change — but only books published after 2020, and only ones written by authors I've already read."* A normal library would make you dig through a card catalogue, then cross-reference a list of authors, then scan the shelves yourself. That takes a while. - -Your smart librarian does all three at the same time — in less than the time it takes to blink. - -That's Brainy. It's a knowledge database that can search by **meaning**, follow **connections**, and filter by **labels** — all at once, in a single question. - ---- - -## The Three Superpowers - -### 1. Meaning Search (the "fuzzy" superpower) - -When you search for "automobile," Brainy also finds results about "car," "vehicle," and "sedan" — because it understands what words *mean*, not just how they're spelled. It reads your data the way a person would, not the way a search box does. - -Think of it like the librarian who finds books on "heartbreak" when you ask for something about "loneliness." - -### 2. Relationship Walking (the "follow the thread" superpower) - -Every piece of information can be connected to other pieces. A Person *works at* a Company. A Project *depends on* a Tool. A Recipe *contains* Ingredients. - -Brainy can follow these connections across many hops in one step. Ask for "everything connected to this author, two steps out" and Brainy returns the author's books, the books' publishers, the publishers' other authors — without you needing to chain four separate lookups yourself. - -Think of it like the librarian who not only hands you the book you asked for, but also knows which shelf it came from, who donated it, and what other books arrived in the same donation. - -### 3. Label Filtering (the "narrow it down" superpower) - -Sometimes meaning and connections aren't enough — you need precision. "Only recipes with fewer than 500 calories." "Only events from last week." "Only documents tagged as urgent." - -Brainy can narrow any result set down by exact labels or ranges in the same breath as the other two searches. No extra steps. - ---- - -## What Else Can It Do? - -- **Virtual file cabinet.** Brainy includes a full filesystem you can use to store, organize, and semantically search files — PDFs, documents, anything — the same way you search everything else. - -- **Live dashboards.** You can define running totals that Brainy keeps updated automatically — things like "total sales this month by region" or "average response time per service." Every time new data comes in, the numbers stay current with no manual recalculation. - -- **Time travel.** Every committed change becomes part of the database's history. You can pin the current state as a frozen view, see the whole knowledge base exactly as it was last week, try out changes in a scratch copy that never touches the real data, and take instant backups. - -- **Universal vocabulary.** Brainy ships with a shared language of 42 kinds of things (Person, Document, Task, Concept, Event…) and 127 kinds of connections (Contains, DependsOn, Creates, RelatedTo…). This means data from different sources speaks the same language without you having to translate. - ---- - -## What is Cor? - -Cor is a turbocharger for Brainy. - -Same car. Same controls. Same fuel. You just swap in a faster engine under the hood, and everything that used to take a moment now happens instantly. - -Technically, Cor is an optional plugin written in Rust — a lower-level language that runs much closer to the raw metal of your processor. It plugs into Brainy and takes over the most compute-intensive work: the distance calculations that power meaning search, the number-crunching behind live aggregates, and the set operations that drive label filtering. - -You install it with one line, register it with one call, and Brainy automatically uses it everywhere it can help. - ---- - -## How Much Faster? - -Plain language: - -- **Searches** go from "the blink of an eye" to "faster than a blink." The overall speedup is **5.2× on average** across all operations. -- **Live aggregates** are rebuilt using all CPU cores in parallel, so re-indexing large datasets takes a fraction of the time. -- **Analytics** that aren't even possible in pure JavaScript — real-time anomaly detection, streaming percentile estimates, approximate unique counts — become available because Cor brings the native capabilities required to run them efficiently. - -If Brainy is what makes knowledge fast, Cor is what makes Brainy feel instant. - ---- - -## What Does Brainy Replace? - -Most applications that need to store and search knowledge end up stitching together several specialized tools. Brainy replaces all of them with one — a single free, open-source library in place of multiple paid services. - -### Before and After - -**Before Brainy** — a pile of services: -- Pinecone (vectors) + Neo4j (graph) + MongoDB (docs) -- Algolia (search) + Redis (cache) + PostgreSQL + pgvector -- Plus glue code, sync jobs, ETL pipelines, and 3am incidents - -**After Brainy** — one thing: -Search, graph, filter, files, time travel, and imports — unified in a single library. - -### What Each Tool Is Missing - -| Tool | Search | Graph | Filter | VFS | Time travel | Import | -|---|:---:|:---:|:---:|:---:|:---:|:---:| -| **Brainy** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| *— Vector databases —* | | | | | | | -| Pinecone | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | -| Weaviate | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | -| Qdrant | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | -| Chroma | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | -| *— Graph databases —* | | | | | | | -| Neo4j | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | -| *— Document stores —* | | | | | | | -| MongoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | -| Firestore | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | -| DynamoDB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | -| *— Relational + vector —* | | | | | | | -| PostgreSQL + pgvector | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | -| MySQL | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | -| SQLite | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | -| *— Search engines —* | | | | | | | -| Elasticsearch | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | -| Algolia | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | -| *— Cache —* | | | | | | | -| Redis | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | - -Brainy is the only row with every box checked. And it runs all of them in a single query — no stitching services together. - -### One library, any scale - -Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way. - -Add Cor and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows. - ---- - -## What Can You Build? - -### Common applications - -- **AI agents with persistent memory** — Give any AI an always-on, self-organizing knowledge graph that persists between sessions and across agents. -- **Searchable knowledge bases** — Build institutional memory that links documents automatically and surfaces answers across the full web of related information. -- **Semantic document search** — Index PDFs, code, or media and find them by meaning, not just keywords. -- **Relationship-aware recommendations** — Power product catalogs or content platforms where every recommendation understands what connects to what. -- **Safe experiments** — Test risky changes against a scratch copy of the knowledge base, audit exactly what changed and when, and roll back to any snapshot instantly. -- **Unified business platforms** — Combine booking, CRM, inventory, and analytics in one queryable knowledge graph with no sync pipeline. - -### What Brainy is good at - -Brainy is the engine underneath production systems that need to combine semantic search, structured filtering, and graph traversal in a single query — agent memory, knowledge-base platforms, business operations consoles, multi-agent coordination, and more. The combination of vector + graph + metadata search in one indexed call is what differentiates it from running three engines side by side. diff --git a/docs/features/complete-feature-list.md b/docs/features/complete-feature-list.md new file mode 100644 index 00000000..df02a3f7 --- /dev/null +++ b/docs/features/complete-feature-list.md @@ -0,0 +1,413 @@ +# 🚀 Brainy 2.0 - Complete Feature List + +> **The Truth**: Brainy is MORE powerful than previously documented! This is the complete list of ALL implemented features. + +## 🧠 Core Intelligence Engine + +### Triple Intelligence System ✅ +Unified query system that automatically combines: +- **Vector Search**: HNSW-indexed semantic similarity (O(log n) performance) +- **Graph Traversal**: Relationship-based discovery +- **Field Filtering**: Metadata and attribute queries +- **Auto-optimization**: Queries are automatically optimized based on data patterns + +```typescript +// All three intelligences work together automatically +const results = await brain.find({ + like: 'AI research', // Vector search + where: { year: 2024 }, // Metadata filtering + connected: { to: authorId } // Graph traversal +}) +``` + +### Neural Query Understanding ✅ +- **220+ embedded patterns** for query intent detection +- Natural language query processing +- Automatic query type detection +- Query rewriting and optimization + +## 🔧 12+ Production Augmentations + +```typescript +// Full crash recovery, checkpointing, replay +``` + +### 2. Entity Registry ✅ +```typescript +import { EntityRegistryAugmentation } from 'brainy' +// Bloom filter-based deduplication for streaming data +// Handles millions of entities with minimal memory +``` + +### 3. Auto-Register Entities ✅ +```typescript +import { AutoRegisterEntitiesAugmentation } from 'brainy' +// Automatically extracts and registers entities from text +``` + +### 4. Intelligent Verb Scoring ✅ +```typescript +import { IntelligentVerbScoringAugmentation } from 'brainy' +// Multi-factor relationship strength: +// - Semantic similarity +// - Temporal decay +// - Frequency amplification +// - Context awareness +``` + +### 5. Batch Processing ✅ +```typescript +import { BatchProcessingAugmentation } from 'brainy' +// Adaptive batching with backpressure +// Dynamically adjusts batch size based on load +``` + +### 6. Connection Pool ✅ +```typescript +import { ConnectionPoolAugmentation } from 'brainy' +// Auto-scaling connection management +// Optimized for distributed operations +``` + +### 7. Request Deduplicator ✅ +```typescript +import { RequestDeduplicatorAugmentation } from 'brainy' +// In-flight request deduplication +// 3x performance boost for concurrent operations +``` + +### 8. WebSocket Conduit ✅ +```typescript +import { WebSocketConduitAugmentation } from 'brainy' +// Real-time bidirectional streaming +// Auto-reconnection and heartbeat +``` + +### 9. WebRTC Conduit ✅ +```typescript +import { WebRTCConduitAugmentation } from 'brainy' +// Peer-to-peer data channels +// Direct browser-to-browser communication +``` + +### 10. Memory Storage Optimization ✅ +```typescript +import { MemoryStorageAugmentation } from 'brainy' +// Memory-specific optimizations +// Circular buffers, compression +``` + +### 11. Server Search Conduit ✅ +```typescript +import { ServerSearchConduitAugmentation } from 'brainy' +// Distributed query execution +// Load balancing across nodes +``` + +### 12. Neural Import ✅ +```typescript +import { NeuralImportAugmentation } from 'brainy' +// AI-powered data understanding +// Automatic entity detection and classification +// Relationship discovery +``` + +## 🤖 Neural Import Capabilities (FULLY IMPLEMENTED!) + +```typescript +const neuralImport = new NeuralImport(brain) + +// ALL of these work TODAY: +await neuralImport.neuralImport('data.csv') +await neuralImport.detectEntitiesWithNeuralAnalysis(data) +await neuralImport.detectNounType(entity) +await neuralImport.detectRelationships(entities) +await neuralImport.generateInsights(data) +``` + +### Features: +- **Auto-detects file format** (CSV, JSON, XML, etc.) +- **Identifies entity types** using AI +- **Discovers relationships** between entities +- **Generates insights** about the data +- **Creates optimal graph structure** automatically + +## 🎯 Zero-Config Model Loading Cascade + +Brainy automatically loads models with ZERO configuration required: + +```typescript +const brain = new Brainy() // That's it! +await brain.init() +// Models load automatically from best available source +``` + +### Loading Priority: +1. **Local Cache** (./models) - Instant, no network +2. **CDN** (models.soulcraft.com) - Fast, global [Coming Soon] +3. **GitHub Releases** - Reliable backup +4. **HuggingFace** - Ultimate fallback + +### Key Features: +- **Automatic fallback** if sources fail +- **Model verification** with checksums +- **Offline support** with bundled models +- **No environment variables needed** +- **Works in all environments** (Node, Browser, Workers) + +## 🏢 Distributed Operation Modes + +### Reader Mode ✅ +```typescript +const brain = new Brainy({ mode: 'reader' }) +// Optimized for read-heavy workloads +// 80% cache ratio, aggressive prefetch +// 1 hour TTL, minimal writes +``` + +### Writer Mode ✅ +```typescript +const brain = new Brainy({ mode: 'writer' }) +// Optimized for write-heavy workloads +// Large write buffers, batch writes +// Minimal caching, fast ingestion +``` + +### Hybrid Mode ✅ +```typescript +const brain = new Brainy({ mode: 'hybrid' }) +// Balanced for mixed workloads +// Adaptive caching and batching +``` + +## 💾 Advanced Caching System + +### 3-Level Cache Architecture ✅ +```typescript +const cacheConfig = { + hotCache: { + size: 1000, // L1 - RAM + ttl: 60000 // 1 minute + }, + warmCache: { + size: 10000, // L2 - Fast storage + ttl: 300000 // 5 minutes + }, + coldCache: { + size: 100000, // L3 - Persistent + ttl: null // No expiry + } +} +``` + +### Cache Features: +- **Automatic promotion/demotion** between levels +- **LRU eviction** within each level +- **Compression** for cold cache +- **Statistics tracking** for optimization + +## 📊 Comprehensive Statistics + +```typescript +const stats = await brain.getStats() +// Returns detailed metrics: +{ + nouns: { + count, created, updated, deleted, + size, avgSize + }, + verbs: { + count, created, types, + weights: { min, max, avg } + }, + vectors: { + dimensions: 384, + indexSize, partitions, + avgSearchTime + }, + cache: { + hits, misses, evictions, + hitRate, sizes + }, + performance: { + operations, avgTimes, + p95Latency, p99Latency + }, + storage: { + used, available, + compression, files + }, + throttling: { + delays, rateLimited, + backoffMs, retries + } +} +``` + +## 🚀 GPU Acceleration Support + +```typescript +// Automatic GPU detection +const device = await detectBestDevice() +// Returns: 'cpu' | 'webgpu' | 'cuda' + +// WebGPU in browser (when available) +if (device === 'webgpu') { + // Transformer models use WebGPU automatically +} + +// CUDA in Node.js (requires ONNX Runtime GPU) +if (device === 'cuda') { + // Automatically uses GPU for embeddings +} +``` + +## 🔄 Adaptive Systems + +### Adaptive Backpressure ✅ +```typescript +// Automatically adjusts flow based on system load +// Prevents OOM and maintains throughput +``` + +### Adaptive Socket Manager ✅ +```typescript +// Dynamic connection pooling +// Scales connections based on traffic patterns +``` + +### Cache Auto-Configuration ✅ +```typescript +// Sizes cache based on available memory +// Adjusts strategies based on usage patterns +``` + +### S3 Throttling Protection ✅ +```typescript +// Built-in exponential backoff +// Rate limit detection and adaptation +// Automatic retry with jitter +``` + +## 🛠️ Storage Adapters + +All included, auto-selected based on environment: + +### FileSystem Storage ✅ +- Default for Node.js +- Efficient file-based storage +- Automatic directory management + +### Memory Storage ✅ +- Ultra-fast in-memory operations +- Perfect for testing and temporary data +- Circular buffer support + +### OPFS Storage ✅ +- Browser persistent storage +- Survives page refreshes +- Quota management + +### S3 Storage ✅ +- AWS S3 compatible +- Automatic multipart uploads +- Throttling protection +- Batch operations + +## 🎨 Natural Language Processing + +### Built-in Patterns (220+) +- Question types (what, why, how, when, where) +- Temporal queries (yesterday, last week, 2024) +- Comparative queries (better than, similar to) +- Aggregations (count, sum, average) +- Filters (only, except, without) +- Relationships (related to, connected with) + +### Coverage: 94-98% of typical queries! + +## 🔐 Security Features + +### Built-in Security ✅ +- Automatic input sanitization +- SQL injection prevention +- XSS protection for web contexts +- Rate limiting support + +### Encryption Ready ✅ +```typescript +import { crypto } from 'brainy/utils' +// AES-256-GCM encryption utilities +// Key derivation functions +// Secure random generation +``` + +## 🎯 Key Design Principles + +### 1. Zero Configuration +```typescript +const brain = new Brainy() +await brain.init() +// Everything else is automatic! +``` + +### 2. Fixed Dimensions (384) +- **ALWAYS** uses all-MiniLM-L6-v2 model +- **ALWAYS** 384 dimensions +- **NOT** configurable (by design) +- Ensures everything works together + +### 3. Progressive Enhancement +- Starts simple, scales automatically +- Adapts to workload patterns +- Optimizes based on usage + +### 4. Universal Compatibility +- Works in Node.js 18+ +- Works in modern browsers +- Works in Web Workers +- Works in Edge environments + +## 📦 What Ships in Core (MIT Licensed) + +**EVERYTHING** is included in the core package: +- ✅ All engines (vector, graph, field, neural) +- ✅ All augmentations (12+) +- ✅ All storage adapters +- ✅ All distributed modes +- ✅ Complete statistics +- ✅ GPU support +- ✅ No feature limitations +- ✅ No premium tiers +- ✅ 100% MIT licensed + +## 🚀 Quick Start + +```typescript +import { Brainy } from 'brainy' + +// Zero config required! +const brain = new Brainy() +await brain.init() + +// Add data (auto-detects type) +await brain.add('Content here') + +// Search with natural language +const results = await brain.find('related content from last week') + +// Everything else is automatic! +``` + +## 📈 Performance Characteristics + +- **Vector Search**: O(log n) with HNSW indexing +- **Graph Traversal**: O(k) for k-hop queries +- **Field Filtering**: O(1) with metadata index +- **Memory Usage**: ~100MB base + data +- **Embedding Speed**: ~100ms for batch of 10 +- **Query Speed**: <10ms for most queries + +## 🎉 Summary + +Brainy 2.0 is a **complete**, **production-ready** AI database that requires **ZERO configuration**. Every feature listed here is **implemented and working** today. No configuration, no setup, no complexity - just powerful AI capabilities that work out of the box! \ No newline at end of file diff --git a/docs/features/v3-features.md b/docs/features/v3-features.md new file mode 100644 index 00000000..f8bc9c96 --- /dev/null +++ b/docs/features/v3-features.md @@ -0,0 +1,296 @@ +# 🚀 Brainy - Production-Ready Features + +> **Status**: All features listed here are IMPLEMENTED and TESTED + +## 📊 Performance Metrics +- **Search Latency**: <10ms for 10,000+ items +- **Write Throughput**: 10,000+ ops/sec +- **Memory Efficiency**: <500MB for 10K items +- **Concurrent Operations**: 100+ simultaneous operations + +## 🧠 Core Intelligence Features + +### Triple Intelligence System ✅ +Unified query system combining three types of intelligence: +```typescript +const results = await brain.find({ + like: 'AI research', // Vector similarity search + where: { year: 2024 }, // Metadata filtering + connected: { to: authorId } // Graph relationships +}) +``` + +### Intelligent Type Mapping ✅ +Prevents semantic degradation by intelligently inferring types: +```typescript +// Automatically infers 'person' from email field +brain.add({ name: "John", email: "john@example.com" }, 'entity') +// → Stored as type: 'person', not generic 'entity' +``` + +### Neural Query Understanding ✅ +- 220+ embedded patterns for intent detection +- Natural language query processing +- Automatic query optimization +- Pattern-based query rewriting + +## 🏢 Enterprise Features + +### Distributed Coordination ✅ +Raft consensus for multi-node deployments: +```typescript +import { DistributedCoordinator } from '@soulcraft/brainy' + +const coordinator = createCoordinator({ + nodeId: 'node-1', + peers: ['node-2', 'node-3'], + electionTimeout: 500 +}) +// Automatic leader election and failover +``` + +### Horizontal Sharding ✅ +Consistent hashing for data distribution: +```typescript +import { ShardManager } from '@soulcraft/brainy' + +const shards = createShardManager({ + nodes: ['node-1', 'node-2', 'node-3'], + replicationFactor: 2, + virtualNodes: 150 +}) +// Automatic shard rebalancing on node changes +``` + +### Read/Write Separation ✅ +Primary-replica architecture for scale: +```typescript +import { ReadWriteSeparation } from '@soulcraft/brainy' + +const replication = createReadWriteSeparation({ + role: 'auto', // Automatic primary/replica detection + consistencyLevel: 'strong', // or 'eventual' + readPreference: 'nearest' +}) +``` + +### Cross-Instance Cache Sync ✅ +Version vector-based cache synchronization: +```typescript +import { CacheSync } from '@soulcraft/brainy' + +const cache = createCacheSync({ + nodeId: 'node-1', + syncInterval: 100, + conflictResolution: 'version-vector' +}) +``` + +## 🔐 Security & Compliance + +### Rate Limiting ✅ +Per-operation configurable limits: +```typescript +const rateLimiter = createRateLimitAugmentation({ + limits: { + searches: 1000, // per minute + writes: 100, + reads: 5000, + deletes: 50 + } +}) +``` + +### Audit Logging ✅ +Comprehensive operation tracking: +```typescript +const auditLogger = createAuditLogAugmentation({ + logLevel: 'detailed', + retention: 90, // days + includeMetadata: true +}) + +// Query audit logs +const logs = auditLogger.queryLogs({ + operation: 'add', + startTime: Date.now() - 3600000 +}) +``` + +## 📦 Storage & Persistence + +Full crash recovery and replay: +```typescript + enabled: true, + checkpointInterval: 1000, + maxLogSize: 100 * 1024 * 1024 // 100MB +})) +``` + +### Multi-Tenancy ✅ +Service-based data isolation: +```typescript +// Isolated data per service +await brain.add(data, 'document', { service: 'tenant-1' }) +await brain.find('query', { service: 'tenant-1' }) +``` + +### Write-Only Mode ✅ +For dedicated write nodes: +```typescript +const brain = new Brainy({ + mode: 'write-only', + storage: 's3://bucket/path' +}) +``` + +## 🚀 Performance Features + +### Batch Operations ✅ +Optimized bulk processing: +```typescript +// Parallel processing with automatic batching +await brain.addMany(items) // <10ms per item +await brain.updateMany(updates) +await brain.deleteMany(filters) +``` + +### Request Deduplication ✅ +Automatic duplicate request handling: +```typescript +brain.use(new RequestDeduplicatorAugmentation()) +// Identical concurrent requests return same result +``` + +### Smart Caching ✅ +Intelligent search result caching: +```typescript +brain.use(new CacheAugmentation({ + maxSize: 10000, + ttl: 300000, // 5 minutes + invalidateOnWrite: true +})) +``` + +## 🔄 Data Processing + +### Entity Registry ✅ +Bloom filter-based deduplication: +```typescript +brain.use(new EntityRegistryAugmentation()) +// Handles millions of entities with minimal memory +``` + +### Neural Import ✅ +Intelligent data import with type inference: +```typescript +await brain.import({ + source: 'data.json', + autoDetectTypes: true, + batchSize: 1000 +}) +``` + +### Streaming Pipeline ✅ +Real-time data processing: +```typescript +brain.stream() + .pipe(transform) + .pipe(enrich) + .pipe(brain.writer()) +``` + +## 📊 Analytics & Monitoring + +### Metrics Collection ✅ +Built-in performance metrics: +```typescript +const metrics = brain.getMetrics() +// { +// operations: { add: 1000, find: 5000 }, +// performance: { p95: 8, p99: 12 }, +// cache: { hits: 4500, misses: 500 } +// } +``` + +### Health Monitoring ✅ +Automatic health checks: +```typescript +const health = brain.getHealth() +// { +// status: 'healthy', +// storage: 'connected', +// memory: { used: 245, limit: 512 } +// } +``` + +## 🛠️ Developer Experience + +### Zero Configuration ✅ +Works out of the box: +```typescript +import Brainy from '@soulcraft/brainy' +const brain = new Brainy() // Auto-configures everything +``` + +### TypeScript First ✅ +Full type safety and inference: +```typescript +// Types are automatically inferred +const results = await brain.find('query') +``` + +### Augmentation System ✅ +Extensible plugin architecture: +```typescript +class CustomAugmentation extends BaseAugmentation { + execute(operation, params, next) { + // Your custom logic + return next() + } +} +``` + +## 🔧 Operational Features + +### Graceful Shutdown ✅ +Clean shutdown with data persistence: +```typescript +process.on('SIGTERM', async () => { + await brain.shutdown() // Saves all pending data +}) +``` + +### Hot Reload ✅ +Configuration updates without restart: +```typescript +brain.updateConfig({ + cache: { enabled: false } +}) +``` + +### Backup & Restore ✅ +Full data backup capabilities: +```typescript +await brain.backup('backup.bin') +await brain.restore('backup.bin') +``` + +## 📈 Proven at Scale + +- **10,000+ items**: Sub-10ms search +- **1M+ operations**: Stable memory usage +- **100+ concurrent users**: No performance degradation +- **Multi-node clusters**: Automatic failover + +## 🚫 NOT Implemented (Planned) + +These features are documented but NOT yet implemented: +- GraphQL API (use REST API instead) +- Kubernetes operators (use Docker) +- Some distributed features require manual configuration + +--- + +*Last Updated: Latest Version* +*All features listed above are production-ready and tested* \ No newline at end of file diff --git a/docs/guides/MIGRATING_TO_V5.11.md b/docs/guides/MIGRATING_TO_V5.11.md deleted file mode 100644 index fa820b06..00000000 --- a/docs/guides/MIGRATING_TO_V5.11.md +++ /dev/null @@ -1,230 +0,0 @@ -# Migrating to v5.11.1 - -## Overview - -v5.11.1 introduces a **breaking change** with **massive performance benefits**: - -- `brain.get()` now loads **metadata-only by default** (76-81% faster!) -- Vector embeddings require **explicit opt-in**: `{ includeVectors: true }` - -**Impact**: Only ~6% of codebases need changes (code that computes similarity on retrieved entities). - -## What Changed - -### Before (v5.11.0 and earlier) - -```typescript -const entity = await brain.get(id) -// entity.vector was ALWAYS loaded (384 dimensions, 6KB) -console.log(entity.vector.length) // 384 -``` - -### After (v5.11.1) - -```typescript -// DEFAULT: Metadata-only (76-81% faster) -const entity = await brain.get(id) -console.log(entity.vector) // [] (empty array - not loaded) - -// EXPLICIT: Full entity with vectors -const entity = await brain.get(id, { includeVectors: true }) -console.log(entity.vector.length) // 384 -``` - -## Who Needs to Update? - -### ✅ NO CHANGES NEEDED (94% of code) - -If you use `brain.get()` for: -- **VFS operations** (readFile, stat, readdir) -- **Existence checks**: `if (await brain.get(id))` -- **Metadata access**: `entity.data`, `entity.type`, `entity.metadata` -- **Relationship traversal** -- **Admin tools**, import utilities, data APIs - -→ **Zero changes needed, automatic 76-81% speedup!** - -### ⚠️ REQUIRES UPDATE (~6% of code) - -If you use `brain.get()` AND then compute similarity on the returned entity: - -```typescript -// ❌ BEFORE (v5.11.0) - will break in v5.11.1 -const entity = await brain.get(id) -const similar = await brain.similar({ to: entity.vector }) // entity.vector is [] ! - -// ✅ AFTER (v5.11.1) - add includeVectors -const entity = await brain.get(id, { includeVectors: true }) -const similar = await brain.similar({ to: entity.vector }) // Works! -``` - -**Note**: `brain.similar({ to: entityId })` (using ID) still works - no changes needed! - -## Migration Steps - -### Step 1: Find Affected Code - -Search your codebase for patterns that use vectors from `brain.get()`: - -```bash -# Find brain.get() calls that access .vector -grep -r "await brain.get(" --include="*.ts" --include="*.js" | \ - grep -E "(\.vector|entity\.vector)" -``` - -### Step 2: Update Pattern-by-Pattern - -#### Pattern 1: Similarity Using Retrieved Entity Vector - -```typescript -// ❌ BEFORE -const entity = await brain.get(id) -const similar = await brain.similar({ to: entity.vector }) - -// ✅ AFTER - Option A: Add includeVectors -const entity = await brain.get(id, { includeVectors: true }) -const similar = await brain.similar({ to: entity.vector }) - -// ✅ AFTER - Option B: Use ID directly (recommended) -const similar = await brain.similar({ to: id }) -``` - -#### Pattern 2: Manual Vector Operations - -```typescript -// ❌ BEFORE -const entity = await brain.get(id) -const magnitude = Math.sqrt(entity.vector.reduce((sum, v) => sum + v*v, 0)) - -// ✅ AFTER -const entity = await brain.get(id, { includeVectors: true }) -const magnitude = Math.sqrt(entity.vector.reduce((sum, v) => sum + v*v, 0)) -``` - -#### Pattern 3: Vector Assertions in Tests - -```typescript -// ❌ BEFORE -const entity = await brain.get(id) -expect(entity.vector).toBeDefined() -expect(entity.vector.length).toBe(384) - -// ✅ AFTER -const entity = await brain.get(id, { includeVectors: true }) -expect(entity.vector).toBeDefined() -expect(entity.vector.length).toBe(384) -``` - -### Step 3: Verify Migration - -Run your test suite to catch any remaining issues: - -```bash -npm test -``` - -Look for errors like: -- `entity.vector is empty` or `entity.vector.length is 0` -- `Cannot compute similarity on empty vector` - -Add `{ includeVectors: true }` wherever these errors occur. - -## Performance Impact - -### Before Migration -``` -brain.get(): 43ms, 6KB per call -VFS readFile(): 53ms per file -VFS readdir(100 files): 5.3s -``` - -### After Migration -``` -brain.get(): 10ms, 300 bytes per call (76-81% faster) ✨ -brain.get({ includeVectors: true }): 43ms, 6KB (unchanged) -VFS readFile(): ~13ms per file (75% faster) ✨ -VFS readdir(100 files): ~1.3s (75% faster) ✨ -``` - -**Result**: -- VFS operations: **75% faster** -- Metadata access: **76-81% faster** -- Vector similarity: **Unchanged** (still fast when needed) - -## TypeScript Support - -The new `GetOptions` interface is fully typed: - -```typescript -interface GetOptions { - /** - * Include 384-dimensional vector embeddings in the response - * - * Default: false (metadata-only for 76-81% speedup) - */ - includeVectors?: boolean -} - -// TypeScript will autocomplete and validate -const entity = await brain.get(id, { includeVectors: true }) -``` - -## Rollback Plan - -If you encounter issues, you can temporarily force full entity loading everywhere: - -```typescript -// Temporary wrapper (NOT RECOMMENDED - defeats optimization) -async function getLegacy(id: string) { - return brain.get(id, { includeVectors: true }) -} - -// Use throughout codebase while migrating -const entity = await getLegacy(id) -``` - -**Important**: This defeats the 76-81% performance improvement. Only use temporarily while fixing affected code. - -## FAQ - -### Q: Why did you make this a breaking change? - -**A**: The performance gains are massive (76-81% speedup, 95% less bandwidth) and affect 94% of code positively. Only ~6% of code needs updates. The net benefit is enormous. - -### Q: Do I need to update my VFS code? - -**A**: No! VFS automatically benefits from the optimization with zero code changes. Your VFS operations are now 75% faster automatically. - -### Q: Will brain.similar() still work? - -**A**: Yes! `brain.similar({ to: entityId })` works exactly as before. Only `brain.similar({ to: entity.vector })` requires the entity to be loaded with `{ includeVectors: true }`. - -### Q: What about backward compatibility? - -**A**: Entities returned without vectors have `vector: []` (empty array), which is type-safe. Code that doesn't use vectors continues to work. Only code that explicitly uses `entity.vector` needs updating. - -### Q: Can I check if vectors are loaded? - -**A**: Yes! Check `entity.vector.length > 0` to detect if vectors were loaded. - -```typescript -const entity = await brain.get(id) -if (entity.vector.length > 0) { - // Vectors are loaded -} else { - // Metadata-only -} -``` - -## Support - -If you encounter migration issues: - -1. Check the [VFS Performance Guide](../vfs/VFS_PERFORMANCE.md) -2. Review [API Reference](../api/README.md) -3. See [Performance Documentation](../PERFORMANCE.md) -4. File an issue: https://github.com/soulcraft/brainy/issues - -## Changelog - -See [CHANGELOG.md](../../CHANGELOG.md) for complete v5.11.1 release notes. diff --git a/docs/guides/aggregation.md b/docs/guides/aggregation.md deleted file mode 100644 index 11d86ec8..00000000 --- a/docs/guides/aggregation.md +++ /dev/null @@ -1,593 +0,0 @@ -# Aggregation Guide - -> Real-time analytics on your entity data with incremental running totals - -## Overview - -Brainy's aggregation engine computes running totals at write time, so reading aggregate results is always O(1) regardless of dataset size. Define an aggregate once, and every `add()`, `update()`, and `delete()` automatically updates the running metrics. - -No batch jobs. No scheduled recalculations. Aggregates stay current with every write. - -**Defining over existing data:** if you define an aggregate on a store that already holds -matching entities, Brainy backfills it from those entities on the first query (a one-time scan, -then purely incremental). So `defineAggregate()` behaves the same whether you define it before -or after the data exists. - -**Reopening a persisted brain:** aggregate state persists across restarts. Re-defining the -same aggregate at boot (the normal declarative pattern) adopts the persisted state directly — -no rescan. A backfill scan runs only when the definition actually changed, when no persisted -state exists, or when the state failed to load; and however many aggregates need backfilling, -they share a single scan. - -## Quick Start - -```typescript -import { Brainy, NounType } from '@soulcraft/brainy' - -const brain = new Brainy() -await brain.init() - -// 1. Define an aggregate -brain.defineAggregate({ - name: 'sales_by_category', - source: { type: NounType.Event }, - groupBy: ['category'], - metrics: { - revenue: { op: 'sum', field: 'amount' }, - count: { op: 'count' }, - average: { op: 'avg', field: 'amount' } - } -}) - -// 2. Add entities — aggregates update automatically -await brain.add({ - data: 'Coffee purchase', - type: NounType.Event, - metadata: { category: 'food', amount: 5.50 } -}) - -await brain.add({ - data: 'Laptop purchase', - type: NounType.Event, - metadata: { category: 'electronics', amount: 1200 } -}) - -await brain.add({ - data: 'Lunch purchase', - type: NounType.Event, - metadata: { category: 'food', amount: 12.00 } -}) - -// 3. Query results -const results = await brain.find({ aggregate: 'sales_by_category' }) - -// Results: -// [ -// { groupKey: { category: 'food' }, metrics: { revenue: 17.50, count: 2, average: 8.75 } }, -// { groupKey: { category: 'electronics' }, metrics: { revenue: 1200, count: 1, average: 1200 } } -// ] -``` - -## Aggregation Operations - -Brainy supports 7 aggregation operations: - -### `sum` — Running Total - -Adds up all values of a numeric field. - -```typescript -metrics: { - total_revenue: { op: 'sum', field: 'amount' } -} -``` - -### `count` — Entity Count - -Counts the number of matching entities. No `field` required. - -```typescript -metrics: { - order_count: { op: 'count' } -} -``` - -### `avg` — Running Average - -Computes `sum / count` incrementally. - -```typescript -metrics: { - average_price: { op: 'avg', field: 'price' } -} -``` - -### `min` — Minimum Value - -Tracks the minimum value across all entities in each group. - -```typescript -metrics: { - lowest_price: { op: 'min', field: 'price' } -} -``` - -### `max` — Maximum Value - -Tracks the maximum value across all entities in each group. - -```typescript -metrics: { - highest_price: { op: 'max', field: 'price' } -} -``` - -### `stddev` — Sample Standard Deviation - -Computes the sample standard deviation using Welford's numerically stable online algorithm. Updates incrementally without storing individual values. - -```typescript -metrics: { - price_spread: { op: 'stddev', field: 'price' } -} -``` - -### `variance` — Sample Variance - -Computes the sample variance (square of standard deviation) using Welford's online algorithm. - -```typescript -metrics: { - price_variance: { op: 'variance', field: 'price' } -} -``` - -## GROUP BY Dimensions - -Every aggregate requires at least one `groupBy` dimension. Results are grouped by the unique combinations of dimension values. - -### Plain Fields - -Group by a metadata field value: - -```typescript -groupBy: ['category'] -// Produces groups: { category: 'food' }, { category: 'electronics' }, ... -``` - -### Multiple Fields - -Group by multiple fields for composite keys: - -```typescript -groupBy: ['category', 'region'] -// Produces groups: { category: 'food', region: 'US' }, { category: 'food', region: 'EU' }, ... -``` - -### Time Windows - -Group by a timestamp field bucketed into time periods: - -```typescript -groupBy: [{ field: 'date', window: 'month' }] -// Produces groups: { date: '2024-01' }, { date: '2024-02' }, ... -``` - -Available time window granularities: - -| Window | Format | Example | -|--------|--------|---------| -| `hour` | `YYYY-MM-DDThh` | `2024-01-15T14` | -| `day` | `YYYY-MM-DD` | `2024-01-15` | -| `week` | `YYYY-Wnn` | `2024-W03` | -| `month` | `YYYY-MM` | `2024-01` | -| `quarter` | `YYYY-Qn` | `2024-Q1` | -| `year` | `YYYY` | `2024` | -| `{ seconds: N }` | ISO 8601 | Custom interval | - -### Combined Dimensions - -Mix plain fields and time windows: - -```typescript -brain.defineAggregate({ - name: 'monthly_sales', - source: { type: NounType.Event }, - groupBy: ['region', { field: 'date', window: 'month' }], - metrics: { - revenue: { op: 'sum', field: 'amount' }, - count: { op: 'count' } - } -}) - -// Produces groups like: -// { region: 'US', date: '2024-01' } -// { region: 'US', date: '2024-02' } -// { region: 'EU', date: '2024-01' } -``` - -### Array Fields (Unnest) - -Group by **each element** of an array-valued field — for tag frequencies, label counts, and -faceted breakdowns. Mark the dimension `{ field, unnest: true }`: - -```typescript -brain.defineAggregate({ - name: 'tag_frequency', - source: { type: NounType.Document }, - groupBy: [{ field: 'tags', unnest: true }], - metrics: { count: { op: 'count' } } -}) - -// A document tagged ['ml', 'ai'] contributes once to the 'ml' group and once to 'ai'. -// Duplicate tags on one entity count once; an entity with no tags joins no group. -const top = await brain.queryAggregate('tag_frequency', { orderBy: 'count', order: 'desc' }) -// [ { groupKey: { tags: 'ai' }, metrics: { count: 3 }, count: 3 }, ... ] -``` - -## Querying Aggregates - -Aggregate results are queried through the standard `find()` method. - -### Basic Query - -```typescript -const results = await brain.find({ aggregate: 'sales_by_category' }) -``` - -### Filter by Group Key - -Use `where` to filter on group key values: - -```typescript -const foodOnly = await brain.find({ - aggregate: 'sales_by_category', - where: { category: 'food' } -}) -``` - -### Filter by Metric Value (HAVING) - -Use `having` to filter groups by their **computed metric values** — the analytics equivalent of -SQL `HAVING`. (`where` filters group *keys*; `having` filters *metrics*.) - -```typescript -const bigCategories = await brain.find({ - aggregate: 'sales_by_category', - having: { revenue: { greaterThan: 1000 } } -}) -``` - -`having` accepts the same operators as `where`, applied to each group's metric results plus -`count`. It is evaluated per group — **O(groups), independent of entity count** — before sorting -and pagination, so it stays cheap even over billions of entities. - -### Sort and Paginate - -Sort by any metric or group key field: - -```typescript -const topCategories = await brain.find({ - aggregate: { - name: 'sales_by_category', - orderBy: 'revenue', - order: 'desc', - limit: 10 - } -}) -``` - -### Combined Parameters - -`where`, `orderBy`, `limit`, and `offset` from the outer `find()` call merge automatically with the aggregate query: - -```typescript -const recentTopSpenders = await brain.find({ - aggregate: 'monthly_sales', - where: { region: 'US' }, - orderBy: 'revenue', - order: 'desc', - limit: 12, - offset: 0 -}) -``` - -### Result Format - -`find({ aggregate })` returns `Result` rows (for uniformity with the rest of `find()`), -with the aggregate fields surfaced **both** at the top level and, for backward compatibility, -flattened into `metadata`: - -```typescript -{ - id: string, - score: 1.0, - type: NounType.Measurement, - groupKey: { category: 'food' }, // top-level — the group key values - metrics: { revenue: 17.50, count: 2, average: 8.75 }, // top-level — computed metrics - count: 2, // top-level — entities in the group - metadata: { // legacy mirror of the same data - __aggregate: 'sales_by_category', - category: 'food', - revenue: 17.50, count: 2, average: 8.75 - }, - entity: Entity -} -``` - -### `queryAggregate()` — the report-friendly view - -For dashboards and reports, prefer `brain.queryAggregate(name, params)`. It returns the clean -`AggregateResult[]` shape directly — no search-result wrapper: - -```typescript -const rows = await brain.queryAggregate('sales_by_category', { - orderBy: 'revenue', - order: 'desc', - limit: 10 -}) -// [ -// { groupKey: { category: 'electronics' }, metrics: { revenue: 1200, count: 1, average: 1200 }, count: 1 }, -// { groupKey: { category: 'food' }, metrics: { revenue: 17.50, count: 2, average: 8.75 }, count: 2 } -// ] -``` - -It accepts the same `where` / `having` / `orderBy` / `order` / `limit` / `offset` params as the -`find({ aggregate })` form. - -## Source Filtering - -Control which entities feed into an aggregate with the `source` property. - -### Filter by Entity Type - -```typescript -brain.defineAggregate({ - name: 'event_stats', - source: { type: NounType.Event }, - groupBy: ['category'], - metrics: { count: { op: 'count' } } -}) -``` - -### Filter by Multiple Types - -```typescript -source: { type: [NounType.Event, NounType.Document] } -``` - -### Filter by Metadata - -Use the same `where` syntax as `find()`: - -```typescript -source: { - type: NounType.Event, - where: { domain: 'financial', subtype: 'transaction' } -} -``` - -### Filter by Service - -For multi-tenant deployments: - -```typescript -source: { service: 'tenant-123' } -``` - -Entities that don't match the source filter are silently skipped during incremental updates. - -## Incremental Updates - -The aggregation engine hooks into every write operation: - -### On `add()` - -When a new entity matches an aggregate's source filter: -1. The group key is computed from the entity's metadata -2. Each metric in the matching group is incremented -3. New groups are created automatically - -### On `update()` - -When an existing entity is updated: -1. The old entity's contribution is reversed from its group -2. The new entity's contribution is applied to its (potentially different) group -3. Handles group key changes — an entity moving from category "food" to "drink" updates both groups - -### On `delete()` - -When an entity is deleted: -1. The entity's contribution is reversed from its group -2. If a group becomes empty (all metric counts reach zero), it's removed - -### Aggregate Entity Exclusion - -Materialized `NounType.Measurement` entities are automatically excluded from all source matching, preventing infinite feedback loops. Entities with `service: 'brainy:aggregation'` or `metadata.__aggregate` are always skipped. - -## Materialization - -Materialization writes aggregate results as `NounType.Measurement` entities, making them automatically available through OData, Google Sheets, SSE, and webhook integrations. - -```typescript -brain.defineAggregate({ - name: 'daily_metrics', - source: { type: NounType.Event }, - groupBy: [{ field: 'date', window: 'day' }], - metrics: { - total: { op: 'sum', field: 'amount' }, - count: { op: 'count' } - }, - materialize: true -}) -``` - -### Debounce Configuration - -During high-throughput ingestion, materialization is debounced to avoid excessive writes: - -```typescript -materialize: { - debounceMs: 2000, // Wait 2 seconds after last update before writing - trackSources: true // Track which entities contributed -} -``` - -The default debounce interval is 1000ms. - -## Multiple Aggregates - -Define multiple aggregates that process the same entities: - -```typescript -// Revenue by category -brain.defineAggregate({ - name: 'category_revenue', - source: { type: NounType.Event }, - groupBy: ['category'], - metrics: { total: { op: 'sum', field: 'amount' } } -}) - -// Monthly trends -brain.defineAggregate({ - name: 'monthly_trends', - source: { type: NounType.Event }, - groupBy: [{ field: 'date', window: 'month' }], - metrics: { - revenue: { op: 'sum', field: 'amount' }, - count: { op: 'count' }, - avg_order: { op: 'avg', field: 'amount' } - } -}) - -// Regional breakdown with statistical analysis -brain.defineAggregate({ - name: 'regional_analysis', - source: { type: NounType.Event }, - groupBy: ['region'], - metrics: { - revenue: { op: 'sum', field: 'amount' }, - spread: { op: 'stddev', field: 'amount' }, - variance: { op: 'variance', field: 'amount' } - } -}) -``` - -Each `add()` call updates all matching aggregates automatically. - -## Removing Aggregates - -Remove an aggregate and clean up its state: - -```typescript -brain.removeAggregate('category_revenue') -``` - -## Persistence - -Aggregate definitions and running state are automatically persisted: - -- **On `flush()`/`close()`**: All dirty aggregate state is written to storage -- **On `init()`**: Definitions and state are restored from storage -- **Change detection**: Definition changes are detected via FNV-1a hashing — only changed aggregates reset their state on restart - -## Native Acceleration - -When [Cor](https://www.npmjs.com/package/@soulcraft/cor) is installed as a plugin, the aggregation engine automatically uses Rust-accelerated computation: - -- Incremental updates run in Rust with BTreeMap-backed precise MIN/MAX -- Welford's online stddev/variance computed natively -- Rebuild uses Rayon parallel iterators across CPU cores (above 1,000 entities) -- Time window bucketing uses integer arithmetic without `Date` object allocation - -```typescript -const brain = new Brainy({ - plugins: ['@soulcraft/cor'] -}) -await brain.init() - -// Aggregation automatically uses native engine -brain.defineAggregate({ ... }) -``` - -Verify native acceleration is active: - -```typescript -const diag = brain.diagnostics() -console.log(diag.providers.aggregation) -// { source: 'plugin' } -``` - -## Common Patterns - -### Financial Analytics - -```typescript -brain.defineAggregate({ - name: 'monthly_spending', - source: { - type: NounType.Event, - where: { domain: 'financial', subtype: 'transaction' } - }, - groupBy: [ - 'category', - { field: 'date', window: 'month' } - ], - metrics: { - total: { op: 'sum', field: 'amount' }, - count: { op: 'count' }, - average: { op: 'avg', field: 'amount' }, - highest: { op: 'max', field: 'amount' }, - lowest: { op: 'min', field: 'amount' } - }, - materialize: true -}) -``` - -### Time-Series Monitoring - -```typescript -brain.defineAggregate({ - name: 'hourly_metrics', - source: { type: NounType.Event, where: { domain: 'monitoring' } }, - groupBy: [ - 'service', - { field: 'timestamp', window: 'hour' } - ], - metrics: { - request_count: { op: 'count' }, - avg_latency: { op: 'avg', field: 'latency_ms' }, - max_latency: { op: 'max', field: 'latency_ms' }, - error_count: { op: 'sum', field: 'is_error' }, - latency_spread: { op: 'stddev', field: 'latency_ms' } - } -}) -``` - -### Content Analytics - -```typescript -brain.defineAggregate({ - name: 'content_stats', - source: { type: NounType.Document }, - groupBy: ['author', { field: 'publishedAt', window: 'month' }], - metrics: { - articles: { op: 'count' }, - total_words: { op: 'sum', field: 'wordCount' }, - avg_words: { op: 'avg', field: 'wordCount' } - } -}) -``` - -## Performance - -Aggregation complexity per write is O(A x G x M) where A = matching aggregates, G = groupBy dimensions, M = metrics. For typical configurations (2-5 aggregates, 1-3 dimensions, 3-5 metrics), this is effectively O(1). - -With Cor native acceleration: - -| Operation | Throughput | Latency | -|-----------|-----------|---------| -| Incremental update (1K entities) | 809 ops/s | 1.2 ms | -| Rebuild (10K entities) | 475 ops/s | 2.1 ms | -| Rebuild (100K entities, Rayon) | 66 ops/s | 15.2 ms | -| Query (1K groups, sort + paginate) | 986 ops/s | 1.0 ms | diff --git a/docs/guides/distributed-system.md b/docs/guides/distributed-system.md new file mode 100644 index 00000000..d368ac5a --- /dev/null +++ b/docs/guides/distributed-system.md @@ -0,0 +1,406 @@ +# Distributed Brainy System Guide + +## Overview + +Brainy introduces a groundbreaking **zero-configuration distributed system** that transforms how vector databases scale. Unlike traditional distributed databases that require complex setup (Consul, etcd, Zookeeper), Brainy uses your existing storage (S3, GCS, R2) as the coordination layer. + +## Key Innovation: Storage-Based Coordination + +Instead of requiring separate infrastructure for coordination, Brainy leverages your storage backend: + +```typescript +// Traditional distributed database setup: +// ❌ Setup Consul/etcd +// ❌ Configure node discovery +// ❌ Setup health checks +// ❌ Configure sharding +// ❌ Setup replication + +// Brainy distributed setup: +const brain = new Brainy({ + storage: { + type: 's3', + options: { bucket: 'my-data' } + }, + distributed: true // ✅ That's it! +}) +``` + +## Real-World Scenarios + +### 1. Processing Multiple Streaming Data Sources (Bluesky, Twitter, etc.) + +**Problem**: You need to ingest millions of posts from multiple social media firehoses simultaneously while maintaining search performance. + +**Traditional Approach**: +- Separate ingestion and search clusters +- Complex queue systems (Kafka, RabbitMQ) +- Manual sharding configuration +- Complicated backpressure handling + +**Brainy Solution**: +```typescript +// Node 1: Bluesky Ingestion +const ingestionNode1 = new Brainy({ + storage: { type: 's3', options: { bucket: 'social-data' }}, + distributed: true, + writeOnly: true // Optimized for writes +}) + +// Continuously ingest Bluesky firehose +blueskyStream.on('post', async (post) => { + await ingestionNode1.add({ + content: post.text, + author: post.author, + timestamp: post.createdAt, + platform: 'bluesky' + }, 'social-post') +}) + +// Node 2: Twitter Ingestion (separate machine) +const ingestionNode2 = new Brainy({ + storage: { type: 's3', options: { bucket: 'social-data' }}, + distributed: true, + writeOnly: true +}) + +// Node 3-5: Search nodes (auto-balanced) +const searchNode = new Brainy({ + storage: { type: 's3', options: { bucket: 'social-data' }}, + distributed: true, + readOnly: true // Optimized for queries +}) + +// Search across ALL data from ALL sources +const results = await searchNode.find('AI trends', 100) +// Automatically queries all shards across all nodes! +``` + +**Benefits**: +- **Auto-sharding**: Data automatically distributed by content hash +- **No bottlenecks**: Each ingestion node writes directly to storage +- **Live search**: Search nodes see new data immediately +- **Auto-scaling**: Add nodes anytime, data rebalances automatically + +### 2. Multi-Tenant SaaS Application + +**Problem**: Each customer needs isolated, fast search across their documents, with ability to scale per customer. + +**Brainy Solution**: +```typescript +// Spin up dedicated nodes per large customer +const enterpriseNode = new Brainy({ + storage: { + type: 's3', + options: { + bucket: 'customer-data', + prefix: 'customer-123/' // Isolated data + } + }, + distributed: true +}) + +// Shared nodes for smaller customers +const sharedNode = new Brainy({ + storage: { + type: 's3', + options: { bucket: 'shared-customers' } + }, + distributed: true +}) + +// Domain-based sharding ensures customer data stays together +await sharedNode.add(document, 'document', { + customerId: 'cust-456', // Used for shard assignment + domain: 'customer-456' // Keeps related data together +}) +``` + +### 3. Global Knowledge Graph with Local Inference + +**Problem**: Building a knowledge graph that needs both global connectivity and local LLM inference. + +**Brainy Solution**: +```typescript +// Edge nodes near users (with GPU) +const edgeNode = new Brainy({ + storage: { type: 's3', options: { bucket: 'knowledge-graph' }}, + distributed: true, + models: { + embed: { model: 'BAAI/bge-base-en-v1.5' }, + chat: { model: 'meta-llama/Llama-3.2-3B-Instruct' } + } +}) + +// Central nodes for graph operations +const graphNode = new Brainy({ + storage: { type: 's3', options: { bucket: 'knowledge-graph' }}, + distributed: true, + augmentations: ['graph', 'triple-intelligence'] +}) + +// Local inference with global knowledge +const context = await edgeNode.find(userQuery, 10) +const response = await edgeNode.chat(userQuery, { context }) + +// Graph traversal across all nodes +const connections = await graphNode.traverse({ + start: 'concept:AI', + depth: 3, + relationship: 'related-to' +}) +``` + +## Competitive Advantages + +### vs. Pinecone/Weaviate/Qdrant + +| Feature | Traditional Vector DBs | Brainy Distributed | +|---------|----------------------|-------------------| +| Setup Complexity | High (k8s, operators) | Zero (just storage) | +| Minimum Nodes | 3-5 for HA | 1 (scale as needed) | +| Coordination | External (etcd, Consul) | Built-in (via storage) | +| Data Locality | Random sharding | Domain-aware sharding | +| Query Planning | Basic | Triple Intelligence | +| Cost at Scale | High (always-on clusters) | Low (scale to zero) | + +### vs. Neo4j/ArangoDB (Graph Databases) + +| Feature | Graph Databases | Brainy Distributed | +|---------|----------------|-------------------| +| Vector Search | Bolt-on/Limited | Native HNSW | +| Embedding Generation | External | Built-in (30+ models) | +| Distributed Transactions | Complex/Slow | Eventually consistent | +| Natural Language | No | Native (Triple Intelligence) | +| Setup | Very Complex | Zero config | + +### vs. Elasticsearch/OpenSearch + +| Feature | Elasticsearch | Brainy Distributed | +|---------|--------------|-------------------| +| Vector Support | Added later (slow) | Native (fast HNSW) | +| Cluster Management | Complex (master nodes) | Automatic (via storage) | +| Shard Rebalancing | Manual/Risky | Automatic/Safe | +| Memory Usage | Very High | Efficient | +| Query Language | Complex DSL | Natural language | + +## Innovative Features + +### 1. Domain-Aware Sharding + +Unlike hash-based sharding, Brainy understands data relationships: + +```typescript +// Documents about the same topic stay on the same shard +await brain.add(doc1, 'document', { domain: 'physics' }) +await brain.add(doc2, 'document', { domain: 'physics' }) +// Both documents on same shard = faster related queries + +// Customer data stays together +await brain.add(order, 'order', { customerId: 'cust-123' }) +await brain.add(invoice, 'invoice', { customerId: 'cust-123' }) +// Same customer = same shard = better locality +``` + +### 2. Streaming Shard Migration + +Zero-downtime data movement between nodes: + +```typescript +// Automatically triggered when nodes join/leave +// Uses HTTP streaming for efficiency +// Validates data integrity +// Atomic ownership transfer +// No query downtime! +``` + +### 3. Storage-Based Consensus + +No Raft/Paxos complexity: + +```typescript +// Leader election via storage atomic operations +// Health monitoring via storage heartbeats +// Configuration consensus via storage CAS +// No split-brain issues! +``` + +### 4. Intelligent Query Planning + +The distributed query planner understands: +- Which shards contain relevant data +- Node health and latency +- Data locality and caching +- Triple Intelligence scoring + +```typescript +// Automatically optimizes query execution +const results = await brain.find('quantum physics') +// Planner knows: +// - Physics domain → shard-3 +// - Node-2 has shard-3 cached +// - Route query to node-2 +// - Merge results with Triple Intelligence +``` + +## Performance Characteristics + +### Scalability +- **Horizontal**: Add nodes anytime +- **Vertical**: Nodes can differ in size +- **Geographic**: Nodes can be globally distributed +- **Elastic**: Scale to zero when idle + +### Throughput +- **Writes**: Linear scaling with nodes +- **Reads**: Sub-linear (due to caching) +- **Mixed**: Read/write optimized nodes + +### Latency +- **Local queries**: ~10ms p50 +- **Distributed queries**: ~50ms p50 +- **Shard migration**: Streaming (no bulk pause) + +## Use Cases Where Brainy Excels + +### ✅ Perfect For: + +1. **Multi-source data ingestion** (social media, logs, events) +2. **Global search applications** (distributed teams) +3. **Multi-tenant SaaS** (customer isolation) +4. **Knowledge graphs** (with vector search) +5. **Edge AI applications** (local inference, global knowledge) +6. **Document intelligence** (contracts, research papers) +7. **Real-time analytics** (streaming + search) + +### ⚠️ Consider Alternatives For: + +1. **Strong consistency requirements** (use PostgreSQL) +2. **Sub-millisecond latency** (use Redis) +3. **Complex transactions** (use traditional RDBMS) +4. **Purely structured data** (use columnar stores) + +## Migration from Other Systems + +### From Pinecone/Weaviate: +```typescript +// Your existing vector search still works +const results = await brain.search(embedding, 10) + +// But now you can scale horizontally! +// And add graph relationships! +// And use natural language! +``` + +### From Elasticsearch: +```typescript +// Import your documents +await brain.import('./elasticsearch-export.json') + +// Queries are simpler +const results = await brain.find('user query') +// No complex DSL needed! +``` + +### From Neo4j: +```typescript +// Import your graph +await brain.importGraph('./neo4j-export.cypher') + +// Now with vector search! +const similar = await brain.find('concepts like quantum computing') +``` + +## Deployment Patterns + +### 1. Start Simple, Scale Later +```typescript +// Day 1: Single node +const brain = new Brainy({ storage: 's3' }) + +// Month 2: Growing data, add distribution +const brain = new Brainy({ + storage: 's3', + distributed: true // Just add this! +}) + +// Month 6: Multiple nodes auto-balance +// No migration needed! +``` + +### 2. Geographic Distribution +```typescript +// US Node +const usNode = new Brainy({ + storage: { region: 'us-east-1' }, + distributed: true +}) + +// EU Node +const euNode = new Brainy({ + storage: { region: 'eu-west-1' }, + distributed: true +}) + +// They automatically coordinate! +``` + +### 3. Specialized Nodes +```typescript +// GPU nodes for embedding +const embedNode = new Brainy({ + distributed: true, + writeOnly: true, + models: { embed: 'large-model' } +}) + +// CPU nodes for search +const searchNode = new Brainy({ + distributed: true, + readOnly: true +}) +``` + +## Monitoring & Operations + +### Health Checks +```typescript +const health = await brain.getClusterHealth() +// { +// nodes: 5, +// healthy: 5, +// shards: 16, +// status: 'green' +// } +``` + +### Shard Distribution +```typescript +const shards = await brain.getShardDistribution() +// Shows which nodes own which shards +``` + +### Migration Status +```typescript +const migrations = await brain.getActiveMigrations() +// Shows ongoing shard movements +``` + +## Conclusion + +Brainy's distributed system is **production-ready** and offers: + +1. **True zero-configuration** - Just add `distributed: true` +2. **Storage-based coordination** - No external dependencies +3. **Intelligent sharding** - Domain-aware data placement +4. **Automatic operations** - Rebalancing, failover, scaling +5. **Unified interface** - Vector + Graph + Document + LLM + +This is not just another distributed database. It's a fundamental rethinking of how distributed systems should work in the cloud era. + +## Next Steps + +1. [Try the distributed quick start](./distributed-quickstart.md) +2. [Read the architecture deep dive](../architecture/distributed-storage.md) +3. [View benchmarks](../benchmarks/distributed-performance.md) +4. [Deploy to production](./production-deployment.md) \ No newline at end of file diff --git a/docs/guides/enterprise-for-everyone.md b/docs/guides/enterprise-for-everyone.md index b79844e9..3196910e 100644 --- a/docs/guides/enterprise-for-everyone.md +++ b/docs/guides/enterprise-for-everyone.md @@ -165,28 +165,32 @@ await brain.syncWith({ - **Webhook support**: React to changes - **API generation**: Auto-generate REST/GraphQL APIs -### 🌍 Scale +### 🌍 Enterprise Scale 🚧 Coming Soon -**Everyone gets the same scale model:** +**Everyone gets planetary scale:** ```typescript -// Pure JS by default; install the optional native provider for billions of vectors -const brain = new Brainy() +// Same architecture Netflix uses, free for you +const brain = new Brainy({ + clustering: { + enabled: true, // Distributed mode + sharding: 'automatic', // Auto-sharding + replication: 3, // Triple replication + consensus: 'raft', // Strong consistency + geoDistribution: true // Multi-region support + } +}) -// 1 → ~1M vectors: pure-JS HNSW, zero extra setup -// 1M → 10B+ vectors: install @soulcraft/cor for the native DiskANN provider +// Handles everything from 1 to 1 billion entities ``` -**Scaling model:** -- **Single process, no cluster**: Brainy runs in one process — no coordinator, - no peer discovery, no consensus to operate -- **Optional native provider**: install `@soulcraft/cor` to back the index with - on-disk DiskANN that scales to 10B+ vectors on one machine -- **Per-tenant pools**: isolate tenants by giving each its own Brainy instance and - storage directory -- **Horizontal read scaling**: run many reader processes against one shared on-disk - store (single writer, many readers); replicate the artifact with your operator - tooling +**Scaling features:** +- **Horizontal scaling**: Add nodes as needed +- **Auto-sharding**: Distributes data automatically +- **Multi-region**: Global distribution +- **Load balancing**: Automatic request distribution +- **Zero-downtime upgrades**: Rolling updates +- **Infinite scale**: No upper limits ### 🛡️ Enterprise Compliance 🚧 Coming Soon @@ -438,4 +442,4 @@ Brainy is more than software—it's a movement to democratize enterprise technol - [Zero Configuration](../architecture/zero-config.md) - [Augmentations System](../architecture/augmentations.md) - [Architecture Overview](../architecture/overview.md) -- [API Reference](../api/README.md) \ No newline at end of file +- [Getting Started](./getting-started.md) \ No newline at end of file diff --git a/docs/guides/export-and-import.md b/docs/guides/export-and-import.md deleted file mode 100644 index 042728b1..00000000 --- a/docs/guides/export-and-import.md +++ /dev/null @@ -1,181 +0,0 @@ ---- -title: Export & Import (portable graph) -slug: guides/export-and-import -public: true -category: guides -template: guide -order: 9 -description: Export part or all of a brain to a portable, versioned PortableGraph document and import it back — by id, collection, connected neighbourhood, VFS subtree, predicate, or the whole brain. export() lives on the immutable Db, so asOf()/with() give time-travel and what-if exports. -next: - - guides/subtypes-and-facets - - api/README ---- - -# Export & Import (portable graph) - -Brainy serializes part or all of a brain — an item, a collection, a connected -neighbourhood, a VFS subtree, a predicate match, or the whole brain — into a single -versioned JSON document (`PortableGraph`), and restores it. - -```typescript -const graph = await brain.export() // whole brain → PortableGraph -await brain.import(graph) // restore (merge by id, re-embed if no vectors) -``` - -It is **portable** (human-readable JSON), **versioned** (`formatVersion`, so a document -written by 7.x imports cleanly into 8.0), and **current-state** (the entities and edges as -they are now — no generation history). Use it for portable artifacts, partial exports, -cross-environment moves, and version upgrades. - -`export()` is a method on the **immutable `Db` value**, so it composes with every way of -obtaining one: - -```typescript -brain.export(sel) // = brain.now().export(sel) -;(await brain.asOf(gen)).export(sel) // time-travel export (a past generation) -brain.now().with(ops).export(sel) // what-if export (a speculative state) -``` - -## When to use which - -| You want… | Use | -|-----------|-----| -| A portable, partial-or-whole, cross-version graph document | **`brain.export()` / `brain.import()`** (this guide) | -| A whole-brain snapshot **with generation history** | `brain.now().persist(path)` / `Brainy.load(path)` (native) | -| To ingest a CSV / PDF / Excel / JSON **file** as new entities | `brain.import(file)` — see [Import Anything](./import-anything.md) | - -`import()` is **polymorphic**: hand it a `PortableGraph` and it does the graph round-trip; -hand it a file/buffer and it does foreign-file ingestion (dispatched on the document's -`format: 'brainy-portable-graph'` tag). - -## Exporting - -```typescript -brain.export(selector?, options?): Promise -// (also on any Db: brain.now().export(...), (await brain.asOf(g)).export(...)) -``` - -### Selectors — *what* to export - -Omit the selector to export the whole brain. Otherwise pick a node set: - -| Scenario | Selector | -|----------|----------| -| Just an item (or items) | `{ ids: ['a', 'b'] }` | -| A collection + its children | `{ collection: collectionId }` (alias `memberOf`) | -| A connected neighbourhood | `{ connected: { from: id, depth: 2, verbs?, direction? } }` | -| A VFS directory / file (+ subtree) | `{ vfsPath: '/docs', recursive?: true }` | -| Everything matching a predicate | `{ type, subtype, where, service, visibility }` | -| The whole brain | *(omit)* | - -The selector reuses `find()`'s grammar — *"export what `find()` would match, minus ranking -and limit."* Structural and predicate selectors **compose**: - -```typescript -// Members of a collection whose status is "open" -await brain.export({ collection: collectionId, where: { status: 'open' } }) - -// Already have find() results? Export exactly those with the ids selector -const hits = await brain.find({ type: NounType.Document }) -await brain.export({ ids: hits.map(r => r.id) }) -``` - -### Options — *how* to serialize - -| Option | Default | Effect | -|--------|---------|--------| -| `includeVectors` | `false` | Carry embedding vectors verbatim. Off ⇒ `import()` re-embeds from `data`. | -| `includeContent` | `false` | Include VFS file bytes in `blobs` so files round-trip byte-identically. | -| `includeSystem` | `false` | Include `visibility:'system'` entities such as the VFS root. | -| `edges` | `'induced'` | `'induced'` (both endpoints in the set), `'incident'` (also dangling edges, recorded in `danglingIds`), or `'none'` (nodes only). | - -## Importing - -```typescript -brain.import(graph, options?): Promise -``` - -The whole graph is applied as **one atomic transaction** — it advances the brain exactly -one generation, or none on failure. - -```typescript -const result = await brain.import(graph, { onConflict: 'merge' }) -// → { imported, merged, skipped, reembedded, blobsWritten, errors } -``` - -| Option | Default | Effect | -|--------|---------|--------| -| `onConflict` | `'merge'` | `'merge'` (update existing id in place — assemble many exported graphs), `'replace'` (delete + recreate), or `'skip'`. | -| `reembed` | `'auto'` | `'auto'` (use the carried vector, else re-embed from `data`) or `'never'` (require a carried vector; record an error if absent). | -| `remapIds` | — | Rewrite every id on the way in, e.g. to clone a template subgraph under fresh ids. | -| `meta` | — | Transaction metadata recorded in the tx-log alongside the new generation. | - -The default `onConflict: 'merge'` lets you assemble one working graph from many exported -documents that share entity ids — re-importing an id merges rather than duplicates. - -## The `PortableGraph` format - -```jsonc -{ - "format": "brainy-portable-graph", // identifies the document type - "formatVersion": 1, // import gates on this (cross-version migration) - "brainyVersion": "8.0.0", - "createdAt": "2026-06-16T…Z", - "embedding": { "model": "all-MiniLM-L6-v2", "dimensions": 384 }, - "selector": { … }, // echoes what was exported (provenance) - "entities": [ - { - "id": "…", "type": "Document", "subtype": "invoice", "visibility": "public", - "data": "…", // the embedding source - "confidence": 1, "weight": 1, "service": "…", - "vector": [ … ], // only with includeVectors - "metadata": { … } // custom fields only (reserved fields are top-level) - } - ], - "relations": [ - { "id":"…", "from":"…", "to":"…", "type":"Contains", "subtype":"…", - "weight":1, "confidence":1, "metadata": { … } } - ], - "blobs": { "": "" }, // only with includeContent - "danglingIds": [ "…" ], // only with edges:'incident' - "stats": { "entityCount": 0, "relationCount": 0, "blobCount": 0, "vectorDimensions": 384 } -} -``` - -Standard fields (`subtype`, `visibility`, `data`, `confidence`, `weight`, `service`) sit at -the top level of each entity; `metadata` holds **only** custom user fields — mirroring the -in-memory `Entity` shape, so `import()` maps each field to its dedicated parameter. The -TypeScript types (`PortableGraph`, `PortableGraphEntity`, `PortableGraphRelation`, `ExportSelector`, -`ExportOptions`, `ImportOptions`, `ImportResult`) are exported from the package root. - -## Generations & time-travel - -The portable document is **current-state** — it never embeds generation history (that keeps -it cross-version-portable). History lives where it's queryable: - -- **During a session:** `brain.asOf(g)` / `brain.now().with(ops)` on the live brain. Because - `export()` is on the `Db`, `(await brain.asOf(g)).export()` serializes a *past* generation - and `brain.now().with(ops).export()` serializes a *speculative* one. -- **A whole-brain snapshot with history:** `brain.now().persist(path)` / `Brainy.load(path)` - (native, generation-preserving) — a separate facility from this portable format. - -Note: only `transact()` (and the write shortcuts that commit through it) advances a -generation, so time-travel export differs across transaction boundaries. - -## Cross-version (7.x → 8.0) - -Because the document is shared and versioned, a PortableGraph written by 7.x imports into 8.0: -`formatVersion` is read forward, `subtype` is carried so 8.0 re-types correctly, and the -same 384-dimension model on both lines means `includeVectors:false` re-embeds identically -(or `true` carries vectors verbatim). - -## VFS - -VFS directories are `Collection` entities and files are entities linked by `Contains`, so -the whole filesystem (or any subtree) exports through the `vfsPath` selector: - -```typescript -await brain.export({ vfsPath: '/' }, { includeContent: true }) // all VFS + bytes -await brain.export({ vfsPath: '/docs' }, { includeContent: true }) // one directory -await brain.export({ vfsPath: '/a/b.txt' }, { includeContent: true }) // one file -``` diff --git a/docs/guides/external-backups-and-sparse-storage.md b/docs/guides/external-backups-and-sparse-storage.md deleted file mode 100644 index f28dcfd7..00000000 --- a/docs/guides/external-backups-and-sparse-storage.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -title: External Backups & Sparse Storage -slug: guides/external-backups -public: true -category: guides -template: guide -order: 10 -description: How to back up a brain directory with external tools (tar, rsync, cp) without exploding sparse files — why a store can show 100+ GB "apparent" size on a small disk, which files are sparse, and how persist()/restore() handle it for you. -next: - - guides/snapshots-and-time-travel - - concepts/storage-adapters ---- - -# External Backups & Sparse Storage - -The built-in snapshot path — [`db.persist()` and `brain.restore()`](/docs/guides/snapshots-and-time-travel) — -already handles everything on this page for you. Read this when you back up a brain directory with -**external tools**: `tar`, `rsync`, `cp`, `scp`, or a filesystem-level backup agent. - -## The one-sentence rule - -> **Always use the sparse-aware flag**: `tar czSf` (capital `S`), `rsync --sparse`, -> `cp --sparse=always`. A naive copy can turn a 2 GB store into a 100+ GB one — or fail -> the disk entirely. - -## Why: some files are sparse - -When a native accelerator plugin is active, parts of the index live in **memory-mapped files** -created at a large fixed virtual size — the file's *apparent* size — while the filesystem only -allocates blocks that were actually written. A brand-new id-mapper file can report tens of -gigabytes in `ls -l` while occupying a few megabytes on disk. - -Check the difference yourself: - -```bash -ls -lh brain-data/_id_mapper/ # APPARENT size (can be huge) -du -sh brain-data/ # ALLOCATED size (the real footprint) -``` - -The sparse candidates in a brain directory: - -| Path | What it is | -|---|---| -| `_id_mapper/` | The native id-mapper's mmap files (large fixed virtual size) | -| `_blobs/` | Native index files (vector base, segments) — may be mmap-backed | - -Everything else (entities, `_system`, `_generations`, `_cas` content blobs) is ordinary dense data. - -## Doing it right - -**tar** — the `S` flag detects holes and stores only real data: - -```bash -tar czSf brain-backup.tgz /data/brain -# restore preserves the holes: -tar xzSf brain-backup.tgz -C /data/ -``` - -**rsync**: - -```bash -rsync -a --sparse /data/brain/ backup-host:/backups/brain/ -``` - -**cp**: - -```bash -cp -a --sparse=always /data/brain /backups/brain -``` - -**What goes wrong without the flag:** the copy *materializes* every hole as real zero bytes. -A store whose apparent size exceeds the target disk fails with `ENOSPC` partway through — and a -copy that *does* fit silently costs the full apparent size in storage and transfer time. - -## What the built-in paths do (so you don't have to) - -- **`db.persist(path)`** snapshots via **hard links** — instant and space-shared, since every data - file is immutable-by-rename. The handful of append-in-place files (the transaction log, the - commit fact log's tail segment) and mmap-mutated directories (`_id_mapper/`) are **byte-copied** - instead, so a post-snapshot write can never reach through a shared inode into your backup. -- **`brain.restore(path, { confirm: true })`** is **non-destructive and sparse-aware**: the snapshot - is copied into a staging area *before* any live data is touched (all-zero blocks stay holes), and - only after the copy fully succeeds does an atomic swap move it into place. A failed copy — - including `ENOSPC` — leaves the live store exactly as it was. A crash mid-swap completes forward - on the next open. - -## Live-store caveats for external tools - -1. **Prefer snapshotting a `persist()` output, not the live directory.** `persist()` produces a - crash-consistent, immutable snapshot; running `tar` against a live, actively-written directory - can capture a torn mid-write state. If you must archive live, stop writes first (or accept that - the archive is only as consistent as the moment's flush state). -2. **Never prune or "clean up" files inside a brain directory.** Index files that look stale or - redundant are load-bearing; the store protects its declared index families from in-process - deletion, but an external `rm` bypasses that fence. If space is the concern, `du -sh` first — - the allocated size is usually far smaller than it looks. -3. **Verify restores by opening them.** `Brainy.load(path)` opens any snapshot or restored - directory read-only — the store verifies its own coherence at open and reports loudly if - anything is missing or torn. diff --git a/docs/guides/find-limits.md b/docs/guides/find-limits.md deleted file mode 100644 index 4c7fd252..00000000 --- a/docs/guides/find-limits.md +++ /dev/null @@ -1,153 +0,0 @@ ---- -title: Query Limits & Pagination -slug: guides/find-limits -public: true -category: guides -template: guide -order: 8 -description: How Brainy caps `find({ limit })` to prevent OOM, the three escape valves when the cap is too tight, and why pagination is the future-proof pattern. -next: - - guides/aggregation - - api/reference ---- - -# Query Limits & Pagination - -Brainy's `find()` returns entities into a JavaScript array. The size of that array is bounded by an auto-configured cap so a single query can never run the host out of memory. This guide explains the cap, the three ways to raise it when your use case justifies it, and the one pattern that scales no matter what cap is in effect: pagination. - -## Why the cap exists - -Every entity Brainy returns carries: - -- A 384-dim float32 embedding vector (1.5 KB) -- Standard fields: `id`, `type`, `subtype`, timestamps, confidence, weight (~200 bytes) -- User metadata (variable — typical 5-10 KB, can spike to 20+ KB) - -Conservative budget: **25 KB per result**. A `find({ limit: 100_000 })` against a brain with rich metadata can claim ~2.5 GB before Brainy's iteration starts. JavaScript's GC + V8's heap targets can't absorb that swing without paging or OOM in production. - -The cap is a safety net. It's not the only reason your query might be slow — graph traversal and HNSW search have their own perf characteristics — but it's the one that turns a slow query into a sudden runtime error. - -## The auto-configured cap (7.30.2+) - -Brainy picks `maxLimit` from the first of these that's available: - -| Priority | Source | Formula | -|---|---|---| -| 1 | Constructor option `maxQueryLimit` | Hard cap at supplied value, max 100 000 | -| 2 | Constructor option `reservedQueryMemory` | `floor(reservedQueryMemory / 25 KB)` capped at 100 000 | -| 3 | Detected container memory limit (Cloud Run, Kubernetes, cgroups v1/v2) | `floor(containerLimit × 0.25 / 25 KB)` capped at 100 000 | -| 4 | Free system memory | `floor(availableMemory / 25 KB)` capped at 100 000 | - -Worked example: a 4 GB Cloud Run container picks priority 3 → `floor(4 GB × 0.25 / 25 KB) = floor(40 960) = 40 000` results. A 900 MB free-memory box on priority 4 gets `floor(900 MB / 25 KB) = ~36 000`. - -The cap is fixed at construction and never changes at runtime. Query timing is recorded -for diagnostics only — a burst of slow queries cannot silently shrink the cap, and the -auto-detected tiers (3 and 4) never go below a floor of 10 000. - -> **Calibration note.** Pre-7.30.2 used 100 KB per result instead of 25 KB, which produced caps that were 4× too tight for typical workloads (an 8 KB / result reality). 7.30.2 recalibrated to match observed entity sizes; existing `limit: 10_000` safety patterns now pass silently on any reasonably-sized box. - -## What happens when you exceed the cap - -`find({ limit })` enforces in **two tiers**: - -### Soft tier: `maxLimit < limit ≤ 2 × maxLimit` - -You get a one-time warning per call site: - -``` -[Brainy] find({ limit: 50000 }) exceeds the auto-configured query limit of -40000 (basis: detected container memory limit). Choose one: - • Increase the cap: new Brainy({ maxQueryLimit: 50000 }) - • Reserve more memory: new Brainy({ reservedQueryMemory: 1310720000 }) - • Paginate: split the query with { limit, offset } pages - at YourService.loadDashboard (/app/src/dashboard.ts:142:18) -Docs: https://soulcraft.com/docs/guides/find-limits -``` - -**The query proceeds.** Brainy returns the result set you asked for; the warning is a teaching signal, not a block. Existing code that relied on the cap silently allowing safety-cap limits (`limit: 10_000` against a 9 K-cap box) keeps working — the warning shows you the recipe so you can fix it intentionally. - -### Hard tier: `limit > 2 × maxLimit` - -Same message, but thrown as an error. This is real OOM territory; the cap stops being a recommendation and becomes a guardrail. - -## The three escape valves - -### 1. Raise the cap at construction — `maxQueryLimit` - -When the auto-config is wrong for your workload (e.g. you know your entities are smaller than 25 KB average and you need bigger result sets), set an explicit cap: - -```typescript -const brain = new Brainy({ - storage: { type: 'filesystem', path: './data' }, - maxQueryLimit: 50_000 // raises the cap; still hard-clamped at 100 000 -}) -``` - -This is the right answer when: -- Your entity metadata is genuinely small (e.g. 1-2 KB) and 25 KB per result is over-conservative -- You're running on a box with lots of headroom and 25% of memory underestimates what you can spare for queries -- You need a known-good limit that doesn't change when the box's free-memory wiggles at startup - -### 2. Reserve more memory for queries — `reservedQueryMemory` - -When you want the cap to be memory-derived but more generous than the default 25% slice: - -```typescript -const brain = new Brainy({ - reservedQueryMemory: 1024 * 1024 * 1024 // 1 GB → ~40 000 result cap -}) -``` - -This is the right answer when: -- Your host's memory budget for queries is known and stable, regardless of free-memory at startup -- You want the formula to scale with the documented per-result size (25 KB) instead of a hard number - -### 3. Paginate — the future-proof pattern - -If your query genuinely needs to walk all matches in a category, don't fight the cap — walk in pages: - -```typescript -async function findAll(params: FindParams, pageSize = 1000): Promise[]> { - const all: Result[] = [] - let offset = 0 - while (true) { - const page = await brain.find({ ...params, limit: pageSize, offset }) - all.push(...page) - if (page.length < pageSize) break - offset += page.length - } - return all -} - -// Use it just like find(): -const allEvents = await findAll({ type: NounType.Event, where: { status: 'open' } }) -``` - -For very large brains, prefer the streaming API which avoids holding the full result set in memory at all: - -```typescript -for await (const entity of brain.streaming.entities({ type: NounType.Event })) { - // process one entity at a time -} -``` - -## When to use which - -| Situation | Recommended valve | -|---|---| -| The cap is unreasonably low for your known entity size | `maxQueryLimit` | -| You want a memory-derived cap but more generous than 25% | `reservedQueryMemory` | -| Your query needs ALL matches in a category | Pagination or `brain.streaming.entities()` | -| You hit the cap once during a one-off migration | `maxQueryLimit` or `migrateField` (which already paginates internally) | -| You're hitting the cap on a recurring user-facing query | Pagination — the cap will get tighter in 8.0, not looser | - -## A note on Brainy 8.0 - -8.0's Datomic-style `Db` API may make per-call limits stricter to keep snapshot semantics cheap. **Pagination is the only pattern that's guaranteed to keep working unchanged.** Code that paginates today doesn't need to revisit when 8.0 ships. - -## Reference - -- `BrainyConfig.maxQueryLimit?: number` — explicit cap override (max 100 000) -- `BrainyConfig.reservedQueryMemory?: number` — memory budget for queries (bytes) -- `find({ limit, offset })` — paginated find -- `brain.streaming.entities(filter)` — streaming alternative for very large traversals diff --git a/docs/guides/framework-integration.md b/docs/guides/framework-integration.md index 984466c5..23364178 100644 --- a/docs/guides/framework-integration.md +++ b/docs/guides/framework-integration.md @@ -1,17 +1,15 @@ # Framework Integration Guide -Brainy is **framework-friendly** - designed to drop into the server side of any modern JavaScript framework. This guide shows you how to integrate Brainy into framework-based apps. +Brainy 3.0 is **framework-first** - designed from the ground up to work seamlessly with modern JavaScript frameworks. This guide shows you how to integrate Brainy into any framework. -> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun (server-side only). It is not a browser library. In framework apps, run Brainy in API routes, server components, server actions, loaders, or a dedicated backend service - never in client-side bundles. +## 🎯 Why Framework-First? -## 🎯 Why Server-Side? - -Brainy embeds an HNSW vector index, a graph engine, and a filesystem-backed persistence layer. These belong on the server: +Traditional AI databases require complex browser polyfills and bundler configurations. Brainy 3.0 trusts your framework to handle this: - **Zero configuration**: Just `import { Brainy } from '@soulcraft/brainy'` -- **Auto storage detection**: `new Brainy()` auto-selects filesystem persistence on Node -- **Cleaner code**: No browser polyfills, no conditional client/server imports -- **Better DX**: One instance shared across your server routes +- **Framework responsibility**: Let Next.js, Vite, Webpack handle Node.js polyfills +- **Cleaner code**: No browser-specific entry points or conditional imports +- **Better DX**: Same API everywhere - browser, server, edge ## 🚀 Quick Start @@ -26,8 +24,7 @@ npm install @soulcraft/brainy ```javascript import { Brainy } from '@soulcraft/brainy' -// Run on the server (API route, server component, backend service) -// new Brainy() auto-detects filesystem persistence on Node +// Works in any framework! const brain = new Brainy() await brain.init() @@ -44,48 +41,52 @@ const results = await brain.find("framework integration") ## ⚛️ React Integration -Brainy runs on the server, so a React client component talks to it through an API endpoint (see the Next.js API route below). The hook fetches results; it never instantiates Brainy in the browser. - ### Basic Hook Pattern ```jsx -import { useState, useCallback } from 'react' +import { useState, useEffect, useCallback } from 'react' +import { Brainy } from '@soulcraft/brainy' -function useBrainySearch(endpoint = '/api/search') { - const [results, setResults] = useState([]) - const [loading, setLoading] = useState(false) +function useBrainy() { + const [brain, setBrain] = useState(null) + const [isReady, setIsReady] = useState(false) - const search = useCallback(async (query) => { - if (!query) return - setLoading(true) - try { - const res = await fetch(endpoint, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ query }) + useEffect(() => { + const initBrain = async () => { + const newBrain = new Brainy({ + storage: { type: 'opfs' } // Browser storage }) - const { results } = await res.json() - setResults(results) - } finally { - setLoading(false) + await newBrain.init() + setBrain(newBrain) + setIsReady(true) } - }, [endpoint]) - return { results, loading, search } + initBrain() + }, []) + + return { brain, isReady } } // Usage in component function SearchComponent() { - const { results, loading, search } = useBrainySearch() + const { brain, isReady } = useBrainy() + const [results, setResults] = useState([]) + + const handleSearch = useCallback(async (query) => { + if (!isReady) return + const searchResults = await brain.find(query) + setResults(searchResults) + }, [brain, isReady]) + + if (!isReady) return
Loading AI...
return (
search(e.target.value)} + onChange={(e) => handleSearch(e.target.value)} /> - {loading &&
Searching...
}
{results.map(result => (
@@ -99,34 +100,48 @@ function SearchComponent() { } ``` -### Shared Server Instance +### React Context Pattern -On the server, create one Brainy instance and reuse it across requests. This module is imported only by server code (API routes, server components), never by client components: - -```javascript -// lib/brain.server.js +```jsx +import React, { createContext, useContext, useEffect, useState } from 'react' import { Brainy } from '@soulcraft/brainy' -let brainPromise +const BrainyContext = createContext() -export function getBrain() { - if (!brainPromise) { - brainPromise = (async () => { - // new Brainy() auto-detects filesystem persistence on Node - const brain = new Brainy() - await brain.init() - return brain - })() +export function BrainyProvider({ children }) { + const [brain, setBrain] = useState(null) + const [isReady, setIsReady] = useState(false) + + useEffect(() => { + const initBrain = async () => { + const newBrain = new Brainy() + await newBrain.init() + setBrain(newBrain) + setIsReady(true) + } + + initBrain() + }, []) + + return ( + + {children} + + ) +} + +export function useBrainContext() { + const context = useContext(BrainyContext) + if (!context) { + throw new Error('useBrainContext must be used within BrainyProvider') } - return brainPromise + return context } ``` ## 🟢 Vue.js Integration -Vue components call a server endpoint (see the Nuxt server route in the [Vue.js Integration Guide](vue-integration.md)); Brainy itself runs on the server. - -### Composition API (client component) +### Composition API ```vue ``` -### Shared Server Instance - -On the server, create one Brainy instance and reuse it across requests: +### Vue 3 Plugin ```javascript -// server/brain.js (server-only module) +// plugins/brainy.js import { Brainy } from '@soulcraft/brainy' -let brainPromise +export default { + install(app, options) { + const brain = new Brainy(options) -export function getBrain() { - if (!brainPromise) { - brainPromise = (async () => { - // new Brainy() auto-detects filesystem persistence on Node - const brain = new Brainy() - await brain.init() - return brain - })() + app.config.globalProperties.$brain = brain + app.provide('brain', brain) + + // Initialize on app mount + brain.init() } - return brainPromise } + +// main.js +import { createApp } from 'vue' +import BrainyPlugin from './plugins/brainy' + +const app = createApp(App) +app.use(BrainyPlugin, { + storage: { type: 'opfs' } +}) ``` ## 🅰️ Angular Integration -The Angular service calls your backend over HTTP; Brainy lives in that backend, not in the browser. - -### Service Pattern (calls the backend) +### Service Pattern ```typescript // brainy.service.ts import { Injectable } from '@angular/core' -import { HttpClient } from '@angular/common/http' -import { Observable } from 'rxjs' +import { BehaviorSubject, Observable } from 'rxjs' +import { Brainy } from '@soulcraft/brainy' @Injectable({ providedIn: 'root' }) export class BrainyService { - constructor(private http: HttpClient) {} + private brain: Brainy + private readySubject = new BehaviorSubject(false) - search(query: string): Observable<{ results: any[] }> { - return this.http.post<{ results: any[] }>('/api/search', { query }) + ready$: Observable = this.readySubject.asObservable() + + constructor() { + this.initBrain() } - add(data: any, type: string, metadata?: any): Observable<{ id: string }> { - return this.http.post<{ id: string }>('/api/add', { data, type, metadata }) + private async initBrain() { + this.brain = new Brainy({ + storage: { type: 'opfs' } + }) + await this.brain.init() + this.readySubject.next(true) + } + + async search(query: string): Promise { + if (!this.readySubject.value) { + throw new Error('Brain not ready') + } + return await this.brain.find(query) + } + + async add(data: any, type: string, metadata?: any): Promise { + if (!this.readySubject.value) { + throw new Error('Brain not ready') + } + return await this.brain.add({ data, type, metadata }) } } ``` @@ -235,52 +280,64 @@ export class SearchComponent { constructor(private brainyService: BrainyService) {} - search() { + async search() { if (!this.query) return - this.brainyService.search(this.query).subscribe(({ results }) => { - this.results = results - }) + this.results = await this.brainyService.search(this.query) } } ``` -The matching backend endpoint uses Brainy directly (Node/Bun): - -```typescript -// server: api/search -import { Brainy } from '@soulcraft/brainy' - -const brain = new Brainy() // auto-detects filesystem persistence on Node -await brain.init() - -export async function handleSearch(query: string) { - return await brain.find(query) -} -``` - ## 🚀 Next.js Integration -In Next.js, Brainy lives in server code only: API routes, server components, or server actions. Create one shared instance in a server-only module. +### App Router (Next.js 13+) -### Shared Server Instance - -```javascript -// lib/brain.server.js (imported only by server code) +```jsx +// app/providers.jsx +'use client' +import { createContext, useContext, useEffect, useState } from 'react' import { Brainy } from '@soulcraft/brainy' -let brainPromise +const BrainyContext = createContext() -export function getBrain() { - if (!brainPromise) { - brainPromise = (async () => { - const brain = new Brainy({ - storage: { type: 'filesystem', path: './data' } - }) - await brain.init() - return brain - })() - } - return brainPromise +export function BrainyProvider({ children }) { + const [brain, setBrain] = useState(null) + const [isReady, setIsReady] = useState(false) + + useEffect(() => { + const initBrain = async () => { + const newBrain = new Brainy() + await newBrain.init() + setBrain(newBrain) + setIsReady(true) + } + + initBrain() + }, []) + + return ( + + {children} + + ) +} + +export const useBrainy = () => useContext(BrainyContext) +``` + +```jsx +// app/layout.jsx +import { BrainyProvider } from './providers' + +export default function RootLayout({ children }) { + return ( + + + + {children} + + + + ) } ``` @@ -288,78 +345,45 @@ export function getBrain() { ```javascript // app/api/search/route.js -import { getBrain } from '@/lib/brain.server' +import { Brainy } from '@soulcraft/brainy' + +const brain = new Brainy({ + storage: { type: 'filesystem', path: './data' } +}) +await brain.init() export async function POST(request) { const { query } = await request.json() - const brain = await getBrain() const results = await brain.find(query) return Response.json({ results }) } ``` -### Server Action - -```javascript -// app/actions.js -'use server' -import { getBrain } from '@/lib/brain.server' - -export async function search(query) { - const brain = await getBrain() - return await brain.find(query) -} -``` - -## 🔷 SvelteKit Integration - -Brainy runs in a server-only module (`*.server.js`); the component fetches results from an endpoint. - -```javascript -// src/lib/server/brain.js (server-only — note the .server suffix) -import { Brainy } from '@soulcraft/brainy' - -let brainPromise - -export function getBrain() { - if (!brainPromise) { - brainPromise = (async () => { - const brain = new Brainy() // auto-detects filesystem persistence - await brain.init() - return brain - })() - } - return brainPromise -} -``` - -```javascript -// src/routes/api/search/+server.js -import { json } from '@sveltejs/kit' -import { getBrain } from '$lib/server/brain' - -export async function POST({ request }) { - const { query } = await request.json() - const brain = await getBrain() - return json({ results: await brain.find(query) }) -} -``` +## 🔷 Svelte Integration ```svelte @@ -377,23 +401,27 @@ export async function POST({ request }) { ## 🌟 Solid.js Integration -The component calls a server endpoint (use SolidStart server routes, or any backend, to host Brainy): - ```jsx -import { createSignal } from 'solid-js' +import { createSignal, onMount } from 'solid-js' +import { Brainy } from '@soulcraft/brainy' function SearchComponent() { + const [brain, setBrain] = createSignal(null) + const [isReady, setIsReady] = createSignal(false) const [query, setQuery] = createSignal('') const [results, setResults] = createSignal([]) + onMount(async () => { + const newBrain = new Brainy() + await newBrain.init() + setBrain(newBrain) + setIsReady(true) + }) + const search = async () => { - if (!query()) return - const res = await fetch('/api/search', { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ query: query() }) - }) - setResults((await res.json()).results) + if (!isReady() || !query()) return + const searchResults = await brain().find(query()) + setResults(searchResults) } return ( @@ -422,25 +450,57 @@ function SearchComponent() { ## 📦 Bundler Configuration -Brainy is a server-side dependency, so keep it out of client bundles. Import it only from server-only modules (`*.server.js`, API routes, server components, server actions). If your bundler ever tries to pull Brainy into a client bundle, that's a sign it's being imported from a client component — move the import to a server module. - -For server builds, mark Brainy as external so the bundler doesn't inline it: +### Vite (Recommended) ```javascript -// vite.config.js (SSR build) +// vite.config.js import { defineConfig } from 'vite' export default defineConfig({ - ssr: { - external: ['@soulcraft/brainy'] + define: { + global: 'globalThis' + }, + optimizeDeps: { + include: ['@soulcraft/brainy'] } }) ``` +### Webpack + ```javascript -// rollup.config.js (server bundle) +// webpack.config.js +module.exports = { + resolve: { + fallback: { + "fs": false, + "path": require.resolve("path-browserify"), + "crypto": require.resolve("crypto-browserify") + } + }, + plugins: [ + new webpack.ProvidePlugin({ + global: 'global' + }) + ] +} +``` + +### Rollup + +```javascript +// rollup.config.js +import { nodeResolve } from '@rollup/plugin-node-resolve' +import commonjs from '@rollup/plugin-commonjs' + export default { - external: ['@soulcraft/brainy', 'node:fs', 'node:path', 'node:crypto'] + plugins: [ + nodeResolve({ + browser: true, + preferBuiltins: false + }), + commonjs() + ] } ``` @@ -448,24 +508,30 @@ export default { ### Server-Side Rendering -Instantiate Brainy on the server and feed its results into the rendered page. Never construct it in client code. - ```javascript -// Server-side data loading (framework loader / getServerSideProps / load fn) -import { getBrain } from './brain.server' +// Check if running in browser +if (typeof window !== 'undefined') { + // Browser-only code + const brain = new Brainy({ + storage: { type: 'opfs' } + }) +} -export async function load({ url }) { - const brain = await getBrain() - const query = url.searchParams.get('q') ?? '' - const results = query ? await brain.find(query) : [] - return { results } +// Or use dynamic imports +const initBrainForBrowser = async () => { + if (typeof window === 'undefined') return null + + const { Brainy } = await import('@soulcraft/brainy') + const brain = new Brainy() + await brain.init() + return brain } ``` ### Static Site Generation ```javascript -// For build-time usage (runs in Node during the build) +// For build-time usage import { Brainy } from '@soulcraft/brainy' export async function generateStaticProps() { @@ -474,8 +540,8 @@ export async function generateStaticProps() { }) await brain.init() - // Build search index (paginate with { limit, offset } for larger stores) - const allContent = await brain.find({ limit: 1000 }) + // Build search index + const allContent = await brain.export() return { props: { searchIndex: allContent } @@ -486,46 +552,67 @@ export async function generateStaticProps() { ## 🔧 Framework-Specific Tips ### React -- Keep components client-side and call a Brainy-backed API route -- Use `useCallback` for fetch handlers to prevent re-renders -- Debounce keystroke-driven searches before hitting the endpoint +- Use `useCallback` for search functions to prevent re-renders +- Consider `useMemo` for expensive brain operations +- Implement cleanup in `useEffect` for proper memory management ### Vue -- Components call an endpoint; the shared instance lives in a server module -- Consider Pinia for caching results client-side -- Debounce reactive search queries +- Use `shallowRef` for the brain instance (it's not reactive data) +- Consider Pinia for global brain state management +- Use `watchEffect` for reactive search queries ### Angular -- Use `HttpClient` and RxJS to call the backend -- Hold the shared Brainy instance in your Node backend, not the app -- Consider lazy loading search features in feature modules +- Implement proper dependency injection with services +- Use RxJS observables for reactive search +- Consider lazy loading brain in feature modules ### Next.js -- Put Brainy in server-only modules (`*.server.js`), API routes, or server actions -- Reuse one shared instance across requests -- Implement proper error boundaries for failed fetches +- Use dynamic imports for client-side only features +- Consider API routes for server-side brain operations +- Implement proper error boundaries ## 🚨 Common Issues & Solutions -### Issue: "fs module not found" / "crypto is not defined" in the browser -**Cause**: Brainy was imported into a client bundle. Brainy 8.0 is a server-side library (Node 22+/Bun) and uses Node built-ins like `fs` and `crypto`. -**Solution**: Import Brainy only from server code — server-only modules (`*.server.js`), API routes, server components, or server actions. From client components, call those endpoints instead. +### Issue: "crypto is not defined" +**Solution**: Your framework should handle this automatically. If not: +```javascript +// Add to your bundle config +define: { + global: 'globalThis' +} +``` -### Issue: Large client bundle size -**Cause**: A client module is pulling in Brainy. -**Solution**: Move the `import { Brainy } from '@soulcraft/brainy'` into a server-only module so it never reaches the browser bundle. +### Issue: "fs module not found" +**Solution**: This is expected in browsers. Use browser-compatible storage: +```javascript +const brain = new Brainy({ + storage: { type: 'opfs' } // Or 'memory' for development +}) +``` + +### Issue: Large bundle size +**Solution**: Use dynamic imports for optional features: +```javascript +const brain = await import('@soulcraft/brainy').then(m => new m.Brainy()) +``` ### Issue: SSR hydration mismatch -**Solution**: Run the search on the server (loader / server action / API route) and pass the results down as props, so server and client render the same markup. +**Solution**: Initialize brain only on client: +```javascript +useEffect(() => { + // Browser-only initialization + initBrain() +}, []) +``` ## 🎯 Best Practices -1. **Initialize Once**: Create one shared Brainy instance per server process, not per request -2. **Server-Only**: Import Brainy only from server modules — never from client components -3. **Endpoint Boundary**: Expose search/add through API routes or server actions -4. **Handle Loading**: Show loading states in the client while the fetch is in flight -5. **Error Handling**: Catch and surface failed endpoint calls gracefully -6. **Storage**: Use `filesystem` for persistence (the default on Node) or `memory` for ephemeral/tests +1. **Initialize Once**: Create brain instance at app level, not component level +2. **Use Context**: Share brain instance across components with context/providers +3. **Handle Loading**: Always show loading states during brain initialization +4. **Error Boundaries**: Implement proper error handling for brain operations +5. **Memory Management**: Clean up brain instances on unmount +6. **Storage Strategy**: Choose appropriate storage for your deployment target ## 📚 Next Steps diff --git a/docs/guides/getting-started.md b/docs/guides/getting-started.md new file mode 100644 index 00000000..0257835d --- /dev/null +++ b/docs/guides/getting-started.md @@ -0,0 +1,333 @@ +# Getting Started with Brainy + +This guide will help you get up and running with Brainy, the multi-dimensional AI database that combines vector similarity, graph relationships, and metadata filtering. + +## Installation + +```bash +npm install @soulcraft/brainy +``` + +## Basic Setup + +### Simple Initialization + +```typescript +import { Brainy } from '@soulcraft/brainy' + +// Create a new Brainy instance with defaults +const brain = new Brainy() + +// Initialize (downloads models if needed) +await brain.init() + +// You're ready to go! +``` + +### Custom Configuration + +```typescript +const brain = new Brainy({ + // Storage configuration + storage: { + type: 'filesystem', // or 's3', 'opfs' + path: './my-data' + }, + + // Vector configuration + vectors: { + dimensions: 384, + model: 'all-MiniLM-L6-v2' + }, + + // Performance tuning + cache: { + enabled: true, + maxSize: 1000 + } +}) + +await brain.init() +``` + +## Your First Operations + +### Adding Data + +```typescript +// Add entities (nouns) with automatic embedding generation +const id = await brain.add("The quick brown fox jumps over the lazy dog", { + category: "demo", + timestamp: Date.now() +}) + +console.log(`Added noun with ID: ${id}`) + +// Add relationships (verbs) between entities +const sourceId = await brain.add("John Smith", { nounType: 'person' }) +const targetId = await brain.add("TechCorp", { nounType: 'organization' }) +await brain.relate(sourceId, targetId, "works_at", { + position: "Engineer", + since: "2024" +}) +``` + +### Searching + +```typescript +// Simple semantic search +const results = await brain.search("fast animals") + +results.forEach(result => { + console.log(`Found: ${result.content} (score: ${result.score})`) +}) +``` + +### Advanced Queries with find() + +```typescript +// Natural language queries - Brainy understands intent! +const results = await brain.find("show me technology articles about AI from 2023") +// Automatically interprets: topic, category, and time range + +// Structured queries with vector similarity and metadata filtering +const structured = await brain.find({ + like: "artificial intelligence", + where: { + category: "technology", + year: { $gte: 2023 } + }, + limit: 10 +}) + +// Complex natural language with multiple filters +const complex = await brain.find("financial reports from Q3 2024 with revenue over 1M") +// Automatically extracts: document type, date range, numeric filters +``` + +## Common Use Cases + +### 1. Semantic Search Engine + +```typescript +// Index documents +const documents = [ + { title: "Introduction to AI", content: "AI is transforming..." }, + { title: "Machine Learning Basics", content: "ML algorithms..." }, + { title: "Deep Learning", content: "Neural networks..." } +] + +for (const doc of documents) { + await brain.add(doc.content, { + title: doc.title, + type: "document" + }) +} + +// Search semantically +const results = await brain.search("how do neural networks work") +``` + +### 2. Recommendation System + +```typescript +// Add user interactions as nouns +const interactionId = await brain.add("user viewed product", { + userId: "user123", + productId: "product456", + action: "view", + timestamp: Date.now() +}) + +// Create relationships between users and products +const userId = await brain.add("user123", { nounType: 'user' }) +const productId = await brain.add("product456", { nounType: 'product' }) +await brain.relate(userId, productId, "viewed", { + timestamp: Date.now() +}) + +// Natural language query for recommendations +const recommendations = await brain.find("products similar to what user123 viewed recently") + +// Or structured query for similar users +const similar = await brain.find({ + like: "user123 interests", + where: { action: "view" }, + limit: 5 +}) +``` + +### 3. Knowledge Graph + +```typescript +// Add entities (nouns) to the knowledge graph +const personId = await brain.add("John Smith, Software Engineer", { + type: "person", + role: "engineer" +}) + +const companyId = await brain.add("TechCorp, Innovation Leader", { + type: "company", + industry: "technology" +}) + +// Create relationship +await brain.relate(personId, companyId, "works_at", { + since: "2020", + position: "Senior Engineer" +}) + +// Natural language query for relationships +const colleagues = await brain.find("people who work at TechCorp") + +// Or structured query for specific relationships +const results = await brain.find({ + connected: { + from: personId, + type: "works_at" + } +}) +``` + +### 4. Real-time Data Processing + +```typescript +// Configure for streaming +const brain = new Brainy({ + augmentations: [ + new EntityRegistryAugmentation(), // Deduplication + new BatchProcessingAugmentation({ batchSize: 100 }) // Batching + ] +}) + +// Process streaming data +async function processStream(item) { + // Entity registry prevents duplicate nouns + const id = await brain.add(item.content, { + externalId: item.id, + timestamp: item.timestamp + }) + + // Real-time natural language queries + if (item.urgent) { + const related = await brain.find(`urgent items similar to ${item.content}`) + // Process related items... + } +} +``` + +## Storage Options + +### Development (FileSystem) +```typescript +const brain = new Brainy({ + storage: { type: 'filesystem', path: '/tmp/brainy-dev' } +}) +// Fast, persistent, perfect for testing +``` + +### Production (FileSystem) +```typescript +const brain = new Brainy({ + storage: { + type: 'filesystem', + path: '/var/lib/brainy' + } +}) +// Persistent, efficient, server-ready +``` + +### Cloud (S3) +```typescript +const brain = new Brainy({ + storage: { + type: 's3', + bucket: 'my-brainy-data', + region: 'us-east-1' + } +}) +// Scalable, distributed, cloud-native +``` + +### Browser (OPFS) +```typescript +const brain = new Brainy({ + storage: { type: 'opfs' } +}) +// Browser-native, persistent, offline-capable +``` + +## Performance Tips + +### 1. Use Batch Operations +```typescript +// Good - batch operations for nouns +const items = ["item1", "item2", "item3"] +for (const item of items) { + await brain.add(item, { batch: true }) +} + +// Create relationships efficiently +const relationships = [ + { source: id1, target: id2, type: "related" }, + { source: id2, target: id3, type: "similar" } +] +for (const rel of relationships) { + await brain.relate(rel.source, rel.target, rel.type) +} +``` + +### 2. Enable Caching +```typescript +const brain = new Brainy({ + cache: { + enabled: true, + maxSize: 1000, + ttl: 300000 // 5 minutes + } +}) +``` + +### 3. Use Appropriate Limits +```typescript +// Always specify reasonable limits +const results = await brain.search("query", { + limit: 20 // Don't fetch more than needed +}) +``` + +### 4. Index Frequently Queried Fields +```typescript +const brain = new Brainy({ + indexedFields: ['category', 'userId', 'timestamp'] +}) +``` + +## Error Handling + +```typescript +try { + await brain.add("content", metadata) +} catch (error) { + if (error.code === 'STORAGE_FULL') { + console.error('Storage is full') + } else if (error.code === 'INVALID_INPUT') { + console.error('Invalid input:', error.message) + } else { + console.error('Unexpected error:', error) + } +} +``` + +## Next Steps + +- [Architecture Overview](../architecture/overview.md) - Understand the system design +- [Triple Intelligence](../architecture/triple-intelligence.md) - Advanced query capabilities +- [API Reference](../api/README.md) - Complete API documentation +- [Examples](https://github.com/brainy-org/brainy/tree/main/examples) - More code examples + +## Getting Help + +- **Issues**: [GitHub Issues](https://github.com/brainy-org/brainy/issues) +- **Discussions**: [GitHub Discussions](https://github.com/brainy-org/brainy/discussions) +- **Examples**: Check the `/examples` directory \ No newline at end of file diff --git a/docs/guides/import-anything.md b/docs/guides/import-anything.md index b1bb15ef..94049a96 100644 --- a/docs/guides/import-anything.md +++ b/docs/guides/import-anything.md @@ -3,8 +3,8 @@ Brainy's import is **ONE magical method** that understands EVERYTHING: - 📊 Data (objects, arrays, strings) - 📁 Files (auto-detects by path) -- 🌐 URLs (auto-fetches with authentication support) -- 📄 Formats (JSON, CSV, Excel, PDF, YAML, DOCX, Markdown - all auto-detected) +- 🌐 URLs (auto-fetches) +- 📄 Formats (JSON, CSV, Excel, PDF, YAML, text - all auto-detected) ## The Ultimate Simplicity @@ -24,8 +24,8 @@ await brain.import(anything) ```javascript // Array of objects? No problem. const people = [ - { name: 'Alice', role: 'Engineer', company: 'TechCorp' }, - { name: 'Bob', role: 'Designer', company: 'TechCorp' } + { name: 'Alice', role: 'Engineer', company: 'TechCorp' }, + { name: 'Bob', role: 'Designer', company: 'TechCorp' } ] await brain.import(people) @@ -49,23 +49,28 @@ await brain.import(csv, { format: 'csv' }) ### 📊 Import Excel - Multi-Sheet Support ```javascript -// Import entire Excel workbook — every sheet is processed automatically +// Import entire Excel workbook await brain.import('sales-report.xlsx') // ✨ Processes all sheets, preserves structure, infers types! -// Mirror the workbook into the VFS, grouped by sheet +// Or specific sheets only await brain.import('data.xlsx', { - vfsPath: '/imports/data', - groupBy: 'sheet' + excelSheets: ['Customers', 'Orders'] }) // ✨ Multi-sheet data becomes interconnected entities! ``` ### 📑 Import PDF - Text & Tables ```javascript -// Import PDF documents — text and tables are extracted automatically +// Import PDF documents await brain.import('research-paper.pdf') // ✨ Extracts text, detects tables, preserves metadata! + +// With table extraction +await brain.import('report.pdf', { + pdfExtractTables: true +}) +// ✨ Converts PDF tables to structured data automatically! ``` ### 📝 Import YAML - File or String @@ -78,34 +83,15 @@ await brain.import('config.yaml') const yaml = ` project: AI Assistant team: - - name: Alice - role: Lead - - name: Bob - role: Dev + - name: Alice + role: Lead + - name: Bob + role: Dev ` await brain.import(yaml, { format: 'yaml' }) // ✨ Hierarchical data becomes a connected graph! ``` -### 📄 Import Word Documents (DOCX) - -```javascript -// From file path -await brain.import('research-paper.docx') -// ✨ Extracts text, headings, tables, and metadata! - -// Or from buffer -const buffer = fs.readFileSync('document.docx') -await brain.import(buffer, { format: 'docx' }) -// ✨ Uses heading hierarchy for entity organization! - -// With neural extraction -await brain.import('report.docx', { - enableNeuralExtraction: true, - enableHierarchicalRelationships: true -}) -// ✨ Extracts entities from paragraphs and creates relationships within sections! -``` - ### 🌐 Import from URLs - Auto-Detected! ```javascript // Just pass the URL - it knows! @@ -115,28 +101,6 @@ await brain.import('https://api.example.com/data.json') // Works with any URL await brain.import('https://data.gov/census.csv') // ✨ Fetches CSV from web, parses, imports! - -// With authentication -await brain.import({ - type: 'url', - data: 'https://api.example.com/private/data.xlsx', - auth: { - username: 'user', - password: 'pass' - } -}) -// ✨ Supports basic authentication for protected resources! - -// With custom headers -await brain.import({ - type: 'url', - data: 'https://api.example.com/data.json', - headers: { - 'Authorization': 'Bearer TOKEN', - 'X-API-Key': 'your-key' - } -}) -// ✨ Full HTTP header customization support! ``` ### 📖 Import Plain Text @@ -154,13 +118,12 @@ await brain.import(article, { format: 'text' }) When you import data, Brainy: -1. **Auto-detects format** - CSV, Excel, PDF, JSON, YAML, DOCX, Markdown, or by file extension -2. **Intelligent parsing** - CSV (encoding/delimiter), Excel (multi-sheet), PDF (text/tables), DOCX (headings/paragraphs) +1. **Auto-detects format** - CSV, Excel, PDF, JSON, YAML, text, or by file extension +2. **Intelligent parsing** - CSV (encoding/delimiter detection), Excel (multi-sheet), PDF (text/tables) 3. **Identifies entity types** - Uses AI to classify as Person, Document, Product, etc. (31 types!) 4. **Finds relationships** - Detects connections like "belongsTo", "createdBy", "references" (40 types!) -5. **Scores confidence & weight** - Every entity and relationship gets quality metrics -6. **Creates embeddings** - Makes everything semantically searchable -7. **Indexes metadata** - Enables lightning-fast filtering with range queries +5. **Creates embeddings** - Makes everything semantically searchable +6. **Indexes metadata** - Enables lightning-fast filtering ## Intelligent Type Detection @@ -171,7 +134,7 @@ Brainy automatically detects what TYPE of data you're importing: { name: 'John', email: 'john@example.com' } // This becomes an Organization -{ companyName: 'Acme', employees: 500 } +{ companyName: 'Acme', employees: 500 } // This becomes a Document { title: 'Report', content: '...', author: 'Jane' } @@ -180,7 +143,7 @@ Brainy automatically detects what TYPE of data you're importing: { latitude: 37.7, longitude: -122.4, city: 'SF' } ``` -**42 noun types and 127 verb types** cover EVERYTHING! +**31 noun types** and **40 verb types** cover EVERYTHING! ## Relationship Detection @@ -188,9 +151,9 @@ Brainy finds connections in your data: ```javascript const data = [ - { id: 'u1', name: 'Alice', managerId: 'u2' }, - { id: 'u2', name: 'Bob', departmentId: 'd1' }, - { id: 'd1', name: 'Engineering' } + { id: 'u1', name: 'Alice', managerId: 'u2' }, + { id: 'u2', name: 'Bob', departmentId: 'd1' }, + { id: 'd1', name: 'Engineering' } ] await brain.import(data) @@ -199,77 +162,22 @@ await brain.import(data) // - Bob "memberOf" Engineering ``` -## Confidence & Weight Scoring - -Every entity and relationship gets confidence and weight scores: - -```javascript -// Import with confidence threshold -await brain.import(data, { - confidenceThreshold: 0.8 // Only extract entities with >80% confidence -}) - -// Query high-confidence entities using range queries -const highConfidence = await brain.find({ - where: { - confidence: { gte: 0.8 } // Get entities with confidence >= 0.8 - } -}) - -// Range query operators: gt, gte, lt, lte, between -const mediumConfidence = await brain.find({ - where: { - confidence: { between: [0.6, 0.8] } - } -}) -``` - -**What do confidence scores mean?** -- **High (>0.8)**: Very confident entity classification -- **Medium (0.6-0.8)**: Reasonable confidence -- **Low (<0.6)**: Uncertain classification (filtered by default) - -**Weights** indicate importance/relevance within the document context. - -## Per-Sheet Excel Extraction - -Excel files with multiple sheets can be organized by sheet: - -```javascript -// Group entities by sheet in VFS -await brain.import('multi-sheet-data.xlsx', { - groupBy: 'sheet' // Creates separate directories for each sheet -}) - -// Result VFS structure: -// /imports/data/ -// ├── Sheet1/ -// │ ├── entity1.json -// │ └── entity2.json -// └── Sheet2/ -// ├── entity3.json -// └── entity4.json - -// Other groupBy options: -// - 'type': Group by entity type (Person, Place, etc.) -// - 'flat': All entities in one directory -// - 'custom': Use custom grouping function -``` - ## Query Your Imported Data Once imported, use Triple Intelligence to query: ```javascript // Vector search -const similar = await brain.find('engineers') +const similar = await brain.search('engineers') // Natural language const results = await brain.find('people in engineering who joined this year') // Graph traversal + filters const connected = await brain.find({ - like: 'Alice', - connected: { depth: 2 }, - where: { department: 'Engineering' } + like: 'Alice', + connected: { depth: 2 }, + where: { department: 'Engineering' } }) ``` @@ -279,40 +187,37 @@ Everything works with zero config, but you can customize: ```javascript await brain.import(data, { - // Format detection - format: 'excel', // Force specific format (auto-detected if not specified) + // Format detection + format: 'excel', // Force specific format (auto-detected if not specified) - // VFS & Organization - vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified) - groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom' - preserveSource: true, // Keep original source file in VFS (default: true) + // VFS & Organization + vfsPath: '/imports/my-data', // Where to store in VFS (auto-generated if not specified) + groupBy: 'type', // Group entities by: 'type' | 'sheet' | 'flat' | 'custom' + preserveSource: true, // Keep original source file in VFS (default: true) - // Entity & Relationship Creation - createEntities: true, // Create entities in knowledge graph (default: true) - createRelationships: true, // Create relationships in knowledge graph (default: true) + // Entity & Relationship Creation + createEntities: true, // Create entities in knowledge graph (default: true) + createRelationships: true, // Create relationships in knowledge graph (default: true) - // Neural Intelligence - enableNeuralExtraction: true, // Use AI to extract entities (default: true) - enableRelationshipInference: true, // Use AI to infer relationships (default: true) - enableConceptExtraction: true, // Extract concepts from text (default: true) - confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6) + // Neural Intelligence + enableNeuralExtraction: true, // Use AI to extract entities (default: true) + enableRelationshipInference: true, // Use AI to infer relationships (default: true) + enableConceptExtraction: true, // Extract concepts from text (default: true) + confidenceThreshold: 0.6, // Minimum confidence for entities (0-1, default: 0.6) - // Deduplication - enableDeduplication: true, // Check for duplicate entities (default: true) - deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85) - // Notes: false disables BOTH the inline merge and the background pass that - // runs ~5 min after the last import (merged duplicates are deleted). - // The inline pass auto-disables for imports >100 entities (O(n²) cost); - // the background pass still covers those unless the flag is false. + // Deduplication + enableDeduplication: true, // Check for duplicate entities (default: true) + deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85) + // Note: Auto-disabled for imports >100 entities - // Performance - chunkSize: 100, // Batch size for processing (default: varies by operation) + // Performance + chunkSize: 100, // Batch size for processing (default: varies by operation) - // History & Progress - enableHistory: true, // Track import history (default: true) - onProgress: (progress) => { // Progress callback - console.log(progress.stage, progress.message) - } + // History & Progress + enableHistory: true, // Track import history (default: true) + onProgress: (progress) => { // Progress callback + console.log(progress.stage, progress.message) + } }) ``` @@ -339,9 +244,9 @@ const results = await brain.import(problematicData) ### 🏢 Business Data ```javascript // Import ANY source - ONE method! -await brain.import('customers.csv') // File -await brain.import('https://api.co/orders') // URL -await brain.import(productsArray) // Data +await brain.import('customers.csv') // File +await brain.import('https://api.co/orders') // URL +await brain.import(productsArray) // Data // Now query across all of it! await brain.find('customers who bought products in Q4') @@ -378,15 +283,15 @@ await brain.find('posts by users following Alice with >10 comments') **Zero Configuration**: Works perfectly out of the box **Maximum Intelligence**: AI understands your data's meaning -**Universal Protocol**: 42 nouns × 127 verbs = ANY data model +**Universal Protocol**: 31 nouns × 40 verbs = ANY data model **Delightful DX**: Simple, clean, modern API ## The ONE Method Philosophy ```javascript // ONE method that understands EVERYTHING: -await brain.import(data) // Objects, arrays, strings -await brain.import('file.csv') // Files (auto-detected) +await brain.import(data) // Objects, arrays, strings +await brain.import('file.csv') // Files (auto-detected) await brain.import('http://..') // URLs (auto-fetched) // It ALWAYS knows what to do! ✨ diff --git a/docs/guides/import-flow.md b/docs/guides/import-flow.md deleted file mode 100644 index bd6fd5b3..00000000 --- a/docs/guides/import-flow.md +++ /dev/null @@ -1,1907 +0,0 @@ -# 🎯 The Complete Import Flow Guide - -> **What happens when you import data into Brainy?** -> Follow the journey of a single Excel row as it transforms into intelligent, queryable knowledge. - ---- - -## 📋 Table of Contents - -1. [The Big Picture](#the-big-picture) -2. [The Journey Begins: Your Data](#the-journey-begins-your-data) -3. [Phase 1: Entry Point](#phase-1-entry-point) -4. [Phase 2: Orchestration](#phase-2-orchestration) -5. [Phase 3: Neural Extraction](#phase-3-neural-extraction-the-magic) -6. [Phase 4: VFS Structure](#phase-4-vfs-structure-creation) -7. [Phase 5: Knowledge Graph](#phase-5-knowledge-graph-creation) -8. [Phase 6: Persistence](#phase-6-persistence-and-finalization) -9. [What Gets Created](#what-gets-created-in-brainy) -10. [Performance & Scale](#performance--scale) - ---- - -## The Big Picture - -When you call `brain.import()`, your data goes through a **6-phase transformation pipeline**: - -``` -Excel File → Format Detection → Neural Extraction → VFS Structure → Knowledge Graph → Persistence -``` - -Each phase adds intelligence and structure to your raw data, transforming it into a queryable knowledge graph with: -- ✅ **Intelligent entity classification** (Person, Product, Concept, etc.) -- ✅ **Smart relationship inference** (CreatedBy, LocatedAt, PartOf, etc.) -- ✅ **Dual storage** (human-readable VFS + high-performance graph) -- ✅ **Vector embeddings** for semantic search -- ✅ **Automatic deduplication** across imports - -**Processing Time**: ~600ms for 10 entities, ~1.8s for 100 entities (with all features enabled) - ---- - -## 🌊 Always-On Streaming Architecture - -All imports use streaming with **progressive flush intervals**: - -### How It Works -- Periodic index flushes during import (automatic) -- Data queryable progressively as import proceeds -- Progressive intervals adjust as import grows -- Works for known and unknown totals -- Minimal overhead (~0.3%) - -### Progressive Flush Intervals - -| Current Count | Flush Interval | Reason | -|---------------|----------------|--------| -| 0-999 entities | Every 100 | Frequent early updates for UX | -| 1K-9.9K | Every 1000 | Balanced performance | -| 10K+ | Every 5000 | Minimal overhead | - -**Key Difference**: Intervals adjust based on **current** entity count (not total), so it works for streaming APIs where total is unknown. - -**Example Usage:** -```typescript -await brain.import(file, { - onProgress: async (progress) => { - // Query data as it's imported - if (progress.queryable) { - const products = await brain.find({ type: 'product', limit: 10000 }) - console.log(`${products.length} products imported so far...`) - } - } -}) -``` - -**Full details**: See [Streaming Imports Guide](./streaming-imports.md) - ---- - -## The Journey Begins: Your Data - -Let's follow a **single Excel row** through the entire pipeline. - -**Input File**: `glossary.xlsx` - -| Term | Definition | Type | Related Terms | -|---------------|-----------------------------------------------|-------------|--------------------| -| Mona Lisa | Famous painting created by Leonardo da Vinci | Product | Leonardo, Louvre | - -**Our Goal**: Transform this into: -1. A `Product` entity with semantic embedding -2. `CreatedBy` relationship to Leonardo da Vinci -3. `RelatedTo` relationships to Leonardo and Louvre -4. Organized VFS structure -5. Queryable knowledge graph - -Let's watch it happen! 🚀 - ---- - -## Phase 1: Entry Point - -**Location**: `src/brainy.ts:1952` - -### What You Write - -```typescript -const result = await brain.import(excelBuffer, { - format: 'excel', - vfsPath: '/imports/glossary', - enableNeuralExtraction: true, - enableRelationshipInference: true, - createEntities: true, - createRelationships: true -}) -``` - -### What Happens - -```typescript -// 1. Lazy load ImportCoordinator (not loaded until first import!) -const { ImportCoordinator } = await import('./import/ImportCoordinator.js') - -// 2. Create coordinator and initialize all 7 Smart importers -const coordinator = new ImportCoordinator(this) -await coordinator.init() // Loads: Excel, PDF, CSV, JSON, Markdown, YAML, DOCX importers - -// 3. Delegate to coordinator -return await coordinator.import(source, options) -``` - -**Why Lazy Load?** If you never import files, the entire import subsystem stays unloaded, saving ~2MB of memory and ~100ms startup time. - -**Progress Callback**: First event fires! -```typescript -{ stage: 'detecting', message: 'Detecting format...' } -``` - ---- - -## Phase 2: Orchestration - -**Location**: `src/import/ImportCoordinator.ts:273` - -The ImportCoordinator is the **traffic controller** for all imports. It handles: -- Format detection -- Routing to the right importer -- VFS structure generation -- Knowledge graph creation -- Progress tracking - -### Step 2.1: Source Normalization - -```typescript -const normalizedSource = await this.normalizeSource(source, options.format) -``` - -**Output**: -```typescript -{ - type: 'buffer', - data: Buffer<89 50 4e 47 0d 0a 1a 0a...>, // Raw Excel bytes - filename: undefined -} -``` - -The normalizer handles **5 source types**: -- `Buffer` → Direct binary data -- `string` → Could be URL, file path, or content -- `object` → JSON data -- `path` → File system path (reads file) -- `url` → HTTP(S) URL (fetches content) - -### Step 2.2: Format Detection - -```typescript -const detection = this.detectFormat(normalizedSource) -``` - -**How Detection Works**: -1. Checks magic bytes: `50 4b 03 04` = ZIP (Excel is ZIP-based) -2. Inspects file structure -3. Falls back to content analysis - -**Output**: -```typescript -{ - format: 'excel', - confidence: 1.0, - evidence: ['Explicitly specified', 'Magic bytes: ZIP container', 'Contains xl/workbook.xml'] -} -``` - -### Step 2.3: Route to Smart Importer - -```typescript -const extractionResult = await this.extract(normalizedSource, 'excel', options) -``` - -This calls `SmartExcelImporter.extract()` - where the **real magic happens**! ✨ - ---- - -## Phase 3: Neural Extraction (The Magic!) - -**Location**: `src/importers/SmartExcelImporter.ts:154` - -This is where your raw data becomes **intelligent knowledge**. Let's trace our "Mona Lisa" row through each step. - -### Step 3.1: Parse Excel File - -```typescript -const processedData = await this.excelHandler.process(buffer, options) -``` - -**Input**: Binary Excel file -**Output**: Array of row objects - -```typescript -const rows = [ - { - 'Term': 'Mona Lisa', - 'Definition': 'Famous painting created by Leonardo da Vinci', - 'Type': 'Product', - 'Related Terms': 'Leonardo, Louvre' - } - // ... more rows -] -``` - -### Step 3.2: Detect Column Structure - -```typescript -const columns = this.detectColumns(rows[0], opts) -``` - -The importer is **smart about column names**. It matches patterns: - -| Column Header | Matches Pattern | Maps To | -|----------------|--------------------------------------|------------------| -| `Term` | `term\|name\|title\|concept\|entity` | `columns.term` | -| `Definition` | `definition\|description\|desc` | `columns.definition` | -| `Type` | `type\|category\|kind\|class` | `columns.type` | -| `Related Terms`| `related\|see also\|links` | `columns.related`| - -**Output**: -```typescript -{ - term: 'Term', - definition: 'Definition', - type: 'Type', - related: 'Related Terms' -} -``` - -### Step 3.3: Batched Parallel Processing - -**The Bottleneck**: Processing 1000 rows sequentially would take ~200 seconds. - -**The Solution**: Process 10 rows at a time in parallel! - -```typescript -const CHUNK_SIZE = 10 // Process 10 rows simultaneously - -for (let chunkStart = 0; chunkStart < rows.length; chunkStart += CHUNK_SIZE) { - const chunk = rows.slice(chunkStart, chunkStart + CHUNK_SIZE) - - // Process entire chunk in parallel - const chunkResults = await Promise.all( - chunk.map(row => this.processRow(row)) - ) -} -``` - -**Performance Improvement**: 1000 rows now takes ~20-50 seconds instead of ~200 seconds! - -Let's zoom into processing our "Mona Lisa" row... - ---- - -### 🔍 Processing "Mona Lisa" Row - -#### Step 3.3a: Extract Row Data - -```typescript -const term = 'Mona Lisa' -const definition = 'Famous painting created by Leonardo da Vinci' -const type = 'Product' -const relatedTerms = 'Leonardo, Louvre' -``` - -#### Step 3.3b: Parallel Neural Extraction - -Here's where it gets **really cool**. Two expensive operations run **simultaneously**: - -```typescript -const [relatedEntities, concepts] = await Promise.all([ - // 1. Neural Entity Extraction (finds entities in the definition) - this.extractor.extract(definition, { - confidence: 0.48, - neuralMatching: true, - cache: { enabled: true } - }), - - // 2. Concept Extraction (extracts key concepts/tags) - this.brain.extractConcepts(definition, { limit: 10 }) -]) -``` - -##### 🧠 Neural Entity Extraction Deep Dive - -**Input**: `"Famous painting created by Leonardo da Vinci"` -**System**: `SmartExtractor` (entity type classifier) - -The SmartExtractor runs **4 signals in parallel**: - -``` -┌─────────────────────────────────────────────────┐ -│ SmartExtractor Ensemble │ -├─────────────────────────────────────────────────┤ -│ │ -│ 1. ExactMatchSignal (40%) │ -│ → Searches 334 noun keywords │ -│ → Finds "painting" → Product │ -│ → Confidence: 0.90 │ -│ │ -│ 2. EmbeddingSignal (35%) │ -│ → Embeds: "Leonardo da Vinci" │ -│ → Compares to 31 type embeddings │ -│ → Closest: Person (similarity: 0.92) │ -│ → Confidence: 0.92 │ -│ │ -│ 3. PatternSignal (20%) │ -│ → Tests regex patterns │ -│ → Matches: /^[A-Z][a-z]+ [A-Z][a-z]+$/ │ -│ → Suggests: Person │ -│ → Confidence: 0.85 │ -│ │ -│ 4. ContextSignal (5%) │ -│ → Checks format hints │ -│ → No prior context yet │ -│ → Confidence: 0.00 │ -│ │ -│ Ensemble Vote: │ -│ → Person: 0.92×0.35 + 0.85×0.20 = 0.49 │ -│ → Product: 0.90×0.40 = 0.36 │ -│ → Agreement boost: +0.05 (2 signals agree) │ -│ │ -│ Winner: Person (0.54 confidence) │ -└─────────────────────────────────────────────────┘ -``` - -**Output**: -```typescript -relatedEntities = [ - { - text: 'Leonardo da Vinci', - type: NounType.Person, - confidence: 0.92, - position: { start: 31, end: 48 } - }, - { - text: 'painting', - type: NounType.Product, - confidence: 0.85, - position: { start: 7, end: 15 } - } -] - -concepts = ['art', 'renaissance', 'painting', 'leonardo', 'italian', 'masterpiece'] -``` - -**Cache Hit Rate**: ~60% on subsequent rows with similar definitions! - -#### Step 3.3c: Determine Main Entity Type - -We have two sources of type information: -1. **Explicit type column**: `"Product"` -2. **Inferred from extraction**: `NounType.Person` - -**Priority**: Explicit type column wins! - -```typescript -const mainEntityType = type - ? this.mapTypeString('Product') // NounType.Product - : (relatedEntities[0].type) // Fallback to first extracted entity - -// Result: NounType.Product -``` - -**Type Mapping**: -```typescript -const mapping = { - 'product': NounType.Product, - 'person': NounType.Person, - 'place': NounType.Location, - 'organization': NounType.Organization, - 'concept': NounType.Concept, - 'event': NounType.Event, - // ... 31 total types -} -``` - -#### Step 3.3d: Generate Entity ID - -```typescript -const entityId = this.generateEntityId('Mona Lisa') - -// Algorithm: -// 1. Normalize: 'Mona Lisa' → 'mona_lisa' -// 2. Add prefix: 'ent_' -// 3. Add timestamp: Date.now() -// Result: 'ent_mona_lisa_1730000000000' -``` - -**Why timestamps?** Ensures globally unique IDs even with identical names. - -#### Step 3.3e: Create Main Entity Object - -```typescript -const mainEntity = { - id: 'ent_mona_lisa_1730000000000', - name: 'Mona Lisa', - type: NounType.Product, - description: 'Famous painting created by Leonardo da Vinci', - confidence: 0.95, // High confidence from explicit type - metadata: { - source: 'excel', - row: 3, - originalData: { - Term: 'Mona Lisa', - Definition: 'Famous painting created by Leonardo da Vinci', - Type: 'Product', - 'Related Terms': 'Leonardo, Louvre' - }, - concepts: ['art', 'renaissance', 'painting', 'leonardo', 'italian', 'masterpiece'], - extractedAt: 1730000000000 - } -} -``` - -#### Step 3.3f: Smart Relationship Inference ✨ - -**The Old Way** (before SmartRelationshipExtractor): -```typescript -// 😢 Everything was just "RelatedTo" -relationships.push({ - from: 'Mona Lisa', - to: 'Leonardo da Vinci', - type: VerbType.RelatedTo, // Generic! - confidence: 0.8 -}) -``` - -**The New Way** (with SmartRelationshipExtractor): - -For each entity found in the definition: - -```typescript -const verbType = await this.inferRelationship( - 'Mona Lisa', // subject - 'Leonardo da Vinci', // object - definition, // full context - NounType.Product, // subject type hint - NounType.Person // object type hint -) -``` - -##### 🎯 SmartRelationshipExtractor in Action - -**Location**: `src/neural/SmartRelationshipExtractor.ts:100` - -The SmartRelationshipExtractor runs **3 signals in parallel**: - -``` -┌──────────────────────────────────────────────────────────┐ -│ SmartRelationshipExtractor Ensemble │ -├──────────────────────────────────────────────────────────┤ -│ │ -│ Input Context: │ -│ "Famous painting created by Leonardo da Vinci" │ -│ │ -│ 1. VerbEmbeddingSignal (55%) │ -│ → Embeds context: [0.23, -0.45, 0.78, ...] │ -│ → Compares to 40 verb embeddings │ -│ → Closest match: CreatedBy (similarity: 0.89) │ -│ → Confidence: 0.89 │ -│ │ -│ 2. VerbPatternSignal (30%) │ -│ → Tests 48+ regex patterns │ -│ → Matches: /\bcreated?\s+by\b/i │ -│ → Maps to: VerbType.CreatedBy │ -│ → Confidence: 0.90 │ -│ │ -│ 3. VerbContextSignal (15%) │ -│ → Type pair: (Product, Person) │ -│ → Hint suggests: CreatedBy │ -│ → Confidence: 0.80 │ -│ │ -│ Ensemble Vote: │ -│ CreatedBy: 0.89×0.55 + 0.90×0.30 + 0.80×0.15 │ -│ = 0.49 + 0.27 + 0.12 │ -│ = 0.88 │ -│ │ -│ Agreement Boost: │ -│ → 3 signals agree on CreatedBy! │ -│ → Boost: +0.05 × (3-1) = +0.10 │ -│ → Final: 0.88 + 0.10 = 0.98 │ -│ │ -│ Winner: CreatedBy (0.98 confidence) 🎯 │ -└──────────────────────────────────────────────────────────┘ -``` - -**Result**: -```typescript -relationships.push({ - from: 'ent_mona_lisa_1730000000000', - to: 'Leonardo da Vinci', // Will be resolved to entity ID later - type: VerbType.CreatedBy, // 🎉 Intelligent classification! - confidence: 0.92, - evidence: 'Extracted from: "Famous painting created by Leonardo da Vinci..."' -}) -``` - -**Also processes "Related Terms" column**: -```typescript -const terms = 'Leonardo, Louvre'.split(',') -for (const relTerm of terms.map(t => t.trim())) { - relationships.push({ - from: 'ent_mona_lisa_1730000000000', - to: relTerm, - type: VerbType.RelatedTo, // Explicit relationships from column - confidence: 0.9, - evidence: 'Explicitly listed in "Related Terms" column' - }) -} -``` - -#### Step 3.3g: Progress Tracking - -Every chunk completion triggers progress: - -```typescript -opts.onProgress({ - processed: 3, - total: 10, - entities: 6, // 3 main + 3 related - relationships: 5, - throughput: 15.2, // rows per second - eta: 458, // milliseconds remaining - phase: 'extracting' -}) -``` - -**Progress Bar Example**: -``` -Extracting entities from excel (15.2 rows/sec, ETA: 0s)... [████████░░] 30% -``` - ---- - -### Step 3.4: Final Extraction Result - -After processing all rows, SmartExcelImporter returns: - -```typescript -{ - rowsProcessed: 3, - entitiesExtracted: 9, // 3 main + 6 related - relationshipsInferred: 8, - rows: [ - { - entity: { - id: 'ent_neural_net_1730000000001', - name: 'Neural Net', - type: NounType.Concept, - description: 'Machine learning model inspired by the brain', - confidence: 0.95, - metadata: { ... } - }, - relatedEntities: [ - { name: 'machine learning', type: NounType.Concept, confidence: 0.88 }, - { name: 'brain', type: NounType.Thing, confidence: 0.82 } - ], - relationships: [ - { from: 'ent_neural_net_...', to: 'AI', type: VerbType.RelatedTo, confidence: 0.9 }, - { from: 'ent_neural_net_...', to: 'Deep Learning', type: VerbType.RelatedTo, confidence: 0.9 } - ], - concepts: ['ml', 'ai', 'neural', 'learning', 'computation'] - }, - { - entity: { - id: 'ent_leonardo_1730000000002', - name: 'Leonardo', - type: NounType.Person, - description: 'Renaissance artist who painted Mona Lisa', - confidence: 0.95, - metadata: { ... } - }, - relatedEntities: [ - { name: 'Mona Lisa', type: NounType.Product, confidence: 0.90 }, - { name: 'Renaissance', type: NounType.Event, confidence: 0.85 } - ], - relationships: [ - { from: 'ent_leonardo_...', to: 'Mona Lisa', type: VerbType.Creates, confidence: 0.91 }, - { from: 'ent_leonardo_...', to: 'Art', type: VerbType.RelatedTo, confidence: 0.9 } - ], - concepts: ['art', 'renaissance', 'painter', 'artist', 'italian'] - }, - { - entity: { - id: 'ent_mona_lisa_1730000000000', - name: 'Mona Lisa', - type: NounType.Product, - description: 'Famous painting created by Leonardo da Vinci', - confidence: 0.95, - metadata: { ... } - }, - relatedEntities: [ - { name: 'Leonardo da Vinci', type: NounType.Person, confidence: 0.92 }, - { name: 'painting', type: NounType.Product, confidence: 0.85 } - ], - relationships: [ - { from: 'ent_mona_lisa_...', to: 'Leonardo da Vinci', type: VerbType.CreatedBy, confidence: 0.92 }, - { from: 'ent_mona_lisa_...', to: 'Leonardo', type: VerbType.RelatedTo, confidence: 0.9 }, - { from: 'ent_mona_lisa_...', to: 'Louvre', type: VerbType.RelatedTo, confidence: 0.9 } - ], - concepts: ['art', 'renaissance', 'painting', 'leonardo', 'italian', 'masterpiece'] - } - ], - entityMap: Map { - 'neural net' => 'ent_neural_net_1730000000001', - 'leonardo' => 'ent_leonardo_1730000000002', - 'mona lisa' => 'ent_mona_lisa_1730000000000' - }, - processingTime: 1243, - stats: { - byType: { - 'Concept': 1, - 'Person': 1, - 'Product': 1 - }, - byConfidence: { - high: 3, // > 0.8 - medium: 0, // 0.6-0.8 - low: 0 // < 0.6 - } - } -} -``` - -**Performance**: 3 rows processed in **1.2 seconds** (with neural extraction + relationship inference) - ---- - -## Phase 4: VFS Structure Creation - -**Location**: `src/importers/VFSStructureGenerator.ts:93` - -**Progress Callback**: -```typescript -{ stage: 'storing-vfs', message: 'Creating VFS structure...' } -``` - -The VFS (Virtual File System) provides a **human-readable, organized view** of imported data. - -### Step 4.1: Normalize Result - -```typescript -const normalizedResult = this.normalizeExtractionResult(extractionResult, 'excel') -``` - -This converts format-specific results into a common structure that VFSStructureGenerator can process. - -### Step 4.2: Generate VFS Hierarchy - -```typescript -await this.vfsGenerator.generate(normalizedResult, { - rootPath: '/imports/glossary', - groupBy: 'type', // Group by NounType - preserveSource: true, // Keep original Excel file - createRelationshipFile: true, // Create _relationships.json - createMetadataFile: true // Create _metadata.json -}) -``` - -**Grouping Strategies**: -- `'type'` → Group by NounType (Person/, Product/, Concept/) -- `'sheet'` → Group by Excel sheet name -- `'flat'` → All entities in root directory -- `'custom'` → Provide custom grouping function - -**VFS Structure Created**: - -``` -/imports/glossary/ -├── source.xlsx # ← Original file preserved -├── _metadata.json # ← Import metadata -├── _relationships.json # ← All relationships (human-readable) -├── Concept/ # ← NounType.Concept entities -│ └── neural_net.json -├── Person/ # ← NounType.Person entities -│ └── leonardo.json -└── Product/ # ← NounType.Product entities - └── mona_lisa.json -``` - -### Step 4.3: File Contents - -**`/imports/glossary/Product/mona_lisa.json`**: -```json -{ - "id": "ent_mona_lisa_1730000000000", - "name": "Mona Lisa", - "type": "Product", - "description": "Famous painting created by Leonardo da Vinci", - "confidence": 0.95, - "metadata": { - "source": "excel", - "row": 3, - "originalData": { - "Term": "Mona Lisa", - "Definition": "Famous painting created by Leonardo da Vinci", - "Type": "Product", - "Related Terms": "Leonardo, Louvre" - }, - "concepts": ["art", "renaissance", "painting", "leonardo", "italian", "masterpiece"], - "extractedAt": 1730000000000, - "vfsPath": "/imports/glossary/Product/mona_lisa.json" - } -} -``` - -**`/imports/glossary/_relationships.json`**: -```json -{ - "importId": "import_xyz789", - "createdAt": 1730000000000, - "totalRelationships": 8, - "relationships": [ - { - "from": "ent_mona_lisa_1730000000000", - "fromName": "Mona Lisa", - "to": "ent_leonardo_1730000000002", - "toName": "Leonardo da Vinci", - "type": "CreatedBy", - "confidence": 0.92, - "evidence": "Extracted from: \"Famous painting created by Leonardo da Vinci...\"" - }, - { - "from": "ent_mona_lisa_1730000000000", - "fromName": "Mona Lisa", - "to": "ent_leonardo_1730000000002", - "toName": "Leonardo", - "type": "RelatedTo", - "confidence": 0.9, - "evidence": "Explicitly listed in \"Related Terms\" column" - } - // ... more relationships - ] -} -``` - -**`/imports/glossary/_metadata.json`**: -```json -{ - "importId": "import_xyz789", - "format": "excel", - "formatConfidence": 1.0, - "sourceFilename": "glossary.xlsx", - "importedAt": 1730000000000, - "options": { - "enableNeuralExtraction": true, - "enableRelationshipInference": true, - "enableConceptExtraction": true, - "confidenceThreshold": 0.6 - }, - "stats": { - "rowsProcessed": 3, - "entitiesExtracted": 9, - "relationshipsInferred": 8, - "processingTime": 1243 - } -} -``` - -### Step 4.4: VFS Benefits - -**Why VFS?** -1. ✅ **Human-readable** - Browse imported data like files -2. ✅ **Organized** - Automatic grouping by type/sheet/custom -3. ✅ **Traceable** - Preserves original source and metadata -4. ✅ **Exportable** - Easy to extract data back out -5. ✅ **Debuggable** - Inspect exactly what was imported - -**VFS Operations**: -```typescript -// Read entity file -const entity = await brain.vfs().readJSON('/imports/glossary/Product/mona_lisa.json') - -// List all products -const products = await brain.vfs().readdir('/imports/glossary/Product') - -// Search VFS -const matches = await brain.vfs().find('/imports/**/*.json', { - type: 'Product' -}) -``` - ---- - -## Phase 5: Knowledge Graph Creation - -**Location**: `src/import/ImportCoordinator.ts:676` - -**Progress Callback**: -```typescript -{ stage: 'storing-graph', message: 'Creating knowledge graph...' } -``` - -This is where your data becomes **queryable knowledge** with vector embeddings and graph relationships. - -### Step 5.1: Smart Deduplication - -Before creating entities, check for duplicates: - -```typescript -const DEDUPLICATION_AUTO_DISABLE_THRESHOLD = 100 - -if (enableDeduplication && rows.length <= 100) { - const mergeResult = await this.deduplicator.createOrMerge(entity, '/imports/glossary', { - threshold: 0.85 // Cosine similarity threshold - }) -} -``` - -**How Deduplication Works**: - -1. **Embed entity name**: `"Mona Lisa"` → `[0.12, -0.45, 0.78, ...]` -2. **Search similar entities**: `brain.similar(embedding, { limit: 10 })` -3. **Check similarity threshold**: If any result > 0.85, it's a match -4. **Merge or create**: - - **Match found**: Merge metadata, update VFS path, return existing ID - - **No match**: Create new entity - -**Why Auto-Disable?** -- Deduplication requires O(n²) vector searches -- For 1000 entities: 1000 searches × ~10ms = **10 seconds** of overhead -- Auto-disabled for imports > 100 entities - -**Override**: -```typescript -await brain.import(buffer, { - enableDeduplication: true, // Force enable even for large imports - deduplicationThreshold: 0.9 // Higher threshold = stricter matching -}) -``` - -### Step 5.2: Create Entity in Knowledge Graph - -For each entity (e.g., "Mona Lisa"): - -```typescript -const entityId = await this.brain.add({ - id: 'ent_mona_lisa_1730000000000', - data: { - name: 'Mona Lisa', - type: NounType.Product, - description: 'Famous painting created by Leonardo da Vinci', - vfsPath: '/imports/glossary/Product/mona_lisa.json' - }, - type: NounType.Product, - metadata: { - source: 'excel', - row: 3, - concepts: ['art', 'renaissance', 'painting', 'leonardo'], - importedFrom: '/imports/glossary', - extractedAt: 1730000000000 - } -}) -``` - -**What Happens Inside `brain.add()`**: - -**Location**: `src/brainy.ts:342` - -#### 5.2a: Generate Embedding - -```typescript -const vector = await this.embed('Mona Lisa') -``` - -**Embedding Service**: -- Uses Candle WASM (local, no API calls, no downloads!) -- Model: `all-MiniLM-L6-v2` embedded in WASM (384 dimensions) -- Performance: ~5-15ms per embedding - -**Output**: -```typescript -vector = [ - 0.123456, -0.456789, 0.789012, -0.234567, 0.567890, ... - // ... 384 total dimensions -] -``` - -**Why Embeddings?** -- Enables **semantic search**: Find similar concepts, not just exact matches -- Powers **neural queries**: "Find paintings like the Mona Lisa" -- Supports **relationship inference**: Similar entities often share relationships - -#### 5.2b: Add to HNSW Index - -```typescript -await this.index.addItem( - { id: 'ent_mona_lisa_...', vector }, - NounType.Product // Type-aware indexing -) -``` - -**HNSW (Hierarchical Navigable Small World) Index**: - -``` - Layer 2 (entry point) - [Neural Net] - | - Layer 1 | - [Leonardo]---[Mona Lisa] - / | | - Layer 0 | | - [AI]--[DL]--+--[Art]--[Louvre] -``` - -**Benefits**: -- **Fast search**: O(log n) instead of O(n) -- **Approximate nearest neighbors**: 95%+ recall at 10x speed -- **Type-aware**: Can search within a specific NounType - -**Structure**: -```typescript -{ - items: Map { - 'ent_mona_lisa_...' => { - vector: [0.123, -0.456, ...], - connections: Map { - 0 => Set(['ent_leonardo_...', 'ent_louvre_...']), // Layer 0 neighbors - 1 => Set(['ent_leonardo_...']) // Layer 1 neighbors - }, - level: 1 // Max layer this node appears in - } - }, - entryPoint: 'ent_neural_net_...', // Top layer entry point - typeMap: Map { - NounType.Product => Set(['ent_mona_lisa_...']), - NounType.Person => Set(['ent_leonardo_...']), - NounType.Concept => Set(['ent_neural_net_...']) - } -} -``` - -#### 5.2c: Save to Storage (Dual Write) - -**Vector Storage** (optimized for retrieval): -```typescript -await this.storage.saveNoun({ - id: 'ent_mona_lisa_...', - vector: [0.123, -0.456, ...], - connections: Map { /* HNSW connections */ }, - level: 1 -}) -``` - -**Metadata Storage** (optimized for filtering): -```typescript -await this.storage.saveNounMetadata('ent_mona_lisa_...', { - name: 'Mona Lisa', - type: NounType.Product, - description: 'Famous painting created by Leonardo da Vinci', - _data: { name: 'Mona Lisa', type: NounType.Product, ... }, - noun: NounType.Product, - service: undefined, - createdAt: 1730000000000, - vfsPath: '/imports/glossary/Product/mona_lisa.json', - source: 'excel', - row: 3, - concepts: ['art', 'renaissance', 'painting', 'leonardo'], - importedFrom: '/imports/glossary' -}) -``` - -**Why Separate Storage?** -- Vectors are large (384 × 4 bytes = 1.5KB each) -- Metadata queries don't need vectors -- Faster metadata filtering without loading vectors -- Better compression (metadata is JSON, vectors are binary) - -#### 5.2d: Update Metadata Index - -```typescript -await this.metadataIndex.addDocument('ent_mona_lisa_...', { - name: 'Mona Lisa', - type: 'Product', - source: 'excel', - vfsPath: '/imports/glossary/Product/mona_lisa.json' -}) -``` - -**Inverted Index Structure**: -```typescript -{ - documents: Map { - 'ent_mona_lisa_...' => { name: 'Mona Lisa', type: 'Product', source: 'excel', ... } - }, - invertedIndex: Map { - 'type:Product' => Set(['ent_mona_lisa_...']), - 'source:excel' => Set(['ent_neural_net_...', 'ent_leonardo_...', 'ent_mona_lisa_...']), - 'name:Mona Lisa' => Set(['ent_mona_lisa_...']) - }, - fieldStats: Map { - 'type' => { cardinality: 3, values: Map { 'Product' => 1, 'Person' => 1, 'Concept' => 1 } }, - 'source' => { cardinality: 1, values: Map { 'excel' => 3 } } - } -} -``` - -**Benefits**: -- **Fast filtering**: `brain.find({ type: 'Product' })` → O(1) lookup -- **Combined queries**: Filter + vector search in one query -- **Field discovery**: List all available fields for dynamic UIs - ---- - -### Step 5.3: Create Relationships in Graph - -For each relationship (e.g., "Mona Lisa" → "Leonardo da Vinci"): - -```typescript -await this.brain.relate({ - from: 'ent_mona_lisa_1730000000000', - to: 'ent_leonardo_1730000000002', - type: VerbType.CreatedBy, - weight: 1.0, - metadata: { - confidence: 0.92, - evidence: 'Extracted from: "Famous painting created by Leonardo da Vinci..."', - importedFrom: '/imports/glossary' - } -}) -``` - -**What Happens Inside `brain.relate()`**: - -**Location**: `src/brainy.ts:744` - -#### 5.3a: Verify Entities Exist - -```typescript -const fromEntity = await this.get('ent_mona_lisa_...') -const toEntity = await this.get('ent_leonardo_...') - -if (!fromEntity || !toEntity) { - throw new Error('Entity not found') -} -``` - -#### 5.3b: Check for Duplicates (Critical Fix) - -**The Bug**: Without duplicate checking, re-importing would create: -``` -Mona Lisa --CreatedBy--> Leonardo -Mona Lisa --CreatedBy--> Leonardo // Duplicate! -Mona Lisa --CreatedBy--> Leonardo // Another duplicate! -``` - -**The Fix**: -```typescript -const existingVerbs = await this.storage.getVerbsBySource('ent_mona_lisa_...') -const duplicate = existingVerbs.find(v => - v.targetId === 'ent_leonardo_...' && - v.verb === VerbType.CreatedBy -) - -if (duplicate) { - console.log('[DEBUG] Skipping duplicate relationship') - return duplicate.id // Return existing relationship ID -} -``` - -#### 5.3c: Compute Relationship Vector - -```typescript -const relationVector = fromEntity.vector.map((v, i) => - (v + toEntity.vector[i]) / 2 -) -``` - -**Why?** The relationship embedding lives "between" the two entities in vector space. - -**Example**: -``` -Mona Lisa vector: [0.8, 0.2, 0.5, ...] -Leonardo vector: [0.6, 0.4, 0.3, ...] -Relation vector: [0.7, 0.3, 0.4, ...] ← Average -``` - -**Use Cases**: -- Find similar relationships -- Cluster relationship types -- Recommend new connections - -#### 5.3d: Save to Storage - -```typescript -const verb: GraphVerb = { - id: 'verb_abc123', - vector: [0.7, 0.3, 0.4, ...], - sourceId: 'ent_mona_lisa_...', - targetId: 'ent_leonardo_...', - source: NounType.Product, - target: NounType.Person, - verb: VerbType.CreatedBy, - type: VerbType.CreatedBy, - weight: 1.0, - metadata: { confidence: 0.92, ... } -} - -await this.storage.saveVerb(verb) -await this.storage.saveVerbMetadata('verb_abc123', { - verb: VerbType.CreatedBy, // ← Critical for count tracking - weight: 1.0, - confidence: 0.92, - evidence: '...', - createdAt: 1730000000000 -}) -``` - -#### 5.3e: Update Graph Adjacency Index - -```typescript -await this.graphIndex.addEdge( - 'ent_mona_lisa_...', - 'ent_leonardo_...', - VerbType.CreatedBy, - 1.0 // weight -) -``` - -**Graph Adjacency Index Structure**: - -```typescript -{ - // Forward edges (source → target) - forward: Map { - 'ent_mona_lisa_...' => Map { - 'CreatedBy' => Set(['verb_abc123']), - 'RelatedTo' => Set(['verb_def456', 'verb_ghi789']) - } - }, - - // Reverse edges (target → source) - reverse: Map { - 'ent_leonardo_...' => Map { - 'CreatedBy' => Set(['verb_abc123']), // Mona Lisa was CreatedBy Leonardo - 'RelatedTo' => Set(['verb_def456']) - } - }, - - // Global verb counts - verbCounts: Map { - 'CreatedBy' => 1, - 'RelatedTo' => 4 - } -} -``` - -**Benefits**: -- **O(1) relationship lookups**: `related(entityId)` is instant -- **Bidirectional traversal**: Find incoming and outgoing edges -- **Type filtering**: Get only `CreatedBy` relationships -- **Global statistics**: Count relationships by type - -**Query Examples**: -```typescript -// What did Mona Lisa create? (outgoing edges) -const outgoing = await brain.related({ from: 'ent_mona_lisa_...' }) - -// What created Mona Lisa? (incoming edges) -const incoming = await brain.related({ to: 'ent_mona_lisa_...' }) - -// Get only CreatedBy relationships -const createdBy = await brain.related({ - from: 'ent_mona_lisa_...', - type: VerbType.CreatedBy -}) -``` - ---- - -## Phase 6: Persistence and Finalization - -**Location**: `src/import/ImportCoordinator.ts:396` - -### Step 6.1: Flush Indexes to Disk - -**Always-On Streaming with Adaptive Flush Intervals:** - -Periodic flushes happen automatically during import: - -```typescript -// During entity loop (ImportCoordinator.ts:914-933): -entitiesSinceFlush++ - -if (entitiesSinceFlush >= flushInterval) { // Adaptive: 100, 1000, or 5000 - await this.brain.flush() - entitiesSinceFlush = 0 - - // Notify that data is queryable - await onProgress?.({ - queryable: true, // ← Indexes are up-to-date! - stage: 'storing-graph', - message: `Flushed indexes (${entities.length}/${rows.length} entities)`, - processed: entities.length, - total: rows.length, - entities: entities.length - }) -} -``` - -**Progress Callback**: -```typescript -{ - stage: 'storing-graph', - message: 'Flushed indexes (3000/10000 entities, 45ms)', - processed: 3000, - total: 10000, - queryable: true // ← Data is now queryable! -} -``` - -**What Gets Flushed**: - -1. **Metadata Index** → `metadata-index.json` - - Inverted index (field → entity mappings) - - Field statistics - - EntityIdMapper (UUID ↔ integer mappings) - -2. **Graph Adjacency Index** → `graph-adjacency.json` - - Forward edges (source → targets) - - Reverse edges (target → sources) - - Verb counts (relationship statistics) - -3. **Storage Counts** → Type statistics - - Noun counts by type - - Verb counts by type - -**What Doesn't Get Flushed** (Already Persisted): -- ✅ Entity vectors (written immediately on `brain.add()`) -- ✅ Entity metadata (written immediately) -- ✅ Relationship vectors (written immediately on `brain.relate()`) -- ✅ Relationship metadata (written immediately) - -**Key Insight**: Flush writes *indexes*, not entities! - -**Without Flushing**: -- ❌ Entities exist but queries are slow (full table scans) -- ❌ Index-accelerated queries won't work -- ❌ In-memory indexes lost on crash - -**With Periodic Flushing** (streaming mode): -- ✅ Queries are fast (index lookups) -- ✅ Data queryable during import -- ✅ Crash resilient (partial imports survive) - -### Step 6.2: Record in Import History - -```typescript -await this.history.recordImport( - 'import_xyz789', // Import ID - { - type: 'buffer', - filename: 'glossary.xlsx', - format: 'excel' - }, - result // Full import result -) -``` - -**History Storage**: `.brainy/import-history.json` - -```json -{ - "imports": [ - { - "id": "import_xyz789", - "timestamp": 1730000000000, - "source": { - "type": "buffer", - "filename": "glossary.xlsx", - "format": "excel" - }, - "stats": { - "entitiesExtracted": 9, - "relationshipsInferred": 8, - "processingTime": 1843 - }, - "vfsPath": "/imports/glossary" - } - ] -} -``` - -**Use Cases**: -- List all imports: `coordinator.getHistory().getHistory()` (the `ImportHistory` entries shown above) -- Reimport with same settings -- Audit trail for compliance -- Rollback imports - -### Step 6.3: Return Complete Result - -```typescript -return { - importId: 'import_xyz789', - format: 'excel', - formatConfidence: 1.0, - - vfs: { - rootPath: '/imports/glossary', - directories: [ - '/imports/glossary/Concept', - '/imports/glossary/Person', - '/imports/glossary/Product' - ], - files: [ - { path: '/imports/glossary/source.xlsx', type: 'source' }, - { path: '/imports/glossary/_metadata.json', type: 'metadata' }, - { path: '/imports/glossary/_relationships.json', type: 'relationships' }, - { path: '/imports/glossary/Concept/neural_net.json', entityId: 'ent_...', type: 'entity' }, - { path: '/imports/glossary/Person/leonardo.json', entityId: 'ent_...', type: 'entity' }, - { path: '/imports/glossary/Product/mona_lisa.json', entityId: 'ent_...', type: 'entity' } - ] - }, - - entities: [ - { id: 'ent_neural_net_...', name: 'Neural Net', type: NounType.Concept, vfsPath: '...' }, - { id: 'ent_leonardo_...', name: 'Leonardo', type: NounType.Person, vfsPath: '...' }, - { id: 'ent_mona_lisa_...', name: 'Mona Lisa', type: NounType.Product, vfsPath: '...' } - ], - - relationships: [ - { id: 'verb_1', from: 'ent_mona_lisa_...', to: 'ent_leonardo_...', type: VerbType.CreatedBy }, - { id: 'verb_2', from: 'ent_mona_lisa_...', to: 'ent_leonardo_...', type: VerbType.RelatedTo }, - { id: 'verb_3', from: 'ent_mona_lisa_...', to: 'ent_louvre_...', type: VerbType.RelatedTo }, - // ... more relationships - ], - - stats: { - entitiesExtracted: 9, - relationshipsInferred: 8, - vfsFilesCreated: 6, - graphNodesCreated: 3, - graphEdgesCreated: 8, - entitiesMerged: 0, // Deduplication found 0 duplicates - entitiesNew: 3, // Created 3 new entities - processingTime: 1843 // Total time: 1.8 seconds - } -} -``` - -**Progress Callback** (final): -```typescript -{ - stage: 'complete', - message: 'Import complete', - entities: 3, - relationships: 8 -} -``` - ---- - -## What Gets Created in Brainy - -After importing `glossary.xlsx`, here's **everything** that gets created: - -### 1. VFS (Virtual File System) - -**Location**: In-memory + flushed to `.brainy/.vfs/` - -``` -/imports/glossary/ -├── source.xlsx # Original Excel file (preserved) -├── _metadata.json # Import metadata -├── _relationships.json # All relationships (human-readable) -├── Concept/ -│ └── neural_net.json # Entity: Neural Net -├── Person/ -│ └── leonardo.json # Entity: Leonardo -└── Product/ - └── mona_lisa.json # Entity: Mona Lisa -``` - -**Access**: -```typescript -// Read entity -const entity = await brain.vfs().readJSON('/imports/glossary/Product/mona_lisa.json') - -// List directory -const files = await brain.vfs().readdir('/imports/glossary/Product') - -// Search -const results = await brain.vfs().find('/imports/**/*.json', { type: 'Product' }) -``` - ---- - -### 2. Storage Layer (File System Adapter) - -**Location**: `.brainy/` directory - -``` -.brainy/ -├── nouns/ # Entity vectors -│ ├── ent_neural_net_1730000000001.json -│ ├── ent_leonardo_1730000000002.json -│ └── ent_mona_lisa_1730000000000.json -│ -├── nouns-metadata/ # Entity metadata -│ ├── ent_neural_net_1730000000001.json -│ ├── ent_leonardo_1730000000002.json -│ └── ent_mona_lisa_1730000000000.json -│ -├── verbs/ # Relationship vectors -│ ├── verb_abc123.json # Mona Lisa --CreatedBy--> Leonardo -│ ├── verb_def456.json # Mona Lisa --RelatedTo--> Leonardo -│ ├── verb_ghi789.json # Mona Lisa --RelatedTo--> Louvre -│ └── ... -│ -├── verbs-metadata/ # Relationship metadata -│ ├── verb_abc123.json -│ ├── verb_def456.json -│ └── ... -│ -├── index.json # HNSW index structure -├── metadata-index.json # Inverted index for filtering -├── graph-adjacency.json # Graph structure for fast traversal -└── import-history.json # Import audit trail -``` - ---- - -### 3. Entity Storage Detail - -**`nouns/ent_mona_lisa_1730000000000.json`**: -```json -{ - "id": "ent_mona_lisa_1730000000000", - "vector": [ - 0.123456, -0.456789, 0.789012, -0.234567, 0.567890, - // ... 384 dimensions total - ], - "connections": { - "0": ["ent_leonardo_1730000000002", "ent_louvre_..."], - "1": ["ent_leonardo_1730000000002"] - }, - "level": 1 -} -``` - -**`nouns-metadata/ent_mona_lisa_1730000000000.json`**: -```json -{ - "name": "Mona Lisa", - "type": "Product", - "description": "Famous painting created by Leonardo da Vinci", - "_data": { - "name": "Mona Lisa", - "type": "Product", - "description": "Famous painting created by Leonardo da Vinci", - "vfsPath": "/imports/glossary/Product/mona_lisa.json" - }, - "noun": "Product", - "service": null, - "createdAt": 1730000000000, - "vfsPath": "/imports/glossary/Product/mona_lisa.json", - "source": "excel", - "row": 3, - "concepts": ["art", "renaissance", "painting", "leonardo", "italian", "masterpiece"], - "importedFrom": "/imports/glossary", - "extractedAt": 1730000000000 -} -``` - ---- - -### 4. Relationship Storage Detail - -**`verbs/verb_abc123.json`**: -```json -{ - "id": "verb_abc123", - "vector": [ - 0.723456, -0.256789, 0.489012, - // ... 384 dimensions (average of source + target vectors) - ], - "sourceId": "ent_mona_lisa_1730000000000", - "targetId": "ent_leonardo_1730000000002", - "source": "Product", - "target": "Person", - "verb": "CreatedBy", - "type": "CreatedBy", - "weight": 1.0 -} -``` - -**`verbs-metadata/verb_abc123.json`**: -```json -{ - "verb": "CreatedBy", - "weight": 1.0, - "confidence": 0.92, - "evidence": "Extracted from: \"Famous painting created by Leonardo da Vinci...\"", - "importedFrom": "/imports/glossary", - "createdAt": 1730000000000 -} -``` - ---- - -### 5. HNSW Index Structure - -**`index.json`**: -```json -{ - "dimensions": 384, - "M": 16, - "efConstruction": 200, - "entryPoint": "ent_neural_net_1730000000001", - "items": [ - { - "id": "ent_mona_lisa_1730000000000", - "level": 1, - "connections": { - "0": ["ent_leonardo_1730000000002", "ent_louvre_..."], - "1": ["ent_leonardo_1730000000002"] - } - }, - { - "id": "ent_leonardo_1730000000002", - "level": 1, - "connections": { - "0": ["ent_mona_lisa_1730000000000", "ent_neural_net_..."], - "1": ["ent_neural_net_1730000000001"] - } - }, - { - "id": "ent_neural_net_1730000000001", - "level": 2, - "connections": { - "0": ["ent_leonardo_1730000000002"], - "1": ["ent_leonardo_1730000000002"], - "2": [] - } - } - ], - "typeMap": { - "Product": ["ent_mona_lisa_1730000000000"], - "Person": ["ent_leonardo_1730000000002"], - "Concept": ["ent_neural_net_1730000000001"] - } -} -``` - -**Visual Representation**: -``` -Layer 2: [Neural Net] ← Entry point - | -Layer 1: [Leonardo]---[Mona Lisa] - | | | -Layer 0: [AI]-+-[DL] [Louvre] -``` - ---- - -### 6. Metadata Index Structure - -**`metadata-index.json`**: -```json -{ - "documents": { - "ent_mona_lisa_1730000000000": { - "name": "Mona Lisa", - "type": "Product", - "source": "excel", - "vfsPath": "/imports/glossary/Product/mona_lisa.json" - }, - "ent_leonardo_1730000000002": { - "name": "Leonardo", - "type": "Person", - "source": "excel", - "vfsPath": "/imports/glossary/Person/leonardo.json" - }, - "ent_neural_net_1730000000001": { - "name": "Neural Net", - "type": "Concept", - "source": "excel", - "vfsPath": "/imports/glossary/Concept/neural_net.json" - } - }, - "invertedIndex": { - "type:Product": ["ent_mona_lisa_1730000000000"], - "type:Person": ["ent_leonardo_1730000000002"], - "type:Concept": ["ent_neural_net_1730000000001"], - "source:excel": [ - "ent_neural_net_1730000000001", - "ent_leonardo_1730000000002", - "ent_mona_lisa_1730000000000" - ], - "name:Mona Lisa": ["ent_mona_lisa_1730000000000"], - "name:Leonardo": ["ent_leonardo_1730000000002"], - "name:Neural Net": ["ent_neural_net_1730000000001"] - }, - "fieldStats": { - "type": { - "cardinality": 3, - "values": { - "Product": 1, - "Person": 1, - "Concept": 1 - } - }, - "source": { - "cardinality": 1, - "values": { - "excel": 3 - } - } - } -} -``` - ---- - -### 7. Graph Adjacency Index Structure - -**`graph-adjacency.json`**: -```json -{ - "forward": { - "ent_mona_lisa_1730000000000": { - "CreatedBy": ["verb_abc123"], - "RelatedTo": ["verb_def456", "verb_ghi789"] - }, - "ent_leonardo_1730000000002": { - "Creates": ["verb_jkl012"], - "RelatedTo": ["verb_mno345"] - } - }, - "reverse": { - "ent_leonardo_1730000000002": { - "CreatedBy": ["verb_abc123"], - "RelatedTo": ["verb_def456"] - }, - "ent_louvre_...": { - "RelatedTo": ["verb_ghi789"] - } - }, - "verbCounts": { - "CreatedBy": 1, - "RelatedTo": 4, - "Creates": 1 - } -} -``` - -**Query Examples**: -```typescript -// What relationships does Mona Lisa have? -forward['ent_mona_lisa_...'] -// → { CreatedBy: [...], RelatedTo: [...] } - -// What created Mona Lisa? -reverse['ent_mona_lisa_...']['CreatedBy'] -// → ['verb_abc123'] → Leonardo da Vinci - -// How many CreatedBy relationships exist? -verbCounts['CreatedBy'] -// → 1 -``` - ---- - -### 8. Storage Layout - -Brainy 8.0 ships two adapters: filesystem and memory. - -#### Filesystem (Default) -``` -.brainy/ -├── nouns/ -├── nouns-metadata/ -├── verbs/ -├── verbs-metadata/ -└── index.json -``` - -**Configuration**: -```typescript -const brain = await Brainy.create({ - storage: { - type: 'filesystem', - path: './.brainy' - } -}) -``` - -For off-site backup, snapshot `path` from your scheduler with `gsutil rsync`, `aws s3 sync`, `rclone`, or `tar`. Brainy itself doesn't reach out to object storage. - ---- - -## Performance & Scale - -### Benchmarks - -**Small Import** (10 entities): -- Extraction: ~400ms -- VFS creation: ~50ms -- Graph creation: ~150ms -- **Total**: ~600ms - -**Medium Import** (100 entities): -- Extraction: ~1200ms (batched parallel) -- VFS creation: ~200ms -- Graph creation: ~400ms -- **Total**: ~1800ms - -**Large Import** (1000 entities): -- Extraction: ~12000ms (batched parallel) -- VFS creation: ~800ms -- Graph creation: ~2000ms -- Deduplication: Auto-disabled (too slow) -- **Total**: ~15 seconds - -**Billion-Scale Performance**: -- HNSW Index: O(log n) search (1B entities = ~30 hops) -- Metadata Index: O(1) filtering -- Graph Adjacency: O(1) relationship lookups -- Storage: Bounded by the filesystem volume backing `path` - -### Optimization Tips - -#### 1. Disable Features for Large Imports - -```typescript -await brain.import(buffer, { - enableNeuralExtraction: false, // Skip entity extraction (10x faster) - enableRelationshipInference: false, // Skip relationship inference (5x faster) - enableConceptExtraction: false, // Skip concept extraction (2x faster) - enableDeduplication: false // Skip deduplication (prevents O(n²)) -}) -``` - -**Speedup**: 1000 entities in ~2 seconds instead of ~15 seconds! - -#### 2. Use Explicit Type Column - -```typescript -// ✅ Fast: Uses explicit type, skips neural classification -{ Term: 'Mona Lisa', Type: 'Product', ... } - -// ❌ Slow: Runs 4 neural signals to infer type -{ Term: 'Mona Lisa', ... } -``` - -#### 3. Batch Multiple Imports - -```typescript -// ❌ Slow: 10 separate imports -for (const file of files) { - await brain.import(file) // Flushes after each import -} - -// ✅ Fast: Combine into one import, flush once -const combined = mergeFiles(files) -await brain.import(combined) -``` - -#### 4. Use Streaming for Huge Files - -```typescript -const { createPipeline } = await brain.streaming() - -await createPipeline() - .source(hugeExcelFile) - .transform(extractEntities) - .transform(createRelationships) - .sink(brain.add.bind(brain)) - .run({ chunkSize: 100 }) -``` - -#### 5. Choose Right Grouping Strategy - -```typescript -// ✅ Fast: Flat structure (no nested directories) -groupBy: 'flat' - -// ❌ Slow: Type-based grouping (creates many directories) -groupBy: 'type' -``` - ---- - -## Summary: The Complete Picture - -``` -┌──────────────────────────────────────────────────────────────┐ -│ brain.import() │ -└──────────────────────────────────────────────────────────────┘ - ↓ - ┌───────────────────────────────────────────────┐ - │ Phase 1: Entry Point (brainy.ts:1952) │ - │ - Lazy load ImportCoordinator │ - │ - Initialize 7 Smart importers │ - └───────────────────────────────────────────────┘ - ↓ - ┌───────────────────────────────────────────────┐ - │ Phase 2: Orchestration (ImportCoordinator) │ - │ - Normalize source (Buffer/URL/path) │ - │ - Detect format (excel/pdf/csv/json/...) │ - │ - Route to SmartExcelImporter │ - └───────────────────────────────────────────────┘ - ↓ - ┌───────────────────────────────────────────────┐ - │ Phase 3: Neural Extraction 🧠 │ - │ │ - │ SmartExtractor (Entity Types): │ - │ ├─ ExactMatchSignal (40%) │ - │ ├─ EmbeddingSignal (35%) │ - │ ├─ PatternSignal (20%) │ - │ └─ ContextSignal (5%) │ - │ │ - │ SmartRelationshipExtractor (Verb Types): │ - │ ├─ VerbEmbeddingSignal (55%) │ - │ ├─ VerbPatternSignal (30%) │ - │ └─ VerbContextSignal (15%) │ - │ │ - │ Result: Intelligent entities + relationships │ - └───────────────────────────────────────────────┘ - ↓ - ┌───────────────────────────────────────────────┐ - │ Phase 4: VFS Structure │ - │ - Group by type/sheet/flat │ - │ - Create directory hierarchy │ - │ - Write entity JSON files │ - │ - Preserve source file │ - └───────────────────────────────────────────────┘ - ↓ - ┌───────────────────────────────────────────────┐ - │ Phase 5: Knowledge Graph │ - │ - Smart deduplication (optional) │ - │ - Generate embeddings (384D vectors) │ - │ - Add to HNSW index │ - │ - Save to storage (dual write) │ - │ - Update metadata index │ - │ - Create relationships │ - │ - Update graph adjacency index │ - └───────────────────────────────────────────────┘ - ↓ - ┌───────────────────────────────────────────────┐ - │ Phase 6: Persistence │ - │ - Flush HNSW index → index.json │ - │ - Flush metadata index → metadata-index.json │ - │ - Flush graph → graph-adjacency.json │ - │ - Flush VFS → .vfs/state.json │ - │ - Record in import history │ - └───────────────────────────────────────────────┘ - ↓ - ┌─────────────────────────────┐ - │ Result: Queryable │ - │ Knowledge Graph! 🎉 │ - └─────────────────────────────┘ -``` - -**What You Get**: -- ✅ Intelligent entity classification (31 types) -- ✅ Smart relationship inference (40 types) -- ✅ Semantic vector embeddings (384D) -- ✅ Fast O(log n) similarity search -- ✅ O(1) metadata filtering -- ✅ O(1) relationship traversal -- ✅ Human-readable VFS structure -- ✅ Filesystem-backed persistence (snapshot/sync the directory for off-site backup) -- ✅ Billion-scale performance -- ✅ Zero mocks, production-ready! - ---- - -## Further Reading - -- [SmartExtractor Architecture](./smart-extractor.md) -- [SmartRelationshipExtractor Architecture](./smart-relationship-extractor.md) -- [VFS Guide](./vfs-guide.md) -- [Storage Adapters](./storage-adapters.md) -- [Query Optimization](./query-optimization.md) -- [Migration to v4.x](./migrating-to-v4.md) - ---- - -**Questions?** Check the [FAQ](../faq.md) or [open an issue](https://github.com/soulcraft/brainy/issues)! 🚀 diff --git a/docs/guides/import-progress-examples.md b/docs/guides/import-progress-examples.md deleted file mode 100644 index 66f50713..00000000 --- a/docs/guides/import-progress-examples.md +++ /dev/null @@ -1,370 +0,0 @@ -# Import Progress - Usage Examples - -**How to Use Progress Tracking in Your Applications** - -Brainy provides real-time progress tracking for **all 7 supported file formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX). - -> **⚠️ KEY FEATURE:** The progress API is **100% standardized**. Write your progress handler ONCE and it works for ALL formats with zero format-specific code! See [Standard Import Progress API](./standard-import-progress.md) for the complete interface documentation. - ---- - -## 🚀 Quick Start - -### Basic Progress Tracking - -```typescript -import { Brainy } from '@soulcraft/brainy' -import * as fs from 'fs' - -const brain = await Brainy.create() - -// Import with progress tracking -const result = await brain.import(fs.readFileSync('large-file.xlsx'), { - onProgress: (progress) => { - console.log(`Progress: ${progress.stage}`) - console.log(` Message: ${progress.message}`) - console.log(` Entities: ${progress.entities || 0}`) - console.log(` Relationships: ${progress.relationships || 0}`) - } -}) - -console.log(`Import complete: ${result.entities.length} entities created`) -``` - -**Expected Output:** -``` -Progress: detecting - Message: Detecting format... - Entities: 0 - Relationships: 0 -Progress: extracting - Message: Loading Excel workbook... - Entities: 0 - Relationships: 0 -Progress: extracting - Message: Reading sheet: Sales (1/3) - Entities: 0 - Relationships: 0 -Progress: extracting - Message: Parsing Excel (33%) - Entities: 0 - Relationships: 0 -Progress: extracting - Message: Reading sheet: Products (2/3) - Entities: 0 - Relationships: 0 -... (more progress updates) -Progress: complete - Message: Import complete - Entities: 1523 - Relationships: 892 -Import complete: 1523 entities created -``` - ---- - -## 🎯 Universal Progress Handler (Works for ALL Formats) - -The examples below show format-specific messages, but **you don't need format-specific code**! The `ImportProgress` interface is the same for all formats: - -```typescript -// ONE HANDLER FOR ALL FORMATS! -function universalProgressHandler(progress) { - console.log(`[${progress.stage}] ${progress.message}`) - - if (progress.processed && progress.total) { - console.log(` Progress: ${progress.processed}/${progress.total}`) - } - - if (progress.entities || progress.relationships) { - console.log(` Extracted: ${progress.entities || 0} entities, ${progress.relationships || 0} relationships`) - } - - if (progress.throughput && progress.eta) { - console.log(` Rate: ${progress.throughput.toFixed(1)}/sec, ETA: ${Math.round(progress.eta/1000)}s`) - } -} - -// Use it for ANY format! -await brain.import(csvBuffer, { onProgress: universalProgressHandler }) -await brain.import(pdfBuffer, { onProgress: universalProgressHandler }) -await brain.import(excelBuffer, { onProgress: universalProgressHandler }) -await brain.import(jsonBuffer, { onProgress: universalProgressHandler }) -await brain.import(markdownString, { onProgress: universalProgressHandler }) -await brain.import(yamlBuffer, { onProgress: universalProgressHandler }) -await brain.import(docxBuffer, { onProgress: universalProgressHandler }) -``` - ---- - -## 📊 What Different Formats Look Like (Same Handler!) - -The examples below show **what messages look like** for different formats using the **same universal handler** above. - -### CSV Import (Row-by-Row Progress) - -```typescript -await brain.import(csvBuffer, { - format: 'csv', - onProgress: (progress) => { - if (progress.stage === 'extracting') { - // CSV reports: "Parsing CSV (45%)", "Extracted 1000 rows", etc. - console.log(progress.message) - } - } -}) -``` - -**CSV Progress Messages:** -- ✅ "Detecting CSV encoding and delimiter..." -- ✅ "Parsing CSV rows (delimiter: ",")" -- ✅ "Parsed 75%" (via bytes processed) -- ✅ "Extracted 1000 rows" -- ✅ "Converting types: 5000/10000 rows..." -- ✅ "CSV processing complete: 10000 rows" - ---- - -### PDF Import (Page-by-Page Progress) - -```typescript -await brain.import(pdfBuffer, { - format: 'pdf', - onProgress: (progress) => { - // PDF reports exact page numbers - console.log(progress.message) - // Example: "Processing page 5 of 23" - } -}) -``` - -**PDF Progress Messages:** -- ✅ "Loading PDF document..." -- ✅ "Processing 23 pages..." -- ✅ "Processing page 5 of 23" -- ✅ "Parsed 22%" (via bytes processed) -- ✅ "Extracted 156 items from PDF" -- ✅ "PDF complete: 23 pages, 156 items extracted" - ---- - -### Excel Import (Sheet-by-Sheet Progress) - -```typescript -await brain.import(excelBuffer, { - format: 'excel', - onProgress: (progress) => { - // Excel reports sheet names - console.log(progress.message) - // Example: "Reading sheet: Q2 Sales (2/5)" - } -}) -``` - -**Excel Progress Messages:** -- ✅ "Loading Excel workbook..." -- ✅ "Processing 3 sheets..." -- ✅ "Reading sheet: Sales (1/3)" -- ✅ "Parsing Excel (33%)" (via bytes processed) -- ✅ "Extracted 5234 rows from Excel" -- ✅ "Excel complete: 3 sheets, 5234 rows" - ---- - -### JSON Import (Node Traversal) - -```typescript -await brain.import(jsonBuffer, { - format: 'json', - onProgress: (progress) => { - // JSON reports every 10 nodes - console.log(`Processed ${progress.processed} nodes, found ${progress.entities} entities`) - } -}) -``` - ---- - -### Markdown Import (Section-by-Section) - -```typescript -await brain.import(markdownString, { - format: 'markdown', - onProgress: (progress) => { - console.log(`Section ${progress.processed}/${progress.total}`) - } -}) -``` - ---- - -## 🎯 Building Progress UI Components - -### React Progress Bar - -```typescript -function ImportProgress({ file }: { file: File }) { - const [progress, setProgress] = useState({ - stage: 'idle', - message: '', - percent: 0, - entities: 0, - relationships: 0 - }) - - const handleImport = async () => { - const buffer = await file.arrayBuffer() - - await brain.import(Buffer.from(buffer), { - onProgress: (p) => { - setProgress({ - stage: p.stage, - message: p.message, - // Estimate percentage from stage - percent: { - detecting: 10, - extracting: 50, - 'storing-vfs': 80, - 'storing-graph': 90, - complete: 100 - }[p.stage] || 0, - entities: p.entities || 0, - relationships: p.relationships || 0 - }) - } - }) - } - - return ( -
- -

{progress.message}

-

Entities: {progress.entities} | Relationships: {progress.relationships}

-
- ) -} -``` - ---- - -### CLI Progress Spinner - -```typescript -import ora from 'ora' - -const spinner = ora('Starting import...').start() - -await brain.import(buffer, { - onProgress: (progress) => { - spinner.text = progress.message - - if (progress.stage === 'complete') { - spinner.succeed(`Import complete: ${progress.entities} entities`) - } - } -}) -``` - -**CLI Output:** -``` -⠋ Detecting format... -⠙ Loading Excel workbook... -⠹ Reading sheet: Sales (1/3) -⠸ Parsing Excel (33%) -⠼ Reading sheet: Products (2/3) -... -✔ Import complete: 1523 entities -``` - ---- - -### Progress Dashboard with ETA - -```typescript -let startTime = Date.now() -let lastUpdate = startTime - -await brain.import(buffer, { - onProgress: (progress) => { - const elapsed = Date.now() - startTime - const rate = progress.entities / (elapsed / 1000) // entities/sec - - console.clear() - console.log('Import Progress Dashboard') - console.log('========================') - console.log(`Stage: ${progress.stage}`) - console.log(`Status: ${progress.message}`) - console.log(`Entities: ${progress.entities}`) - console.log(`Relationships: ${progress.relationships}`) - console.log(`Rate: ${rate.toFixed(1)} entities/sec`) - console.log(`Elapsed: ${(elapsed / 1000).toFixed(1)}s`) - } -}) -``` - ---- - -## 🔧 Advanced: Format-Specific Optimization - -### Detecting Format to Show Appropriate Progress - -```typescript -const formatMessages = { - csv: (p) => `CSV: ${p.message}`, - pdf: (p) => `PDF: ${p.message}`, - excel: (p) => `Excel: ${p.message}`, - json: (p) => `JSON: ${p.processed} nodes, ${p.entities} entities`, - markdown: (p) => `Markdown: Section ${p.processed}/${p.total}`, - yaml: (p) => `YAML: ${p.processed} nodes`, - docx: (p) => `DOCX: ${p.processed} paragraphs` -} - -await brain.import(buffer, { - onProgress: (progress) => { - // Format is available in progress.stage metadata - const message = formatMessages[detectedFormat]?.(progress) || progress.message - console.log(message) - } -}) -``` - ---- - -## ⚡ Performance Tips - -### Throttle UI Updates - -```typescript -let lastUIUpdate = 0 -const THROTTLE_MS = 100 // Update UI max once per 100ms - -await brain.import(buffer, { - onProgress: (progress) => { - const now = Date.now() - if (now - lastUIUpdate < THROTTLE_MS && progress.stage !== 'complete') { - return // Skip this update - } - - lastUIUpdate = now - updateUI(progress) // Only update every 100ms - } -}) -``` - -**Note:** Brainy already throttles progress callbacks internally, but additional UI throttling can help with heavy rendering. - ---- - -## 📝 Summary - -✅ **All 7 formats** have consistent progress reporting -✅ **Real-time updates** during long imports (no more "0%" hangs) -✅ **Contextual messages** show exactly what's happening -✅ **Build reliable tools** with standardized progress callbacks -✅ **Problem SOLVED** - users see progress throughout import - -**Files Modified:** -- 3 handlers: `csvHandler.ts`, `pdfHandler.ts`, `excelHandler.ts` -- 7 importers: `SmartCSVImporter.ts`, `SmartPDFImporter.ts`, `SmartExcelImporter.ts`, `SmartJSONImporter.ts`, `SmartMarkdownImporter.ts`, `SmartYAMLImporter.ts`, `SmartDOCXImporter.ts` - -**Result:** Comprehensive, consistent progress tracking across ALL import formats! diff --git a/docs/guides/import-progress-implementation.md b/docs/guides/import-progress-implementation.md deleted file mode 100644 index efc82be8..00000000 --- a/docs/guides/import-progress-implementation.md +++ /dev/null @@ -1,734 +0,0 @@ -# Import Progress Implementation Guide -**For Developers: How to Add Progress Tracking to ANY File Handler** - -> This guide shows the **standard pattern** for implementing rich progress tracking in Brainy import handlers. Follow this template for **all 7 supported formats** (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX) or any future file format. - ---- - -## 📊 Supported Formats & Consistent Progress Reporting - -> **⚠️ IMPORTANT FOR DEVELOPERS:** The public API (`ImportProgress`) is 100% standardized across all formats. You can build ONE progress handler that works for CSV, PDF, Excel, JSON, Markdown, YAML, and DOCX with **zero format-specific code**. See [Standard Import Progress API](./standard-import-progress.md) for details. - -**ALL 7 formats now have consistent, standardized progress reporting** for building reliable import tools: - -| Format | Category | Progress Points | File Location | Status | -|--------|----------|-----------------|---------------|--------| -| **CSV** | Tabular | Parsing → Row extraction → Type conversion → Complete | `handlers/csvHandler.ts` + `SmartCSVImporter.ts` | ✅ Complete | -| **PDF** | Document | Loading → Page-by-page → Item extraction → Complete | `handlers/pdfHandler.ts` + `SmartPDFImporter.ts` | ✅ Complete | -| **Excel** | Tabular | Loading → Sheet-by-sheet → Row extraction → Type conversion → Complete | `handlers/excelHandler.ts` + `SmartExcelImporter.ts` | ✅ Complete | -| **JSON** | Structured | Parsing → Node traversal (every 10 nodes) → Complete | `SmartJSONImporter.ts` | ✅ Complete | -| **Markdown** | Document | Parsing → Section-by-section → Complete | `SmartMarkdownImporter.ts` | ✅ Complete | -| **YAML** | Structured | Parsing → Node traversal (every 10 nodes) → Complete | `SmartYAMLImporter.ts` | ✅ Complete | -| **DOCX** | Document | Parsing → Paragraph-by-paragraph (every 10) → Complete | `SmartDOCXImporter.ts` | ✅ Complete | - -### The Standard Public API - -**Developers calling `brain.import()` see ONE standardized interface** regardless of format: - -```typescript -// THE PUBLIC API - Same for ALL 7 formats! -brain.import(buffer, { - onProgress: (progress: ImportProgress) => { - // These fields work for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX - progress.stage // 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete' - progress.message // Human-readable status (varies by format, always readable) - progress.processed // Items processed (optional) - progress.total // Total items (optional) - progress.entities // Entities extracted (optional) - progress.relationships // Relationships inferred (optional) - progress.throughput // Items/sec (optional, during extraction) - progress.eta // Time remaining in ms (optional) - } -}) -``` - -**Internal Implementation** (for developers adding new format handlers): - -The table below shows how formats implement progress *internally*. Normal developers don't need to know this - they just use the standard `ImportProgress` interface above! - -```typescript -// Internal: Binary formats use handler hooks (you added these!) -interface FormatHandlerProgressHooks { - onBytesProcessed?: (bytes: number) => void - onCurrentItem?: (message: string) => void - onDataExtracted?: (count: number, total?: number) => void -} - -// Internal: Text formats use importer callbacks -interface ImporterProgressCallback { - onProgress?: (stats: { processed, total, entities, relationships }) => void -} - -// Both are converted to ImportProgress by ImportCoordinator! -``` - -### Developer Benefits - -✅ **Consistent API** - Same pattern across all 7 formats -✅ **Throttled Updates** - Progress reported every 10-1000 items (no spam) -✅ **Contextual Messages** - "Processing page 5 of 23", "Reading sheet: Sales (2/5)" -✅ **Real-time Estimates** - Users see progress during long imports -✅ **Build Monitoring Tools** - Reliable progress data for UIs, dashboards, CLI tools - ---- - -## 🎯 Overview - -Brainy supports comprehensive, multi-dimensional progress tracking for imports: -- **Bytes processed** (always available, most deterministic) -- **Entities extracted** (AI extraction phase) -- **Stage-specific metrics** (parsing: MB/s, extraction: entities/s) -- **Time estimates** (remaining time, total time) -- **Context information** ("Processing page 5 of 23") - -All handlers follow a simple, consistent pattern using **progress hooks**. - ---- - -## 📋 The Progress Hooks Pattern - -### 1. Progress Hooks Interface - -```typescript -export interface FormatHandlerProgressHooks { - /** - * Report bytes processed - * Call this as you read/parse the file - */ - onBytesProcessed?: (bytes: number) => void - - /** - * Set current processing context - * Examples: "Processing page 5", "Reading sheet: Q2 Sales" - */ - onCurrentItem?: (item: string) => void - - /** - * Report structured data extraction progress - * Examples: "Extracted 100 rows", "Parsed 50 paragraphs" - */ - onDataExtracted?: (count: number, total?: number) => void -} -``` - -### 2. Handler Options (Automatic) - -Progress hooks are automatically passed to your handler via `FormatHandlerOptions`: - -```typescript -export interface FormatHandlerOptions { - // ... existing options ... - - /** - * Progress hooks - * Handlers call these to report progress during processing - */ - progressHooks?: FormatHandlerProgressHooks - - /** - * Total file size in bytes - * Used for progress percentage calculation - */ - totalBytes?: number -} -``` - -**You don't need to modify FormatHandlerOptions** - it's already done! - -### 3. Standard Implementation Pattern - -Every handler follows these 5 steps: - -```typescript -async process(data: Buffer | string, options: FormatHandlerOptions): Promise { - const progressHooks = options.progressHooks // Step 1: Get hooks - - // Step 2: Report initial progress - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem('Starting import...') - } - - // Step 3: Report bytes as you process - const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data) - if (progressHooks?.onBytesProcessed) { - progressHooks.onBytesProcessed(0) // Start - } - - // ... do parsing ... - - if (progressHooks?.onBytesProcessed) { - progressHooks.onBytesProcessed(buffer.length) // Complete - } - - // Step 4: Report data extraction - if (progressHooks?.onDataExtracted) { - progressHooks.onDataExtracted(data.length, data.length) - } - - // Step 5: Report completion - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem(`Complete: ${data.length} items processed`) - } - - return { format, data, metadata } -} -``` - ---- - -## 📚 Complete Example: CSV Handler - -Here's the **ACTUAL implementation** from CSV handler showing all the key progress points: - -```typescript -async process(data: Buffer | string, options: FormatHandlerOptions): Promise { - const startTime = Date.now() - const progressHooks = options.progressHooks // ✅ Step 1 - - // Convert to buffer if string - const buffer = Buffer.isBuffer(data) ? data : Buffer.from(data, 'utf-8') - const totalBytes = buffer.length - - // ✅ Step 2: Report start - if (progressHooks?.onBytesProcessed) { - progressHooks.onBytesProcessed(0) - } - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem('Detecting CSV encoding and delimiter...') - } - - // Detect encoding - const detectedEncoding = options.encoding || this.detectEncodingSafe(buffer) - const text = buffer.toString(detectedEncoding as BufferEncoding) - - // Detect delimiter - const delimiter = options.csvDelimiter || this.detectDelimiter(text) - - // ✅ Progress update: Parsing phase - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem(`Parsing CSV rows (delimiter: "${delimiter}")...`) - } - - // Parse CSV - const records = parse(text, { /* options */ }) - - // ✅ Step 3: Report bytes processed (entire file parsed) - if (progressHooks?.onBytesProcessed) { - progressHooks.onBytesProcessed(totalBytes) - } - - const data = Array.isArray(records) ? records : [records] - - // ✅ Step 4: Report data extraction - if (progressHooks?.onDataExtracted) { - progressHooks.onDataExtracted(data.length, data.length) - } - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem(`Extracted ${data.length} rows, inferring types...`) - } - - // Type inference and conversion - const fields = data.length > 0 ? Object.keys(data[0]) : [] - const types = this.inferFieldTypes(data) - - const convertedData = data.map((row, index) => { - const converted = this.convertRow(row, types) - - // ✅ Progress update every 1000 rows (avoid spam) - if (progressHooks?.onCurrentItem && index > 0 && index % 1000 === 0) { - progressHooks.onCurrentItem(`Converting types: ${index}/${data.length} rows...`) - } - - return converted - }) - - // ✅ Step 5: Report completion - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem(`CSV processing complete: ${convertedData.length} rows`) - } - - return { - format: this.format, - data: convertedData, - metadata: { /* ... */ } - } -} -``` - -### Key Progress Points in CSV Handler - -| Progress Point | Hook Used | Message Example | -|----------------|-----------|-----------------| -| **Start** | `onCurrentItem` | "Detecting CSV encoding and delimiter..." | -| **Start bytes** | `onBytesProcessed(0)` | 0 bytes | -| **Parsing** | `onCurrentItem` | "Parsing CSV rows (delimiter: \",\")..." | -| **Bytes complete** | `onBytesProcessed(totalBytes)` | All bytes read | -| **Data extracted** | `onDataExtracted(count, total)` | Number of rows extracted | -| **Type conversion** | `onCurrentItem` (every 1000 rows) | "Converting types: 5000/10000 rows..." | -| **Complete** | `onCurrentItem` | "CSV processing complete: 10000 rows" | - ---- - -## 📖 Implementation Guide by File Type - -### Supported Formats - -Brainy supports **7 file formats** with full progress tracking: - -**Binary Formats** (use handlers): -1. **CSV** - Row-by-row parsing with type inference -2. **PDF** - Page-by-page extraction with table detection -3. **Excel** - Sheet-by-sheet processing with formula evaluation - -**Text/Structured Formats** (parse inline): -4. **JSON** - Recursive traversal of nested structures -5. **Markdown** - Section-by-section with heading extraction -6. **YAML** - Hierarchical traversal with relationship inference -7. **DOCX** - Paragraph-by-paragraph with structure analysis - ---- - -### PDF Handler (Multi-Page) - -```typescript -async process(data: Buffer, options: FormatHandlerOptions): Promise { - const progressHooks = options.progressHooks - const totalBytes = data.length - - // Report start - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem('Loading PDF document...') - } - - const pdfDoc = await loadPDF(data) - const totalPages = pdfDoc.numPages - - const extractedData: any[] = [] - let bytesProcessed = 0 - - for (let pageNum = 1; pageNum <= totalPages; pageNum++) { - // ✅ Report current page - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem(`Processing page ${pageNum} of ${totalPages}`) - } - - const page = await pdfDoc.getPage(pageNum) - const text = await page.getTextContent() - extractedData.push(this.processPageText(text)) - - // ✅ Estimate bytes processed (pages are sequential) - bytesProcessed = Math.floor((pageNum / totalPages) * totalBytes) - if (progressHooks?.onBytesProcessed) { - progressHooks.onBytesProcessed(bytesProcessed) - } - - // ✅ Report extraction progress - if (progressHooks?.onDataExtracted) { - progressHooks.onDataExtracted(pageNum, totalPages) - } - } - - // Final progress - if (progressHooks?.onBytesProcessed) { - progressHooks.onBytesProcessed(totalBytes) - } - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem(`PDF complete: ${totalPages} pages processed`) - } - - return { format: 'pdf', data: extractedData, metadata: { /* ... */ } } -} -``` - -### Excel Handler (Multi-Sheet) - -```typescript -async process(data: Buffer, options: FormatHandlerOptions): Promise { - const progressHooks = options.progressHooks - const totalBytes = data.length - - // Load workbook - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem('Loading Excel workbook...') - } - - const workbook = XLSX.read(data) - const sheetNames = options.excelSheets === 'all' - ? workbook.SheetNames - : (options.excelSheets || [workbook.SheetNames[0]]) - - const allData: any[] = [] - let bytesProcessed = 0 - - for (let i = 0; i < sheetNames.length; i++) { - const sheetName = sheetNames[i] - - // ✅ Report current sheet - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem(`Reading sheet: ${sheetName} (${i + 1}/${sheetNames.length})`) - } - - const sheet = workbook.Sheets[sheetName] - const sheetData = XLSX.utils.sheet_to_json(sheet) - allData.push(...sheetData) - - // ✅ Estimate bytes processed (sheets processed sequentially) - bytesProcessed = Math.floor(((i + 1) / sheetNames.length) * totalBytes) - if (progressHooks?.onBytesProcessed) { - progressHooks.onBytesProcessed(bytesProcessed) - } - - // ✅ Report data extraction - if (progressHooks?.onDataExtracted) { - progressHooks.onDataExtracted(allData.length, undefined) // Total unknown until done - } - } - - // Final progress - if (progressHooks?.onBytesProcessed) { - progressHooks.onBytesProcessed(totalBytes) - } - if (progressHooks?.onCurrentItem) { - progressHooks.onCurrentItem(`Excel complete: ${sheetNames.length} sheets, ${allData.length} rows`) - } - - return { format: 'xlsx', data: allData, metadata: { /* ... */ } } -} -``` - -### JSON Importer (Recursive Traversal) - -```typescript -async extract(data: any, options: SmartJSONOptions = {}): Promise { - // ✅ Report parsing start - options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) - - // Parse JSON if string - let jsonData = typeof data === 'string' ? JSON.parse(data) : data - - // ✅ Report parsing complete - options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) - - // Traverse and extract (reports progress every 10 nodes) - const entities: ExtractedJSONEntity[] = [] - const relationships: ExtractedJSONRelationship[] = [] - let nodesProcessed = 0 - - await this.traverseJSON( - jsonData, - entities, - relationships, - () => { - nodesProcessed++ - if (nodesProcessed % 10 === 0) { - options.onProgress?.({ - processed: nodesProcessed, - entities: entities.length, - relationships: relationships.length - }) - } - } - ) - - // ✅ Report completion - options.onProgress?.({ - processed: nodesProcessed, - entities: entities.length, - relationships: relationships.length - }) - - return { nodesProcessed, entitiesExtracted: entities.length, ... } -} -``` - -### Markdown Importer (Section-Based) - -```typescript -async extract(markdown: string, options: SmartMarkdownOptions = {}): Promise { - // ✅ Report parsing start - options.onProgress?.({ processed: 0, total: 0, entities: 0, relationships: 0 }) - - // Parse markdown into sections - const parsedSections = this.parseMarkdown(markdown, options) - - // ✅ Report parsing complete - options.onProgress?.({ processed: 0, total: parsedSections.length, entities: 0, relationships: 0 }) - - // Process each section (reports progress after each section) - const sections: MarkdownSection[] = [] - for (let i = 0; i < parsedSections.length; i++) { - const section = await this.processSection(parsedSections[i], options) - sections.push(section) - - options.onProgress?.({ - processed: i + 1, - total: parsedSections.length, - entities: sections.reduce((sum, s) => sum + s.entities.length, 0), - relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0) - }) - } - - // ✅ Report completion - options.onProgress?.({ - processed: sections.length, - total: sections.length, - entities: sections.reduce((sum, s) => sum + s.entities.length, 0), - relationships: sections.reduce((sum, s) => sum + s.relationships.length, 0) - }) - - return { sectionsProcessed: sections.length, ... } -} -``` - -### YAML Importer (Hierarchical) - -```typescript -async extract(yamlContent: string | Buffer, options: SmartYAMLOptions = {}): Promise { - // ✅ Report parsing start - options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) - - // Parse YAML - const yamlString = typeof yamlContent === 'string' ? yamlContent : yamlContent.toString('utf-8') - const data = yaml.load(yamlString) - - // ✅ Report parsing complete - options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) - - // Traverse YAML structure (reports progress every 10 nodes) - // ... similar to JSON traversal ... - - // ✅ Report completion (already implemented) - options.onProgress?.({ - processed: nodesProcessed, - entities: entities.length, - relationships: relationships.length - }) - - return { nodesProcessed, entitiesExtracted: entities.length, ... } -} -``` - -### DOCX Importer (Paragraph-Based) - -```typescript -async extract(buffer: Buffer, options: SmartDOCXOptions = {}): Promise { - // ✅ Report parsing start - options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) - - // Extract text and HTML using Mammoth - const textResult = await mammoth.extractRawText({ buffer }) - const htmlResult = await mammoth.convertToHtml({ buffer }) - - // ✅ Report parsing complete - options.onProgress?.({ processed: 0, entities: 0, relationships: 0 }) - - // Process paragraphs (reports progress every 10 paragraphs) - const paragraphs = textResult.value.split(/\n\n+/).filter(p => p.trim().length >= minLength) - - for (let i = 0; i < paragraphs.length; i++) { - await this.processParagraph(paragraphs[i]) - - if (i % 10 === 0) { - options.onProgress?.({ - processed: i + 1, - entities: entities.length, - relationships: relationships.length - }) - } - } - - // ✅ Report completion (already implemented) - options.onProgress?.({ - processed: paragraphs.length, - entities: entities.length, - relationships: relationships.length - }) - - return { paragraphsProcessed: paragraphs.length, ... } -} -``` - ---- - -## 🎯 Best Practices - -### 1. Always Check if Hooks Exist - -Progress hooks are **optional**. Always check before calling: - -```typescript -// ✅ Good - safe -if (progressHooks?.onBytesProcessed) { - progressHooks.onBytesProcessed(bytes) -} - -// ❌ Bad - will crash if hooks undefined -progressHooks.onBytesProcessed(bytes) // TypeError! -``` - -### 2. Report Bytes at Start and End - -```typescript -// ✅ Good - clear start and end -progressHooks?.onBytesProcessed(0) // Start -// ... processing ... -progressHooks?.onBytesProcessed(totalBytes) // End - -// ❌ Bad - no clear boundaries -// ... just start processing without reporting start -``` - -### 3. Throttle Frequent Updates - -```typescript -// ✅ Good - report every 1000 items -for (let i = 0; i < items.length; i++) { - processItem(items[i]) - - if (i > 0 && i % 1000 === 0) { - progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) - } -} - -// ❌ Bad - report EVERY item (spam!) -for (let i = 0; i < items.length; i++) { - processItem(items[i]) - progressHooks?.onCurrentItem(`Processing: ${i}/${items.length}`) // 1M callbacks! -} -``` - -### 4. Provide Contextual Messages - -```typescript -// ✅ Good - specific and helpful -progressHooks?.onCurrentItem('Parsing CSV rows (delimiter: ",")') -progressHooks?.onCurrentItem('Processing page 5 of 23') -progressHooks?.onCurrentItem('Reading sheet: Q2 Sales Data') - -// ❌ Bad - vague -progressHooks?.onCurrentItem('Processing...') -progressHooks?.onCurrentItem('Working...') -``` - -### 5. Report Data Extraction with Totals (if known) - -```typescript -// ✅ Good - total known -progressHooks?.onDataExtracted(100, 1000) // 100 of 1000 rows - -// ✅ Also good - total unknown (streaming) -progressHooks?.onDataExtracted(100, undefined) // 100 rows so far - -// ✅ Also good - complete -progressHooks?.onDataExtracted(1000, 1000) // All 1000 rows -``` - ---- - -## 🔧 Testing Your Handler - -### Manual Test - -```typescript -import { CSVHandler } from './csvHandler.js' -import * as fs from 'fs' - -const handler = new CSVHandler() -const data = fs.readFileSync('./test.csv') - -const result = await handler.process(data, { - filename: 'test.csv', - progressHooks: { - onBytesProcessed: (bytes) => { - console.log(`Bytes: ${bytes}`) - }, - onCurrentItem: (item) => { - console.log(`Status: ${item}`) - }, - onDataExtracted: (count, total) => { - console.log(`Extracted: ${count}${total ? `/${total}` : ''}`) - } - } -}) - -console.log(`Complete: ${result.data.length} rows`) -``` - -### Expected Output - -``` -Status: Detecting CSV encoding and delimiter... -Bytes: 0 -Status: Parsing CSV rows (delimiter: ",")... -Bytes: 52438 -Extracted: 1000/1000 -Status: Extracted 1000 rows, inferring types... -Status: CSV processing complete: 1000 rows -Complete: 1000 rows -``` - ---- - -## 📊 Progress Flow Diagram - -``` -User Imports File - ↓ -ImportManager - ↓ -Creates ProgressTracker - ↓ -Calls Handler.process() with progressHooks - ↓ -Handler Reports Progress: - ├─ onBytesProcessed(0) → ProgressTracker → overall_progress calculated - ├─ onCurrentItem("Parsing...") → ProgressTracker → stage_message updated - ├─ onBytesProcessed(bytes) → ProgressTracker → bytes_per_second calculated - ├─ onDataExtracted(count) → ProgressTracker → entities_extracted updated - └─ onCurrentItem("Complete") → ProgressTracker → final progress - ↓ -ProgressTracker emits to callback (throttled 100ms) - ↓ -User sees: - "Overall: 45% | PARSING | 12.5 MB/s | Parsing CSV rows..." -``` - ---- - -## ✅ Checklist for New Handlers - -When implementing a new file format handler: - -- [ ] Get `progressHooks` from `options` -- [ ] Get `totalBytes` (if available) -- [ ] Report `onBytesProcessed(0)` at start -- [ ] Report `onCurrentItem()` for key stages -- [ ] Report `onBytesProcessed()` as you process -- [ ] Report `onDataExtracted()` when you extract data -- [ ] Throttle frequent updates (every 1000 items max) -- [ ] Report `onBytesProcessed(totalBytes)` at end -- [ ] Report final `onCurrentItem()` with summary -- [ ] Test with progress callback to verify output - ---- - -## 🎓 Summary - -**The Pattern (5 Steps)**: -1. Get `progressHooks` from options -2. Report start (`onBytesProcessed(0)`, `onCurrentItem("Starting...")`) -3. Report progress as you process (`onBytesProcessed(bytes)`, `onCurrentItem("Page 5...")`) -4. Report data extraction (`onDataExtracted(count, total)`) -5. Report completion (`onBytesProcessed(totalBytes)`, `onCurrentItem("Complete")`) - -**Always Check**: `progressHooks?.method()` - -**Throttle**: Report every N items, not every single item - -**Context**: Provide specific, helpful messages - -**Testing**: Use manual test with console.log callbacks - ---- - -**This pattern makes it trivial to add progress tracking to ANY file format. Copy this template and adapt for your handler!** diff --git a/docs/guides/import-quick-reference.md b/docs/guides/import-quick-reference.md deleted file mode 100644 index 7837d49e..00000000 --- a/docs/guides/import-quick-reference.md +++ /dev/null @@ -1,461 +0,0 @@ -# 📥 Import Quick Reference - -> **Quick guide to importing data into Brainy** - ---- - -## Basic Import - -```typescript -import { Brainy } from '@soulcraft/brainy' - -const brain = new Brainy() -await brain.init() - -// Import from file path -await brain.import('/path/to/data.xlsx') - -// Import from buffer -const buffer = fs.readFileSync('data.csv') -await brain.import(buffer) - -// Import from object -const jsonData = { items: [...] } -await brain.import(jsonData) -``` - ---- - -## Supported Formats - -| Format | Extensions | Auto-Detect | -|--------|------------|-------------| -| **Excel** | `.xlsx`, `.xls` | ✅ Yes | -| **CSV** | `.csv` | ✅ Yes | -| **JSON** | `.json` | ✅ Yes | -| **Markdown** | `.md` | ✅ Yes | -| **PDF** | `.pdf` | ✅ Yes | -| **YAML** | `.yaml`, `.yml` | ✅ Yes | -| **DOCX** | `.docx` | ✅ Yes | - ---- - -## Common Options - -### Basic Options - -```typescript -await brain.import(file, { - // Specify format (optional - auto-detects by default) - format: 'excel', - - // VFS destination path - vfsPath: '/imports/products', - - // Enable/disable features - createEntities: true, // Create graph entities (default: true) - createRelationships: true, // Create relationships (default: true) - preserveSource: true, // Keep original file (default: true) - - // Progress tracking - onProgress: (progress) => { - console.log(`${progress.processed}/${progress.total}`) - } -}) -``` - -### Neural Intelligence - -```typescript -await brain.import(file, { - // Entity type classification - enableNeuralExtraction: true, // Auto-classify entity types (default: true) - - // Relationship type inference - enableRelationshipInference: true, // Auto-infer relationship types (default: true) - - // Concept extraction - enableConceptExtraction: true, // Extract key concepts (default: true) - - // Confidence threshold - confidenceThreshold: 0.6 // Min confidence for extraction (default: 0.6) -}) -``` - -### Deduplication - -```typescript -await brain.import(file, { - enableDeduplication: true, // Check for duplicates (default: true) - deduplicationThreshold: 0.85 // Similarity threshold (default: 0.85) -}) -``` - -Deduplication merges entities judged duplicates — the non-primary records are -**deleted**. Set `enableDeduplication: false` to disable it entirely: the flag -gates both the inline merge during import and the background pass that runs -about 5 minutes after the last import. - -```typescript -await brain.import(file, { - enableDeduplication: false // No merging, inline or background -}) -``` - -### Import Tracking - -Track and organize imports by project: - -```typescript -await brain.import(file, { - projectId: 'worldbuilding', // Group related imports - importId: 'import-001', // Custom ID (auto-generated if not provided) - customMetadata: { // Additional metadata - campaign: 'fall-2024', - author: 'gamemaster' - } -}) - -// Query all entities in a project -const entities = await brain.find({ - where: { projectId: 'worldbuilding' } -}) - -// Query entities from specific import -const importedEntities = await brain.find({ - where: { importIds: { $includes: 'import-001' } } -}) - -// Exclude a project from search -const results = await brain.find({ - query: 'dragon', - where: { projectId: { $ne: 'archived-project' } } -}) -``` - -**All created items (entities, relationships, VFS files) are automatically tagged with:** -- `importIds: string[]` - Import operation IDs -- `projectId: string` - Project identifier -- `importedAt: number` - Timestamp -- `importFormat: string` - Format type ('excel', 'csv', etc.) -- `importSource: string` - Source filename/URL - -### VFS Organization - -```typescript -await brain.import(file, { - vfsPath: '/imports/catalog', - - // Grouping strategy - groupBy: 'type', // Group by entity type (default) - // OR - groupBy: 'sheet', // Group by Excel sheet name - // OR - groupBy: 'flat', // All entities in root directory - // OR - groupBy: 'custom', - customGrouping: (entity) => { - return `/by-category/${entity.category}` - } -}) -``` - -### Always-On Streaming - -All imports use streaming with adaptive flush intervals. Query data as it's imported: - -```typescript -await brain.import(file, { - onProgress: async (progress) => { - // Query data during import - if (progress.queryable) { - const products = await brain.find({ type: 'product', limit: 10000 }) - console.log(`${products.length} products imported so far`) - } - } -}) -``` - -**Progressive intervals** (automatic): -- 0-999 entities: Flush every 100 (frequent early updates) -- 1K-9.9K: Flush every 1000 (balanced) -- 10K+: Flush every 5000 (minimal overhead) -- Adjusts dynamically as import grows - ---- - -## Complete Example - -```typescript -import { Brainy } from '@soulcraft/brainy' -import * as fs from 'fs' - -async function importCatalog() { - const brain = new Brainy({ - storage: { - type: 'gcs', - bucket: 'my-bucket', - prefix: 'brainy/' - } - }) - await brain.init() - - const buffer = fs.readFileSync('catalog.xlsx') - - const result = await brain.import(buffer, { - format: 'excel', - vfsPath: '/imports/product-catalog', - groupBy: 'type', - - // Neural intelligence - enableNeuralExtraction: true, - enableRelationshipInference: true, - confidenceThreshold: 0.7, - - // Deduplication - enableDeduplication: true, - deduplicationThreshold: 0.85, - - // Progress tracking (streaming always enabled) - onProgress: async (progress) => { - console.log(`Stage: ${progress.stage}`) - console.log(`Progress: ${progress.processed}/${progress.total}`) - - // Query live data (available after each flush) - if (progress.queryable) { - const products = await brain.find({ type: 'product', limit: 100000 }) - const people = await brain.find({ type: 'person', limit: 100000 }) - const all = await brain.find({ limit: 100000 }) - - const stats = { - products: products.length, - people: people.length, - total: all.length - } - console.log('Current counts:', stats) - } - } - }) - - console.log('Import complete!') - console.log(`Entities: ${result.entities.length}`) - console.log(`Relationships: ${result.relationships.length}`) - console.log(`VFS path: ${result.vfs.rootPath}`) - console.log(`Processing time: ${result.stats.processingTime}ms`) - - return result -} - -importCatalog() -``` - ---- - -## Import Result - -```typescript -interface ImportResult { - importId: string - format: string - formatConfidence: number - - vfs: { - rootPath: string - directories: string[] - files: Array<{ - path: string - entityId?: string - type: 'entity' | 'metadata' | 'source' | 'relationships' - }> - } - - entities: Array<{ - id: string - name: string - type: NounType - vfsPath?: string - }> - - relationships: Array<{ - id: string - from: string - to: string - type: VerbType - }> - - stats: { - entitiesExtracted: number - relationshipsInferred: number - vfsFilesCreated: number - graphNodesCreated: number - graphEdgesCreated: number - entitiesMerged: number // From deduplication - entitiesNew: number // Newly created - processingTime: number // In milliseconds - } -} -``` - ---- - -## Progress Callback - -```typescript -interface ImportProgress { - stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete' - message: string - processed?: number // Current item number - total?: number // Total items - entities?: number // Entities extracted - relationships?: number // Relationships inferred - throughput?: number // Rows per second - eta?: number // Estimated time remaining (ms) - queryable?: boolean // Data queryable now (streaming mode) -} -``` - ---- - -## Tips & Best Practices - -### Performance - -```typescript -// Streaming is always on with adaptive intervals (zero config) -// - Small imports (<1K): Flush every 100 entities -// - Medium (1K-10K): Flush every 1000 entities -// - Large (>10K): Flush every 5000 entities - -// Disable features you don't need for faster imports -await brain.import(file, { - enableNeuralExtraction: false, // 10x faster - enableRelationshipInference: false, // 5x faster - enableConceptExtraction: false // 2x faster -}) -``` - -### Error Handling - -```typescript -try { - const result = await brain.import(file, { - vfsPath: '/imports/data', - onProgress: (p) => console.log(p.message) - }) - console.log('Success:', result.stats) -} catch (error) { - console.error('Import failed:', error.message) - - // Check partial results in VFS - const files = await brain.vfs().readdir('/imports') - console.log('Partial files:', files) -} -``` - -### Querying Imported Data - -```typescript -// After import completes -const result = await brain.import(file) - -// Find entities by type -const products = await brain.find({ type: 'Product' }) - -// Get entity relationships -const relations = await brain.related(products[0].id) - -// Search VFS -const vfsFiles = await brain.vfs().find(result.vfs.rootPath + '/**/*.json') - -// Read entity from VFS -const entity = await brain.vfs().readJSON(vfsFiles[0].path) -``` - ---- - -## Excel-Specific Tips - -### Column Detection - -Brainy auto-detects columns with flexible matching: - -| Your Column | Matches Pattern | -|-------------|-----------------| -| `Name` | term\|name\|title\|concept | -| `Description` | definition\|description\|desc\|details | -| `Type` | type\|category\|kind\|class | -| `Related` | related\|see also\|links\|references | - -### Multiple Sheets - -All sheets are processed automatically: - -```typescript -// catalog.xlsx with 3 sheets: Products, People, Places -const result = await brain.import('catalog.xlsx', { - groupBy: 'sheet' // Creates /Products/, /People/, /Places/ -}) -``` - ---- - -## CSV-Specific Tips - -### Headers - -First row is treated as headers. Ensure headers exist: - -```csv -Term,Definition,Type -Product A,Description A,Product -Product B,Description B,Product -``` - -### Large CSVs - -For large CSV files (>100K rows), streaming is automatic: - -```typescript -await brain.import(largeCsv, { - // Automatically flushes every 5000 entities (adaptive) - enableNeuralExtraction: false // Faster for large imports -}) -``` - ---- - -## JSON-Specific Tips - -### Supported Structures - -```javascript -// Array of objects -[ - { name: "Item 1", type: "Product" }, - { name: "Item 2", type: "Product" } -] - -// Nested objects (creates hierarchical relationships) -{ - "company": { - "name": "Acme Corp", - "products": [ - { "name": "Widget", "price": 9.99 } - ] - } -} -``` - ---- - -## Further Reading - -- [Import Flow Guide](./import-flow.md) - Deep dive into how imports work -- [Streaming Imports](./streaming-imports.md) - Progressive imports for large files -- [VFS Guide](./vfs-guide.md) - Working with the virtual file system -- [Type Classification](./type-classification.md) - How entity types are inferred -- [Relationship Inference](./relationship-inference.md) - How relationships are classified - ---- - -**Questions?** Check the [FAQ](../faq.md) or [open an issue](https://github.com/soulcraft/brainy/issues)! diff --git a/docs/guides/inspection.md b/docs/guides/inspection.md deleted file mode 100644 index 240e81ae..00000000 --- a/docs/guides/inspection.md +++ /dev/null @@ -1,213 +0,0 @@ ---- -title: Inspecting a Live Brainy -slug: guides/inspection -public: true -category: guides -template: guide -order: 30 -description: Operator recipes for diagnosing a running Brainy data directory — counts, queries, query plans, health checks, and snapshots — without stopping the live writer. -next: - - concepts/multi-process ---- - -# Inspecting a Live Brainy - -When something is wrong in production, you need to see what's actually in the -store. This guide covers the safe ways to query a running Brainy directory. - -## The cardinal rule - -**Never open a second writer on the same directory.** Filesystem storage will throw, and any other write path will corrupt the live writer's state. Use `Brainy.openReadOnly()` or the `brainy inspect` CLI instead. - -## The CLI is the fastest path - -```bash -# What's in this brain? -brainy inspect stats /data/brain - -# Find specific entities -brainy inspect find /data/brain --type Event --where '{"status":"paid"}' --limit 20 - -# Single entity by ID -brainy inspect get /data/brain 0b7a9... - -# Why is this query returning empty? -brainy inspect explain /data/brain --where '{"entityType":"booking"}' - -# Quick invariants -brainy inspect health /data/brain - -# Random sample (no query needed) -brainy inspect sample /data/brain --type Event --n 20 - -# Tail new writes as they happen -brainy inspect watch /data/brain --type Event - -# Save a snapshot -brainy inspect backup /data/brain /backups/brain-$(date +%Y%m%d).tar -``` - -Every subcommand internally: - -1. Asks the live writer to flush via the cross-process RPC (skip with `--no-fresh`). -2. Opens the data directory via `Brainy.openReadOnly()`. -3. Runs the query. -4. Closes cleanly. - -Results are JSON by default. Add `--pretty` for indented output. - -## When a query returns surprising results - -If `find()` returns `0` for a query you expect to match: run `inspect -explain` first. It shows which index path will serve each `where` clause: - -```bash -$ brainy inspect explain /data/brain --where '{"entityType":"booking","status":"paid"}' -{ - "query": { "where": { "entityType": "booking", "status": "paid" } }, - "fieldPlan": [ - { "field": "entityType", "path": "none", "notes": "No index entries for field..." }, - { "field": "status", "path": "column-store", "notes": "O(log n) binary search..." } - ], - "warnings": [ - "Field \"entityType\" has no index entries. find() will return [] silently." - ] -} -``` - -The `"path": "none"` is the smoking gun. It means the field has no column -store manifest and no sparse chunked index — so `find()` will return `[]` -regardless of what's actually on disk. Likely causes: - -- The writer registered the field in memory but hasn't flushed. Run - `brain.requestFlush()` from the writer side, or use `brainy inspect - --fresh` (default). -- The field name has a typo or wrong casing. -- The field is genuinely absent from every entity. - -## Health checks - -`inspect health` runs a fixed battery of cheap invariant checks: - -```bash -$ brainy inspect health /data/brain -{ - "overall": "warn", - "checks": [ - { "name": "index-parity", "status": "pass", "message": "Vector (1851) and metadata (1851) agree." }, - { "name": "field-registry", "status": "pass", "message": "23 fields registered for 1851 entities." }, - { "name": "seeded-records", "status": "warn", "message": "15 entities tagged _seeded:true." }, - { "name": "writer-heartbeat", "status": "pass", "message": "Writer healthy (PID 1774431...)." } - ] -} -``` - -Each check returns `pass`, `warn`, or `fail`. The exit code is `2` when any -check fails — useful for piping into monitoring or CI. - -## Programmatic inspection - -```typescript -import { Brainy } from '@soulcraft/brainy' - -const reader = await Brainy.openReadOnly({ - storage: { type: 'filesystem', path: '/data/brain' } -}) - -// Force the writer to flush before reading -await reader.requestFlush({ timeoutMs: 5000 }) - -// What's in there? -const stats = await reader.stats() -console.log(`${stats.entityCount} entities`, stats.entitiesByType) - -// Why is this query empty? -const plan = await reader.explain({ where: { entityType: 'booking' } }) -for (const f of plan.fieldPlan) { - console.log(`${f.field} -> ${f.path}`) -} - -// Run invariants -const health = await reader.health() -console.log(health.overall) - -await reader.close() -``` - -Every mutation method (`add`, `update`, `remove`, `relate`, `transact`, -`restore`, ...) throws on a read-only instance with a clear message. - -## Backups - -`brainy inspect backup` asks the writer to flush first, then tars the -directory. The snapshot reflects the writer's state at the moment of the -flush: - -```bash -brainy inspect backup /data/brain /backups/brain-2026-05-15.tar -``` - -For periodic backups (hourly, daily), schedule this via cron or your -container scheduler. For point-in-time recovery, use the Db API's -`db.persist(path)` — a self-contained hard-link snapshot that later writes -can never alter, restorable with `brain.restore(path, { confirm: true })`. -See [Snapshots & Time Travel](./snapshots-and-time-travel.md). - -## Comparing two stores - -`brainy inspect diff` returns a JSON summary of counts and a sample of -entity IDs present in one but not the other. Useful when debugging -replication or migrations: - -```bash -brainy inspect diff /data/brain-prod /data/brain-staging -``` - -Sample-based — for a full diff, dump both with `inspect dump` and compare -the JSONL. - -## Auditing graph-read truth - -`brain.auditGraph()` (8.6.0+) proves — or disproves — that relationship reads -return canonical truth on a given brain, without mutating anything. It walks -every stored relationship record, asks the same read path your application -uses (`related()`, VFS `readdir`) with every visibility tier included, and -classifies every discrepancy: - -```typescript -const report = await brain.auditGraph() - -report.coherent // true = related()/readdir can be trusted on this brain -report.missingFromReadsCount // records the read path omits — stale index -report.danglingEndpointsCount // relationships whose endpoint entity is gone -report.readOnlyCount // read-path edges with NO stored record — ghosts -report.visibilityHiddenCount // internal/system edges hidden by design (not a fault) -``` - -Counts are always exact; the example lists (`missingFromReads`, -`danglingEndpoints`, `readOnlyVerbIds`) are capped at `maxExamples` -(default 100) and `truncatedExamples` says so when they are. - -Run it after any engine upgrade, restore, or migration. If it reports -discrepancies, run `brain.repairIndex()` and audit again — a `coherent` -report after the repair is the verified statement that the heal worked. -Cost: one relationship-record walk plus one indexed read per distinct -source entity — safe on a live brain. - -## Repairing a corrupted store - -If invariants fail and you suspect index corruption, `inspect repair` -opens the store in writer mode and rebuilds all indexes from raw storage. -**Stop the live writer first** — `repair` will throw if another writer -holds the lock. Add `--force` only if you have personally verified the -existing lock is stale. - -```bash -brainy inspect repair /data/brain -``` - -## Multi-process safety summary - -See [concepts/multi-process](../concepts/multi-process.md) for the lock -semantics, heartbeat behavior, and what's not yet enforced on cloud -backends. diff --git a/docs/guides/installation.md b/docs/guides/installation.md deleted file mode 100644 index 0a36f632..00000000 --- a/docs/guides/installation.md +++ /dev/null @@ -1,89 +0,0 @@ ---- -title: Installation -slug: getting-started/installation -public: true -category: getting-started -template: guide -order: 1 -description: Install Brainy with npm, bun, yarn, or pnpm. Works in Node.js 22+ and Bun 1.0+ (server-only since 8.0). TypeScript included. -next: - - getting-started/quick-start - - guides/storage-adapters ---- - -# Installation - -## Requirements - -- **Node.js 22+** or **Bun 1.0+** -- TypeScript is optional — Brainy ships with full type definitions - -## Install - -```bash -npm install @soulcraft/brainy -``` - -Or with your preferred package manager: - -```bash -bun add @soulcraft/brainy -yarn add @soulcraft/brainy -pnpm add @soulcraft/brainy -``` - -## Verify - -```typescript -import { Brainy } from '@soulcraft/brainy' - -const brain = new Brainy() -await brain.init() - -console.log('Brainy ready.') -``` - -## Native Acceleration (Optional) - -For production workloads, add Cor for Rust-accelerated SIMD distance calculations and native embeddings: - -```bash -npm install @soulcraft/cor -``` - -```typescript -import { Brainy } from '@soulcraft/brainy' - -const brain = new Brainy({ plugins: ['@soulcraft/cor'] }) -await brain.init() // native providers registered during init -``` - -Cor registers native (Rust/SIMD) vector, metadata, and graph engines behind the same Brainy `find()` API — no code changes, an optional dependency for production-scale workloads. - -## Server-only since 8.0 - -Brainy 8.0 runs on Node.js 22+ and Bun 1.0+. Browser support (OPFS storage, -Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line -remains available on npm if you need it. - -## TypeScript - -Brainy ships with full TypeScript types. No `@types/` package needed: - -```typescript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' - -const brain = new Brainy() -await brain.init() - -const id = await brain.add({ - data: 'Hello, Brainy', - type: NounType.Concept, - metadata: { created: Date.now() } -}) -``` - -## Next Steps - -- [Quick Start](/docs/getting-started/quick-start) — build your first knowledge graph in 60 seconds -- [Storage Adapters](/docs/guides/storage-adapters) — choose the right storage for your deployment diff --git a/docs/guides/migrating-to-v4.md b/docs/guides/migrating-to-v4.md deleted file mode 100644 index e1b07df5..00000000 --- a/docs/guides/migrating-to-v4.md +++ /dev/null @@ -1,491 +0,0 @@ -# Migrating from Brainy v3.x to v4.x - -**Brainy v4.0.0** introduces breaking changes to the import API for improved clarity, better defaults, and more powerful features. - -This guide will help you migrate your code quickly and painlessly. - ---- - -## 🎯 Quick Migration Checklist - -If you just want to fix your code fast, here's what to do: - -- [ ] Replace `extractRelationships` with `enableRelationshipInference` -- [ ] Remove `autoDetect` (auto-detection is now always enabled) -- [ ] Replace `createFileStructure: true` with `vfsPath: '/your/path'` -- [ ] Remove `excelSheets` (all sheets are now processed automatically) -- [ ] Remove `pdfExtractTables` (table extraction is now automatic) -- [ ] Add `enableNeuralExtraction: true` to enable AI entity extraction -- [ ] Add `preserveSource: true` if you want to keep the original file - ---- - -## 📋 Option Name Changes - -### Complete Mapping Table - -| v3.x Option | v4.x Option | Action Required | -|-------------|-------------|-----------------| -| `extractRelationships` | `enableRelationshipInference` | **Rename option** | -| `autoDetect` | *(removed)* | **Delete option** (always enabled) | -| `createFileStructure` | `vfsPath` | **Replace** with VFS directory path | -| `excelSheets` | *(removed)* | **Delete option** (all sheets processed) | -| `pdfExtractTables` | *(removed)* | **Delete option** (always enabled) | -| - | `enableNeuralExtraction` | **Add option** (new in v4.x) | -| - | `enableConceptExtraction` | **Add option** (new in v4.x) | -| - | `preserveSource` | **Add option** (new in v4.x) | - ---- - -## 🔄 Migration Examples - -### Example 1: Basic Excel Import - -**Before (v3.x):** -```typescript -const result = await brain.import('./glossary.xlsx', { - extractRelationships: true, - createFileStructure: true, - groupBy: 'type' -}) -``` - -**After (v4.x):** -```typescript -const result = await brain.import('./glossary.xlsx', { - enableRelationshipInference: true, // ✅ Renamed - vfsPath: '/imports/glossary', // ✅ Replaced createFileStructure - groupBy: 'type' // ✅ No change -}) -``` - ---- - -### Example 2: Full-Featured Import - -**Before (v3.x):** -```typescript -const result = await brain.import('./data.xlsx', { - extractRelationships: true, - autoDetect: true, - createFileStructure: true, - groupBy: 'type', - enableDeduplication: true -}) -``` - -**After (v4.x):** -```typescript -const result = await brain.import('./data.xlsx', { - // AI features - enableNeuralExtraction: true, // ✅ NEW - Extract entity names - enableRelationshipInference: true, // ✅ Renamed from extractRelationships - enableConceptExtraction: true, // ✅ NEW - Extract entity types - - // VFS features - vfsPath: '/imports/data', // ✅ Replaced createFileStructure - groupBy: 'type', // ✅ No change - preserveSource: true, // ✅ NEW - Save original file - - // Performance - enableDeduplication: true // ✅ No change -}) -``` - ---- - -### Example 3: Simple Import (Defaults) - -**Before (v3.x):** -```typescript -const result = await brain.import('./data.csv', { - autoDetect: true, - extractRelationships: true -}) -``` - -**After (v4.x):** -```typescript -// Auto-detection is always enabled now -// Just enable the features you want -const result = await brain.import('./data.csv', { - enableRelationshipInference: true -}) - -// Or use all defaults (AI features enabled) -const result = await brain.import('./data.csv') -``` - ---- - -### Example 4: PDF Import - -**Before (v3.x):** -```typescript -const result = await brain.import('./document.pdf', { - pdfExtractTables: true, - extractRelationships: true, - createFileStructure: true -}) -``` - -**After (v4.x):** -```typescript -const result = await brain.import('./document.pdf', { - // pdfExtractTables removed - always enabled - enableRelationshipInference: true, - vfsPath: '/imports/documents' -}) -``` - ---- - -## 💡 Why These Changes? - -### Clearer Option Names - -**v3.x naming was ambiguous:** -- `extractRelationships` → Could mean "create relationships" or "infer relationships" -- `createFileStructure` → Doesn't explain what structure or where - -**v4.x naming is explicit:** -- `enableRelationshipInference` → Clearly means "use AI to infer semantic relationships" -- `vfsPath` → Explicitly sets the virtual filesystem directory path -- `enableNeuralExtraction` → Clearly indicates AI-powered entity extraction - -### Separation of Concerns - -**v4.x separates import features into clear categories:** - -1. **Neural/AI Features:** - - `enableNeuralExtraction` - Extract entity names and metadata - - `enableRelationshipInference` - Infer semantic relationships - - `enableConceptExtraction` - Extract entity types and concepts - -2. **VFS Features:** - - `vfsPath` - Virtual filesystem directory - - `groupBy` - Grouping strategy - - `preserveSource` - Keep original file - -3. **Performance Features:** - - `enableDeduplication` - Merge similar entities - - `confidenceThreshold` - AI confidence threshold - - `onProgress` - Progress callbacks - -### Better Defaults - -**v3.x required explicit enabling:** -```typescript -// Had to enable everything manually -await brain.import(file, { - autoDetect: true, - extractRelationships: true, - createFileStructure: true -}) -``` - -**v4.x has smart defaults:** -```typescript -// Auto-detection and AI features enabled by default -await brain.import(file) - -// Or customize specific features -await brain.import(file, { - vfsPath: '/my/data', - confidenceThreshold: 0.8 -}) -``` - ---- - -## 🆕 New Features in v4.x - -### Neural Entity Extraction -Extract entity names, types, and metadata using AI: - -```typescript -const result = await brain.import('./glossary.xlsx', { - enableNeuralExtraction: true, // Extract entity names from "Term" column - enableConceptExtraction: true, // Detect entity types (Place, Person, etc.) - confidenceThreshold: 0.7 // Minimum AI confidence (0-1) -}) - -// Result includes rich entity metadata -result.entities.forEach(entity => { - console.log(`${entity.name} (${entity.type})`) - console.log(`Confidence: ${entity.confidence}`) -}) -``` - -### VFS Integration -Imported data is organized in a virtual filesystem: - -```typescript -const result = await brain.import('./data.xlsx', { - vfsPath: '/projects/myproject/data', - groupBy: 'type', // Group by entity type - preserveSource: true // Save original .xlsx file -}) - -// Access via VFS -const vfs = brain.vfs() -const files = await vfs.readdir('/projects/myproject/data') -// ['Places/', 'Characters/', 'Concepts/', '_source.xlsx', '_metadata.json'] - -// Read entity file -const content = await vfs.readFile('/projects/myproject/data/Places/Talifar.json') -``` - -### Semantic Relationship Inference -AI infers relationship types from context: - -```typescript -const result = await brain.import('./glossary.xlsx', { - enableRelationshipInference: true -}) - -// Instead of generic "contains" relationships, -// you get semantic verbs like: -// - "capital_of" -// - "located_in" -// - "guards" -// - "part_of" -// - "related_to" - -const relations = await brain.related({ limit: 100 }) -const types = new Set(relations.map(r => r.label)) -console.log(types) -// Set { 'capital_of', 'guards', 'located_in', 'related_to' } -``` - ---- - -## 🔍 What Breaks & How to Fix It - -### Error: "Invalid import options: 'extractRelationships'" - -**Cause:** Using v3.x option name - -**Fix:** -```typescript -// Before -await brain.import(file, { extractRelationships: true }) - -// After -await brain.import(file, { enableRelationshipInference: true }) -``` - ---- - -### Error: "Invalid import options: 'autoDetect'" - -**Cause:** Using v3.x option that's been removed - -**Fix:** -```typescript -// Before -await brain.import(file, { autoDetect: true }) - -// After - just remove it (auto-detection always enabled) -await brain.import(file) -``` - ---- - -### Error: "Invalid import options: 'createFileStructure'" - -**Cause:** Using v3.x option name - -**Fix:** -```typescript -// Before -await brain.import(file, { createFileStructure: true }) - -// After - specify VFS path explicitly -await brain.import(file, { vfsPath: '/imports/mydata' }) -``` - ---- - -### Issue: Import succeeds but entities have generic names like "Entity_144" - -**Cause:** Neural extraction is disabled - -**Fix:** -```typescript -// Ensure AI features are enabled -await brain.import(file, { - enableNeuralExtraction: true, // ✅ Extract entity names - enableRelationshipInference: true, // ✅ Infer relationships - enableConceptExtraction: true // ✅ Extract types -}) -``` - ---- - -### Issue: All relationships are type "contains" - -**Cause:** Relationship inference is disabled - -**Fix:** -```typescript -// Enable relationship inference -await brain.import(file, { - enableRelationshipInference: true // ✅ Use AI to detect semantic relationships -}) -``` - ---- - -### Issue: VFS directory doesn't exist in filesystem - -**This is NORMAL!** VFS is virtual - it uses Brainy entities, not physical files. - -**How to access VFS:** -```typescript -// DON'T do this: -// ls brainy-data/vfs/ ❌ Won't work - -// DO this instead: -const vfs = brain.vfs() -await vfs.init() -const files = await vfs.readdir('/imports') // ✅ Correct -``` - ---- - -## 📦 TypeScript Users - -### Compile-Time Errors - -If you're using TypeScript, you'll get compile-time errors when using deprecated options: - -```typescript -// TypeScript will show error: -// "Type 'true' is not assignable to type 'never'" -await brain.import(file, { - extractRelationships: true // ❌ Type error -}) - -// Fix: Use correct option name -await brain.import(file, { - enableRelationshipInference: true // ✅ Type correct -}) -``` - -### IDE Autocomplete - -Your IDE will show deprecation warnings and suggest the correct option names: - -```typescript -await brain.import(file, { - extract... // IDE suggests: enableNeuralExtraction, enableRelationshipInference -}) -``` - ---- - -## 🎓 Best Practices for v4.x - -### 1. Enable All AI Features by Default - -```typescript -// Good: Enable all intelligent features -await brain.import('./data.xlsx', { - enableNeuralExtraction: true, - enableRelationshipInference: true, - enableConceptExtraction: true, - vfsPath: '/imports/data' -}) -``` - -### 2. Use VFS for Organization - -```typescript -// Good: Organize by project -await brain.import('./project-A.xlsx', { - vfsPath: '/projects/project-a/data' -}) - -await brain.import('./project-B.csv', { - vfsPath: '/projects/project-b/data' -}) -``` - -### 3. Preserve Source Files - -```typescript -// Good: Keep original files for reference -await brain.import('./important-data.xlsx', { - preserveSource: true, // Saves original .xlsx in VFS - vfsPath: '/archives/2025' -}) -``` - -### 4. Tune Confidence Threshold - -```typescript -// For high-quality data: Lower threshold -await brain.import('./curated-glossary.xlsx', { - confidenceThreshold: 0.5 // Extract more entities -}) - -// For noisy data: Higher threshold -await brain.import('./scraped-data.csv', { - confidenceThreshold: 0.8 // Only high-confidence entities -}) -``` - -### 5. Disable Deduplication for Large Imports - -```typescript -// For small imports: Keep deduplication -await brain.import('./small-data.xlsx', { - enableDeduplication: true -}) - -// For large imports (>1000 rows): Disable for performance -await brain.import('./huge-database.csv', { - enableDeduplication: false // Much faster -}) -``` - ---- - -## 🚀 Migration Automation (Future) - -We're working on an automated migration tool: - -```bash -# Coming soon -npx @soulcraft/brainy-migrate - -# Will scan your code and automatically update: -# - Option names -# - TypeScript types -# - Import patterns -``` - ---- - -## 📚 Additional Resources - -- **API Documentation:** [https://brainy.dev/docs/api/import](https://brainy.dev/docs/api/import) -- **Examples:** [examples/import-excel/](../../examples/import-excel/) -- **Changelog:** [CHANGELOG.md](../../CHANGELOG.md) -- **Support:** [GitHub Issues](https://github.com/soulcraft/brainy/issues) - ---- - -## 💬 Need Help? - -If you're stuck migrating: - -1. Check the error message - it includes migration hints -2. Review the examples in this guide -3. Open an issue on GitHub with your use case -4. Join our Discord community for real-time help - ---- - -**Happy migrating! 🎉** diff --git a/docs/guides/model-loading.md b/docs/guides/model-loading.md index e5b7b1d6..020c420e 100644 --- a/docs/guides/model-loading.md +++ b/docs/guides/model-loading.md @@ -1,238 +1,358 @@ -# Model Loading Guide +# 🤖 Model Loading Guide -Brainy uses AI embedding models to understand and process your data. With the Candle WASM engine, the model is **embedded at compile time** - no downloads, no configuration, no external dependencies. +Brainy uses AI embedding models to understand and process your data. This guide explains how model loading works and how to handle different scenarios. -## Zero Configuration (Default) +## 🚀 Zero Configuration (Default) -**For all developers, no configuration is needed:** +**For most developers, no configuration is needed:** ```typescript const brain = new Brainy() -await brain.init() // Model is already embedded - nothing to download! +await brain.init() // Models load automatically ``` **What happens automatically:** -1. Candle WASM module loads (~90MB, includes model weights) -2. Model initializes in ~200ms -3. Ready to use immediately +1. Checks for local models in `./models/` +2. Downloads All-MiniLM-L6-v2 if needed (384 dimensions) +3. Configures optimal settings for your environment +4. Ready to use immediately -**No downloads. No CDN. No configuration. Just works.** +## 📦 Model Loading Cascade -## How It Works - -The all-MiniLM-L6-v2 model is embedded in the WASM binary using Rust's `include_bytes!` macro: +Brainy tries multiple sources in this order: ``` -candle_embeddings_bg.wasm (~90MB) -├── Candle ML Runtime (~3MB) -├── Model Weights (safetensors format, ~87MB) -└── Tokenizer (HuggingFace tokenizers, ~450KB) +1. LOCAL CACHE (./models/) + ↓ (if not found) +2. CDN DOWNLOAD (fast mirrors) + ↓ (if fails) +3. GITHUB RELEASES (github.com/xenova/transformers.js) + ↓ (if fails) +4. HUGGINGFACE HUB (huggingface.co) + ↓ (if fails) +5. FALLBACK STRATEGIES (different model variants) ``` -This single WASM file contains everything needed for sentence embeddings. - -## Environments - -### Bun (Recommended) - -```bash -# Bun as a runtime — supported and recommended -bun add @soulcraft/brainy -bun run server.ts -``` - -Brainy is pure WebAssembly with no native binaries, so the module graph stays -bundler-friendly. Single-binary `bun build --compile` is **not a supported -target** at present: Bun 1.3.10 has a `--compile` codegen regression -(`__promiseAll is not defined`) triggered by top-level `await` in the bundled -graph. Run Brainy under the Bun runtime (above) instead. - -### Node.js - -```typescript -// Standard Node.js -node dist/server.js - -// Runs identically to Bun -``` +## 🌍 Environment-Specific Behavior ### Browser - ```typescript -// Model loads via WASM (single file, no additional assets) +// Automatically configured for browsers +const brain = new Brainy() // Works in React, Vue, vanilla JS +await brain.init() // Downloads models via CDN +``` + +### Node.js Development +```typescript +// Zero config - downloads to ./models/ const brain = new Brainy() -await brain.init() +await brain.init() // Downloads once, cached forever +``` + +### Production Server +```typescript +// Preload models during build/deployment +const brain = new Brainy() +await brain.init() // Uses cached local models ``` ### Docker/Kubernetes - ```dockerfile -FROM oven/bun:1.1 -WORKDIR /app -COPY package*.json ./ -RUN bun install -COPY . . -EXPOSE 3000 -CMD ["bun", "run", "server.ts"] - -# That's it! No model download step needed. -# Model is embedded in the npm package. +# Dockerfile - preload models +RUN npm run download-models +ENV BRAINY_ALLOW_REMOTE_MODELS=false ``` -## Model Information +## 🛠️ Manual Model Management -### all-MiniLM-L6-v2 (Embedded) -- **Dimensions**: 384 (fixed) -- **Format**: Safetensors (FP32) -- **Size**: ~87MB (embedded in WASM) -- **Total WASM Size**: ~90MB -- **Language**: English-optimized, works with all languages -- **Inference**: ~2-10ms per embedding -- **Initialization**: ~200ms - -### Memory Usage -- **Loaded WASM**: ~90MB -- **Inference peak**: ~140MB total -- **Steady state**: ~100MB - -## Comparing to Previous Architecture - -| Feature | Before (ONNX) | Now (Candle WASM) | -|---------|--------------|-------------------| -| Model downloads | Required on first use | None - embedded | -| External dependencies | onnxruntime-web | None | -| Model files | model.onnx, tokenizer.json | Embedded in WASM | -| Offline support | Required setup | Works by default | -| Bun compile | Broken | Works | -| Configuration | Environment variables | None needed | - -## Troubleshooting - -### "Failed to initialize Candle Embedding Engine" - -**Cause**: WASM loading issue. - -**Solutions**: +### Pre-download Models ```bash -# Rebuild the WASM -npm run build:candle +# Download models during build/deployment +npm run download-models -# Verify WASM exists -ls dist/embeddings/wasm/pkg/candle_embeddings_bg.wasm -# Should be ~90MB +# Custom location +BRAINY_MODELS_PATH=./my-models npm run download-models ``` -### Out of Memory +### Verify Models +```bash +# Check if models exist +ls ./models/Xenova/all-MiniLM-L6-v2/ -**Cause**: Container/environment has less than 256MB RAM. - -**Solutions**: -```dockerfile -# Increase memory limit (recommended: 512MB+) -docker run -m 512m my-app +# Should see: +# - config.json +# - tokenizer.json +# - onnx/model.onnx ``` -### Slow Initialization (>500ms) - -**Cause**: Cold start, large WASM parsing. - -**Solutions**: -```typescript -// Initialize once at startup, not per-request -await brain.init() // Do this once - -// Then reuse for all requests -app.get('/api', async (req, res) => { - const results = await brain.find(req.query) - res.json(results) -}) -``` - -## Migration from Previous Versions - -### From v6.x (ONNX) - -No changes needed for most users: - -```typescript -// Same API - just upgrade -const brain = new Brainy() -await brain.init() -``` - -**What's removed:** -- `BRAINY_ALLOW_REMOTE_MODELS` - no downloads -- `BRAINY_MODELS_PATH` - no external model files -- `npm run download-models` - no longer needed - -**What's new:** -- Faster initialization -- Bundler-friendly (pure WASM, no native binaries) -- No network requirements - -### From Custom Embedding Functions - -If you provided a custom embedding function, it still works: - +### Custom Model Path ```typescript const brain = new Brainy({ - embeddingFunction: myCustomEmbedder // Still supported + embedding: { + cacheDir: './custom-models' + } }) ``` -## Advanced: Building Custom WASM - -For contributors who want to modify the embedding engine: +## 🔒 Offline & Air-Gapped Environments +### Complete Offline Setup ```bash -# Navigate to Candle WASM source -cd src/embeddings/candle-wasm +# 1. Download models on connected machine +npm run download-models -# Build with wasm-pack -wasm-pack build --target web --release +# 2. Copy models to offline machine +cp -r ./models /path/to/offline/project/ -# Copy to pkg folder -cp pkg/* ../wasm/pkg/ - -# Build TypeScript -npm run build +# 3. Force local-only mode +export BRAINY_ALLOW_REMOTE_MODELS=false ``` -## Best Practices +### Container/Server Deployment +```dockerfile +FROM node:18 +WORKDIR /app +COPY package*.json ./ +RUN npm ci + +# Download models during build +RUN npm run download-models + +# Force local-only in production +ENV BRAINY_ALLOW_REMOTE_MODELS=false + +COPY . . +EXPOSE 3000 +CMD ["npm", "start"] +``` + +## ⚙️ Environment Variables + +### BRAINY_ALLOW_REMOTE_MODELS +Controls whether remote model downloads are allowed: + +```bash +# Allow remote downloads (default in most environments) +export BRAINY_ALLOW_REMOTE_MODELS=true + +# Force local-only (recommended for production) +export BRAINY_ALLOW_REMOTE_MODELS=false +``` + +### BRAINY_MODELS_PATH +Custom model storage location: + +```bash +# Custom model path +export BRAINY_MODELS_PATH=/opt/brainy/models + +# Relative path +export BRAINY_MODELS_PATH=./my-custom-models +``` + +## 🚨 Troubleshooting + +### "Failed to load embedding model" Error + +**Cause**: Models not found locally and remote download blocked/failed. + +**Solutions**: +```bash +# Option 1: Allow remote downloads +export BRAINY_ALLOW_REMOTE_MODELS=true + +# Option 2: Download models manually +npm run download-models + +# Option 3: Check internet connectivity +ping huggingface.co + +# Option 4: Use custom model path +export BRAINY_MODELS_PATH=/path/to/existing/models +``` + +### Models Download Very Slowly + +**Cause**: Network issues or regional restrictions. + +**Solutions**: +```bash +# Pre-download during build/CI +npm run download-models + +# Use faster mirrors (automatic in newer versions) +# No action needed - Brainy tries multiple CDNs +``` + +### Container Out of Memory During Model Load + +**Cause**: Limited container memory during model initialization. + +**Solutions**: +```dockerfile +# Increase memory limit +docker run -m 2g my-app + +# Use quantized models (default) +ENV BRAINY_MODEL_DTYPE=q8 + +# Pre-load models at build time (recommended) +RUN npm run download-models +``` + +### Permission Denied Creating Model Cache + +**Cause**: Write permissions for model cache directory. + +**Solutions**: +```bash +# Make directory writable +chmod 755 ./models + +# Use custom writable path +export BRAINY_MODELS_PATH=/tmp/brainy-models + +# Or use memory-only storage +const brain = new Brainy({ + storage: { forceMemoryStorage: true } +}) +``` + +## 🎯 Best Practices ### Development ```typescript -// Just works - no setup +// ✅ Zero config - just works const brain = new Brainy() await brain.init() ``` ### Production -```typescript -// Initialize once at startup -const brain = new Brainy() -await brain.init() +```dockerfile +# ✅ Pre-download models +RUN npm run download-models -// Singleton pattern recommended -export { brain } +# ✅ Force local-only +ENV BRAINY_ALLOW_REMOTE_MODELS=false + +# ✅ Verify models exist +RUN test -f ./models/Xenova/all-MiniLM-L6-v2/onnx/model.onnx ``` -### Deployment -```bash -# Option 1: Bun runtime -bun run server.ts +### CI/CD Pipeline +```yaml +# .github/workflows/build.yml +- name: Download AI Models + run: npm run download-models + +- name: Verify Models + run: | + test -f ./models/Xenova/all-MiniLM-L6-v2/onnx/model.onnx + echo "✅ Models verified" -# Option 2: Docker -docker build -t my-app . -docker run -p 3000:3000 my-app +- name: Test Offline Mode + env: + BRAINY_ALLOW_REMOTE_MODELS: false + run: npm test +``` + +### Lambda/Serverless +```typescript +// ✅ Models in deployment package +const brain = new Brainy({ + embedding: { + localFilesOnly: true, // No downloads in lambda + cacheDir: './models' // Bundled with deployment + } +}) +``` + +## 📊 Model Information + +### All-MiniLM-L6-v2 (Default) +- **Dimensions**: 384 (fixed) +- **Size**: ~80MB compressed, ~330MB uncompressed +- **Language**: English (optimized) +- **Speed**: Very fast inference +- **Quality**: High quality for most use cases + +### Model Files Structure +``` +models/ +└── Xenova/ + └── all-MiniLM-L6-v2/ + ├── config.json # Model configuration + ├── tokenizer.json # Text tokenizer + ├── tokenizer_config.json + └── onnx/ + ├── model.onnx # Main model file + └── model_quantized.onnx # Optimized version +``` + +## 🔄 Migration from Other Embedding Solutions + +### From OpenAI Embeddings +```typescript +// Before: OpenAI API calls +const response = await openai.embeddings.create({ + model: "text-embedding-ada-002", + input: "Your text" +}) + +// After: Local Brainy embeddings +const brain = new Brainy() +await brain.init() // One-time setup +const id = await brain.add("Your text", { nounType: 'content' }) // Embedded automatically +``` + +### From Sentence Transformers +```python +# Before: Python sentence-transformers +from sentence_transformers import SentenceTransformer +model = SentenceTransformer('all-MiniLM-L6-v2') + +# After: JavaScript Brainy (same model!) +const brain = new Brainy() // Uses same all-MiniLM-L6-v2 +await brain.init() +``` + +## 🚀 Advanced Configuration + +### Custom Embedding Options +```typescript +const brain = new Brainy({ + embedding: { + model: 'Xenova/all-MiniLM-L6-v2', // Default + dtype: 'q8', // Quantized for speed + device: 'cpu', // CPU inference + localFilesOnly: false, // Allow downloads + verbose: true // Debug logging + } +}) +``` + +### Multiple Model Support (Advanced) +```typescript +// Use custom embedding function +import { createEmbeddingFunction } from 'brainy' + +const customEmbedder = createEmbeddingFunction({ + model: 'Xenova/all-MiniLM-L12-v2', // Larger model + dtype: 'fp32' // Higher precision +}) + +const brain = new Brainy({ + embeddingFunction: customEmbedder +}) ``` --- -## Additional Resources +## 📚 Additional Resources -- [Production Service Architecture](../PRODUCTION_SERVICE_ARCHITECTURE.md) -- [Zero Configuration Guide](../architecture/zero-config.md) +- [Zero Configuration Guide](./zero-config.md) +- [Enterprise Deployment](./enterprise-deployment.md) - [Troubleshooting Guide](../troubleshooting.md) +- [API Reference](../api/README.md) -**Need help?** [Open an issue](https://github.com/soulcraftlabs/brainy/issues) +**Need help?** Check our [troubleshooting guide](../troubleshooting.md) or [open an issue](https://github.com/your-repo/brainy/issues). \ No newline at end of file diff --git a/docs/guides/natural-language.md b/docs/guides/natural-language.md index 00c50425..4731af92 100644 --- a/docs/guides/natural-language.md +++ b/docs/guides/natural-language.md @@ -280,4 +280,5 @@ While powerful, the NLP system has some limitations: ## Next Steps - [Triple Intelligence Architecture](../architecture/triple-intelligence.md) -- [API Reference](../api/README.md) \ No newline at end of file +- [API Reference](../api/README.md) +- [Getting Started Guide](./getting-started.md) \ No newline at end of file diff --git a/docs/guides/neural-api.md b/docs/guides/neural-api.md new file mode 100644 index 00000000..a703162b --- /dev/null +++ b/docs/guides/neural-api.md @@ -0,0 +1,356 @@ +# Neural API Guide + +> Semantic intelligence features for clustering, similarity, and analysis + +## Overview + +The Neural API provides advanced AI-powered features for understanding relationships and patterns in your data. Access it through `brain.neural` after initializing Brainy. + +## Quick Start + +```javascript +import { Brainy } from '@soulcraft/brainy' + +const brain = new Brainy() +await brain.init() + +// Access Neural API +const neural = brain.neural + +// Find similar items +const similarity = await neural.similar('text1', 'text2') + +// Auto-cluster your data +const clusters = await neural.clusters() +``` + +## Core Features + +### 1. Semantic Clustering + +Automatically group related items based on their meaning: + +```javascript +// Simple clustering - let Brainy decide +const clusters = await neural.clusters() + +// Each cluster contains: +// - id: Unique identifier +// - members: Array of item IDs in this cluster +// - centroid: The "center" of the cluster +// - label: Optional descriptive label +// - confidence: How confident the clustering is + +// Example: Organize customer feedback +const feedback = [ + await brain.add("The app crashes when I upload photos"), + await brain.add("Photo upload feature is broken"), + await brain.add("Great customer service!"), + await brain.add("Support team was very helpful"), + await brain.add("Pricing is too high"), + await brain.add("Too expensive for what it offers") +] + +const themes = await neural.clusters() +// Results in 3 clusters: bugs, support, pricing +``` + +#### Advanced Clustering Options + +```javascript +// Control clustering behavior +const clusters = await neural.clusters({ + algorithm: 'kmeans', // Algorithm to use + maxClusters: 5, // Maximum clusters to create + threshold: 0.7 // Minimum similarity within clusters +}) + +// Cluster specific items only +const techItems = ['id1', 'id2', 'id3', 'id4'] +const techClusters = await neural.clusters(techItems) + +// Find clusters near a specific item +const relatedClusters = await neural.clusters('central-item-id') +``` + +### 2. Similarity Calculation + +Compare any two items to see how similar they are: + +```javascript +// Compare by ID +const score = await neural.similar('item1-id', 'item2-id') +// Returns 0-1 (0 = completely different, 1 = identical) + +// Compare text directly +const score = await neural.similar( + "Machine learning is fascinating", + "AI and deep learning are interesting" +) +// Returns ~0.75 (pretty similar) + +// Compare vectors +const v1 = await brain.embed("concept 1") +const v2 = await brain.embed("concept 2") +const score = await neural.similar(v1, v2) + +// Get detailed similarity analysis +const detailed = await neural.similar('id1', 'id2', { + detailed: true +}) +// Returns: { +// score: 0.85, +// confidence: 0.92, +// explanation: "High semantic overlap in technology domain" +// } +``` + +### 3. Finding Neighbors + +Discover items similar to a given item: + +```javascript +// Find 5 most similar items +const neighbors = await neural.neighbors('item-id', 5) + +// Each neighbor has: +// - id: The neighbor's ID +// - similarity: How similar (0-1) +// - data: The actual content + +// Example: Recommend similar articles +const articleId = await brain.add("Guide to React Hooks") +const similar = await neural.neighbors(articleId, 3) + +for (const article of similar) { + console.log(`${article.similarity * 100}% similar: ${article.data}`) +} +``` + +### 4. Semantic Hierarchy + +Build a hierarchy showing relationships between items: + +```javascript +const hierarchy = await neural.hierarchy('item-id') + +// Returns structure like: +// { +// self: { id: 'item-id', type: 'article' }, +// parent: { id: 'parent-id', similarity: 0.8 }, +// siblings: [ +// { id: 'sibling1', similarity: 0.75 }, +// { id: 'sibling2', similarity: 0.72 } +// ], +// children: [ +// { id: 'child1', similarity: 0.85 } +// ] +// } + +// Use for navigation or breadcrumbs +const hier = await neural.hierarchy(currentDoc) +console.log(`You are here: ${hier.self.id}`) +if (hier.parent) { + console.log(`Parent topic: ${hier.parent.id}`) +} +``` + +### 5. Outlier Detection + +Find unusual or anomalous items in your data: + +```javascript +// Find items that don't fit patterns +const outliers = await neural.outliers(0.3) +// Returns array of IDs that are > 0.3 distance from others + +// Example: Detect spam or unusual content +const messages = [ + await brain.add("Meeting at 3pm"), + await brain.add("Lunch plans for tomorrow"), + await brain.add("BUY NOW!!! AMAZING DEALS!!!"), + await brain.add("Project deadline next week") +] + +const suspicious = await neural.outliers(0.4) +// Returns the spam message ID +``` + +### 6. Visualization Support + +Generate data for visualization libraries: + +```javascript +// Create force-directed graph data +const vizData = await neural.visualize({ + maxNodes: 100, // Limit nodes for performance + dimensions: 2, // 2D or 3D + algorithm: 'force' // Layout algorithm +}) + +// Returns: +// { +// nodes: [ +// { id: 'n1', x: 10, y: 20, cluster: 'c1' }, +// { id: 'n2', x: 30, y: 40, cluster: 'c1' } +// ], +// edges: [ +// { source: 'n1', target: 'n2', weight: 0.8 } +// ], +// clusters: [ +// { id: 'c1', color: '#ff6b6b', size: 15 } +// ] +// } + +// Use with D3.js, Cytoscape, or other viz libraries +const data = await neural.visualize({ dimensions: 3 }) +// Now feed to Three.js for 3D visualization +``` + +## Practical Examples + +### Content Recommendation System + +```javascript +// User reads an article +const currentArticle = 'article-123' + +// Find similar content +const recommendations = await neural.neighbors(currentArticle, 5) + +// Group all content into topics +const topics = await neural.clusters() + +// Find which topic this article belongs to +const currentTopic = topics.find(t => + t.members.includes(currentArticle) +) + +// Recommend from same topic first, then similar items +const sameTopicArticles = currentTopic.members + .filter(id => id !== currentArticle) + .slice(0, 3) +``` + +### Customer Feedback Analysis + +```javascript +// Add feedback with metadata +const feedbackIds = [] +for (const feedback of customerFeedback) { + const id = await brain.add(feedback.text, { + rating: feedback.rating, + date: feedback.date, + product: feedback.product + }) + feedbackIds.push(id) +} + +// Cluster to find themes +const themes = await neural.clusters(feedbackIds) + +// Analyze each theme +for (const theme of themes) { + const items = await brain.getNouns(theme.members) + + const avgRating = items.reduce((sum, item) => + sum + item.metadata.rating, 0) / items.length + + console.log(`Theme with ${theme.members.length} items`) + console.log(`Average rating: ${avgRating}`) + + // Find representative feedback for this theme + const centroidId = theme.members[0] // Closest to center + const example = await brain.getNoun(centroidId) + console.log(`Example: "${example.data}"`) +} +``` + +### Knowledge Base Organization + +```javascript +// Analyze existing knowledge base +const allDocs = await brain.getNouns({ type: 'document' }) + +// Find duplicate or highly similar content +const duplicates = [] +for (let i = 0; i < allDocs.length; i++) { + for (let j = i + 1; j < allDocs.length; j++) { + const similarity = await neural.similar( + allDocs[i].id, + allDocs[j].id + ) + if (similarity > 0.95) { + duplicates.push([allDocs[i].id, allDocs[j].id]) + } + } +} + +// Build topic hierarchy +const mainTopics = await neural.clusters({ + maxClusters: 10, + algorithm: 'hierarchical' +}) + +// For each main topic, find subtopics +for (const topic of mainTopics) { + const subtopics = await neural.clusters(topic.members) + console.log(`Topic has ${subtopics.length} subtopics`) +} +``` + +## Performance Tips + +1. **Caching**: Neural API automatically caches results. Repeated calls with same parameters are instant. + +2. **Batch Operations**: Process multiple items together rather than one at a time. + +3. **Sampling**: For large datasets, use sampling: + ```javascript + const clusters = await neural.clusters({ + algorithm: 'sample', + sampleSize: 1000 // Only analyze 1000 items + }) + ``` + +4. **Async Processing**: All neural operations are async and non-blocking. + +## Error Handling + +```javascript +try { + const similarity = await neural.similar('id1', 'id2') +} catch (error) { + // Handle errors + if (error.message.includes('not found')) { + console.log('One of the items does not exist') + } +} + +// Safe clustering with empty data +const clusters = await neural.clusters([]) +// Returns empty array, doesn't throw + +// Non-existent IDs return 0 similarity +const sim = await neural.similar('fake-id-1', 'fake-id-2') +// Returns 0 +``` + +## Advanced Configuration + +```javascript +// Configure neural behavior at initialization +const brain = new Brainy({ + neural: { + cacheSize: 1000, // Cache up to 1000 results + defaultAlgorithm: 'kmeans', + similarityMetric: 'cosine' + } +}) +``` + +## Next Steps + +- Explore [Triple Intelligence](../architecture/triple-intelligence.md) for combined vector + graph + metadata queries +- Learn about [Augmentations](../augmentations/README.md) to extend Neural API +- See [API Reference](../api/README.md) for complete method documentation \ No newline at end of file diff --git a/docs/guides/optimistic-concurrency.md b/docs/guides/optimistic-concurrency.md deleted file mode 100644 index 268bc5fa..00000000 --- a/docs/guides/optimistic-concurrency.md +++ /dev/null @@ -1,214 +0,0 @@ ---- -title: Optimistic concurrency with _rev -slug: guides/optimistic-concurrency -public: true -category: guides -template: guide -order: 8 -description: Use the per-entity `_rev` counter and `update({ ifRev })` to coordinate concurrent writes safely. Covers the lock pattern, idempotent inserts with `ifAbsent`, and recovery on conflict. -next: - - concepts/consistency-model - - guides/find-limits ---- - -# Optimistic concurrency with `_rev` - -Brainy 7.31.0 adds a per-entity revision counter so multiple writers can coordinate without a global lock or external coordinator. The pattern is the same one CouchDB, PouchDB, and ETag-based HTTP caches use: read the current revision, do your work, write back with `ifRev: `. If the revision moved, your write is rejected and you retry against the latest state. - -## What gets added - -| Surface | Behavior | -|---|---| -| `entity._rev: number` | Returned on every `get()`, `find()`, `search()`. Initialized to `1` on `add()`. Bumped by `1` on every successful `update()`. Pre-7.31.0 entities without `_rev` are surfaced as `1`. | -| `update({ id, ..., ifRev: number })` | If the persisted `_rev` does not equal `ifRev`, throws `RevisionConflictError`. Omitting `ifRev` keeps the prior (unconditional) update behavior. | -| `RevisionConflictError` | Carries `{ id, expected, actual }` for principled recovery. | -| `add({ id, ifAbsent: true })` | By-ID idempotent insert. Returns the existing `id` if one is already present; no throw, no overwrite. | -| `addMany({ items, ifAbsent: true })` | Applies `ifAbsent` to every item. Per-item `ifAbsent` overrides the batch flag. | - -`_rev` is the **per-entity** counter. Its store-wide counterpart is the generation counter behind the [Db API](../concepts/consistency-model.md): `brain.transact(ops, { ifAtGeneration })` is CAS over the whole store, `update({ ifRev })` (and `ifRev` on `transact()` update operations) is CAS over one entity. - -## The lock pattern - -Every distributed-job scheduler eventually wants this exact loop: - -```ts -import { Brainy, RevisionConflictError } from '@soulcraft/brainy' - -const LOCK_ID = '...uuid for this job slot...' - -// Bootstrap the lock document once (idempotent). -await brain.add({ - id: LOCK_ID, - type: NounType.Document, - data: { owner: null, expiresAt: 0 }, - ifAbsent: true -}) - -async function tryAcquireLock(workerId: string, ttlMs: number) { - const lock = await brain.get(LOCK_ID) - if (!lock) throw new Error('lock document missing') - - const state = lock.data as { owner: string | null; expiresAt: number } - const now = Date.now() - - // Only take the lock if it's free or expired. - if (state.owner && state.expiresAt > now) return false - - try { - await brain.update({ - id: LOCK_ID, - data: { owner: workerId, expiresAt: now + ttlMs }, - ifRev: lock._rev - }) - return true - } catch (err) { - if (err instanceof RevisionConflictError) { - // Another worker grabbed it between our read and write. - return false - } - throw err - } -} -``` - -No external lock service, no Redis SETNX, no Cloud Tasks. The CAS check is the lock. - -## Read-modify-write with retry - -The other common shape is "update a counter / config object" with bounded retries on conflict: - -```ts -async function incrementCounter(id: string, by: number) { - for (let attempt = 0; attempt < 5; attempt++) { - const entity = await brain.get(id) - if (!entity) throw new Error('counter does not exist') - const current = (entity.data as { value: number }).value - try { - await brain.update({ - id, - data: { value: current + by }, - ifRev: entity._rev - }) - return - } catch (err) { - if (err instanceof RevisionConflictError) continue // refetch + retry - throw err - } - } - throw new Error('counter update conflict after 5 attempts') -} -``` - -The retry bound matters — without one, two unlucky writers can ping-pong forever. - -## Idempotent bootstrap with `ifAbsent` - -For singletons (config rows, well-known seed entities, job-state documents) where the natural ID is deterministic: - -```ts -await brain.add({ - id: 'config:singleton', - type: NounType.Document, - data: { tenantQuota: 1000 }, - ifAbsent: true -}) -``` - -- First caller writes; gets back `'config:singleton'`. -- Every subsequent caller short-circuits at the pre-read; gets back the same `'config:singleton'` without touching the existing entity. -- `_rev` is **not** bumped on the no-op path (no write happened). - -`ifAbsent` is only meaningful when you supply an `id`. With no `id`, Brainy generates a fresh UUID that can never collide, so the flag is silently ignored. - -`addMany({ items, ifAbsent: true })` applies the flag to every item. Mixing per-item overrides with the batch flag works as you'd expect: per-item `ifAbsent: false` opts an individual row out, per-item `ifAbsent: true` opts it in. - -### Why no `addIfMissing({ match, add })` for attribute-based dedup? - -You may want "create if no entity with this email exists" (lookup by attribute, not by ID). That's a different operation: - -```ts -// What we DID NOT ship in 7.31.0 — the attribute-based variant. -await brain.addIfMissing({ // ← not a real API - match: { type: 'Person', where: { email: 'x@y.com' } }, - add: { data: '...', metadata: { email: 'x@y.com' } } -}) -``` - -It's race-prone as a plain read-then-write: two concurrent imports both see "not found," both insert, you get duplicates. Without a unique-index primitive (which Brainy doesn't have today), close the race with whole-store CAS — read at a pinned generation, then commit only if nothing moved: - -```ts -import { GenerationConflictError } from '@soulcraft/brainy' - -async function addIfMissingByEmail(email: string, data: string) { - for (let attempt = 0; attempt < 5; attempt++) { - const db = brain.now() - try { - const existing = await db.find({ - type: NounType.Person, - where: { email }, - limit: 1 - }) - if (existing.length > 0) return existing[0].id - - const committed = await brain.transact( - [{ op: 'add', type: NounType.Person, subtype: 'customer', data, metadata: { email } }], - { ifAtGeneration: db.generation } // rejects if ANYTHING committed since the read - ) - return committed.receipt!.ids[0] - } catch (err) { - if (err instanceof GenerationConflictError) continue // world moved — re-read + retry - throw err - } finally { - await db.release() - } - } - throw new Error('addIfMissingByEmail conflict after 5 attempts') -} -``` - -`ifAtGeneration` is deliberately coarse — *any* committed write invalidates it — so keep the retry bound. When you control the ID, `ifAbsent` stays the cheaper tool. - -## How `_rev` relates to generations - -Brainy 8.0 has exactly two write-coordination counters, at two granularities: - -| Counter | Scope | What it tracks | CAS surface | Conflict error | -|---|---|---|---|---| -| **`_rev`** | One entity | Per-entity write count, bumped on every successful update | `update({ ifRev })`, `{ op: 'update', ifRev }` in `transact()` | `RevisionConflictError` | -| **Generation** | Whole store | One tick per committed `transact()` batch or single-operation write | `transact(ops, { ifAtGeneration })` | `GenerationConflictError` | - -They compose: a `transact()` batch can carry per-entity `ifRev` checks *and* a whole-store `ifAtGeneration`; any failed check rejects the entire batch before anything is staged. Generations also power snapshots and time travel (`brain.now()`, `brain.asOf()`, `db.persist()`) — see the [consistency model](../concepts/consistency-model.md) and [Snapshots & Time Travel](./snapshots-and-time-travel.md). - -A snapshot or historical view captures each entity *including* its `_rev` at that moment, so reading the past and writing back with `ifRev` against the live state works exactly as you'd hope: the write fails if the entity moved since the state you copied from. - -## The transact envelope: batch size, budget, and bulk imports - -`transact()` applies its batch atomically under one commit — which means the whole batch -shares one **apply budget**. Since 8.7.0 the budget scales with the batch: -`max(30 s, opCount × 2 s)`, or exactly what you pass as `timeoutMs`. A tripped budget rolls -the entire batch back (nothing partial survives) and throws a retryable -`TransactionTimeoutError` that names the operation it stopped at, the batch size, and the -elapsed vs budgeted time — a diagnosis, not just a failure: - -``` -Transaction timed out at operation 41/120 ('add') — 246012ms elapsed, budget 240000ms. -The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch. -``` - -Practical envelope guidance for bulk work: - -1. **Precompute embeddings outside the commit path.** Embedding inside `transact()` spends - the budget on model inference. Use `brain.embedBatch(texts)` and pass each vector via - the op's `vector` field — the commit then pays only storage costs, and a retried batch - never re-pays inference. (The win is *where* the inference happens, not raw embedding - throughput: on the default WASM engine, batch and sequential embedding measure - comparably, ~160 ms/text; native embedding providers may batch faster.) -2. **Chunk very large imports** into batches of a few hundred ops with one `transact()` - each. You lose whole-import atomicity but keep per-chunk atomicity, bounded memory, and - resumability — pair with `ifAbsent` upserts so a retried chunk is idempotent. -3. **Slow disks change the math, not the contract.** On network-attached storage a single - op can cost ~2 s (canonical write + fsync + index maintenance). The scaled default - absorbs that; pass an explicit `timeoutMs` only when you know better than the scale. -4. **`addMany`/`relateMany` are the convenience tier** — they chunk and batch-embed for - you, with per-item error reporting instead of batch atomicity. Choose by what you need: - atomic-all-or-nothing → `transact()`; resilient bulk load → `addMany`. diff --git a/docs/guides/quick-start.md b/docs/guides/quick-start.md deleted file mode 100644 index 097c55fe..00000000 --- a/docs/guides/quick-start.md +++ /dev/null @@ -1,111 +0,0 @@ ---- -title: Quick Start -slug: getting-started/quick-start -public: true -category: getting-started -template: guide -order: 2 -description: Build your first knowledge graph in 60 seconds. Add entities, create relationships, and query with Triple Intelligence — vector + graph + metadata in one call. -next: - - concepts/triple-intelligence - - api/reference ---- - -# Quick Start - -Get Brainy running in under a minute. - -## 1. Install - -```bash -npm install @soulcraft/brainy -``` - -## 2. Initialize - -```typescript -import { Brainy, NounType, VerbType } from '@soulcraft/brainy' - -const brain = new Brainy() -await brain.init() -``` - -That's it. Brainy auto-configures storage, loads the embedding model, and builds the indexes. - -## 3. Add Knowledge - -```typescript -// Text is automatically embedded into 384-dim vectors -const reactId: string = await brain.add({ - data: 'React is a JavaScript library for building user interfaces', - type: NounType.Concept, - subtype: 'library', // Sub-classification within Concept - metadata: { category: 'frontend', year: 2013 } -}) - -const nextId: string = await brain.add({ - data: 'Next.js framework for React with server-side rendering', - type: NounType.Concept, - subtype: 'framework', - metadata: { category: 'framework', year: 2016 } -}) -``` - -`type` is one of Brainy's 42 stable NounTypes. `subtype` is your free-form sub-classification within that type — flat string, no hierarchy, indexed on the fast path. See **[Subtypes & Facets](./subtypes-and-facets.md)** for the full guide. - -## 4. Create Relationships - -```typescript -// Typed graph relationships -await brain.relate({ - from: nextId, - to: reactId, - type: VerbType.DependsOn -}) -``` - -## 5. Query with Triple Intelligence - -```typescript -import type { Result } from '@soulcraft/brainy' - -// All three search paradigms in one call -const results: Result[] = await brain.find({ - query: 'modern frontend frameworks', // Vector similarity search - where: { year: { greaterThan: 2015 } }, // Metadata filtering - connected: { to: reactId, depth: 2 } // Graph traversal -}) - -console.log(results[0].data) // 'Next.js framework for React...' -console.log(results[0].score) // 0.94 -``` - -## What Just Happened - -Every entity you `add()` lives in three indexes simultaneously: - -| Index | What it stores | Query with | -|-------|---------------|------------| -| Vector | 384-dim embedding of `data` | `find({ query: '...' })` | -| Metadata | All `metadata` fields | `find({ where: { ... } })` | -| Graph | Typed relationships from `relate()` | `find({ connected: { ... } })` | - -`find()` queries all three in parallel and fuses the results. - -## Natural Language Queries - -Brainy understands 220+ natural language patterns: - -```typescript -// These all work without any configuration -await brain.find({ query: 'recent documents about machine learning' }) -await brain.find({ query: 'articles created this week' }) -await brain.find({ query: 'people who work at Anthropic' }) -``` - -## Next Steps - -- [Triple Intelligence](/docs/concepts/triple-intelligence) — understand how the query engine works -- [The Find System](/docs/guides/find-system) — advanced queries, operators, and graph traversal -- [API Reference](/docs/api/reference) — complete method documentation -- [Storage Adapters](/docs/guides/storage-adapters) — filesystem, memory diff --git a/docs/guides/reacting-to-changes.md b/docs/guides/reacting-to-changes.md deleted file mode 100644 index 7367a5e3..00000000 --- a/docs/guides/reacting-to-changes.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -title: Reacting to Changes -slug: guides/reacting-to-changes -public: true -category: guides -template: guide -order: 11 -description: Subscribe to every committed mutation with `brain.onChange` — the in-process change feed behind live UIs, cache invalidation, and realtime sync. Covers the event shape, delivery guarantees, and catch-up patterns. -next: - - guides/optimistic-concurrency - - guides/snapshots-and-time-travel ---- - -# Reacting to Changes - -`brain.onChange(cb)` is Brainy's in-process change feed: subscribe once and -receive one event per committed mutation — **every** mutation, regardless of -how it happened. Direct calls, batch methods, `transact()`, imports, and -Virtual Filesystem writes all funnel through the same commit point the feed is -emitted from, so nothing slips past it. - -```ts -const off = brain.onChange((e) => { - if (e.kind === 'entity') { - console.log(`${e.op} ${e.entity?.type} ${e.id} @ generation ${e.generation}`) - } -}) - -await brain.add({ data: 'Ada Lovelace', type: 'person' }) -// → "add person 0198... @ generation 42" - -off() // unsubscribe when done -``` - -## The event - -```ts -interface BrainyChangeEvent { - kind: 'entity' | 'relation' | 'store' - op: 'add' | 'update' | 'remove' | 'relate' | 'unrelate' | 'updateRelation' - | 'clear' | 'restore' - id?: string - entity?: { id: string; type: string; subtype?: string; - metadata: Record; service?: string } - relation?: { id: string; from: string; to: string; type: string; - metadata?: Record } - generation?: number - timestamp: number -} -``` - -- **Entity events** (`add` / `update` / `remove`) carry the post-commit indexed - view — `type`, `subtype`, and the full custom `metadata`, so you can match - your own `where`-style filters against events without a read. -- **Deletes are fully described.** A `remove` or `unrelate` event carries the - record's *last committed state* (sourced from the commit's own history - record), not just an id. -- **Batches emit per item.** `addMany` / `updateMany` / `relateMany` / - `removeMany` emit one event per affected record; a `transact()` batch emits - one event per item, all sharing the batch's single `generation`. -- **Cascades are visible.** Removing an entity also emits `unrelate` for each - relationship the delete cascaded to. -- **Store-level events** (`kind: 'store'`) fire for the two wholesale - operations — `clear()` and `restore()` — and mean *"everything may have - changed; refetch what you care about."* - -## Delivery guarantees - -- **Post-commit only.** An aborted write — a losing - [`ifRev` compare-and-swap](optimistic-concurrency.md), a rejected - transaction — never emits. If you received the event, the write is durable. -- **Commit-ordered.** Events arrive in the order writes committed; - `generation` is monotonic. -- **Asynchronous, never blocking.** Delivery happens in a microtask after the - write completes. A slow listener cannot delay a write; a throwing listener - is logged and isolated from other listeners. -- **Zero overhead when unused.** With no subscribers, the write path does no - event work at all. -- **Fire-and-forget.** There is no replay or backpressure. For catch-up after - a disconnect, use the `generation` on each event together with - [`asOf()` / the transaction log](snapshots-and-time-travel.md): record the - last generation you processed, and on reconnect diff from there. For file - content specifically, `vfs.readFile(path, { asOf })` and - `vfs.history(path)` are the temporal read — see - [Snapshots & Time Travel](snapshots-and-time-travel.md). - -## Patterns - -**Cache invalidation** — drop cached reads for whatever changed: - -```ts -brain.onChange((e) => { - if (e.kind === 'store') return cache.clear() - if (e.id) cache.delete(e.id) -}) -``` - -**Live queries (notify-and-refetch)** — re-run a query when a relevant change -lands, rather than diffing incrementally: - -```ts -brain.onChange((e) => { - if (e.kind === 'entity' && e.entity?.type === 'order') { - refreshOpenOrdersView() // debounce as needed - } -}) -``` - -**Forwarding to other processes** — the feed is in-process by design. To push -changes to browsers or other services, forward events through your own -transport (WebSocket, SSE) from the process that owns the brain. - -## Lifecycle - -`onChange` returns an unsubscribe function — call it when tearing down a -subscriber (for example, when evicting a pooled instance). `brain.close()` -drops all listeners; no events are delivered for or after `close()`. diff --git a/docs/guides/schema-migrations.md b/docs/guides/schema-migrations.md deleted file mode 100644 index 007ecd8a..00000000 --- a/docs/guides/schema-migrations.md +++ /dev/null @@ -1,284 +0,0 @@ -# Schema Migrations - -Brainy includes a built-in migration system for transforming entity and verb metadata across storage versions. Migrations are pure functions that run once per storage instance, with optional snapshot backup (`backupTo`), resume support, and error tracking. - ---- - -## Quick Start - -### 1. Define a migration - -Add your migration to the `MIGRATIONS` array in `src/migration/migrations.ts`: - -```typescript -import type { Migration } from './types.js' - -export const MIGRATIONS: Migration[] = [ - { - id: '7.17.0-rename-status', - version: '7.17.0', - description: 'Rename "state" field to "status"', - applies: 'nouns', - transform: (m) => { - if ('state' in m) { - const { state, ...rest } = m - return { ...rest, status: state } - } - return null // already migrated or not applicable - } - } -] -``` - -### 2. Ship the new version - -That's it. Brainy detects pending migrations on `init()` and either runs them automatically or warns the user to call `brain.migrate()`. - ---- - -## How It Works - -When `brain.init()` runs: - -1. **Detection** — reads migration state from storage (one key lookup). Compares completed migration IDs against the `MIGRATIONS` array. If nothing pending, cost is ~0ms. - -2. **Small datasets** (`autoMigrate: true`, <10K entities) — migrates inline during `init()`. - -3. **Large datasets or manual mode** — logs a warning. User calls `brain.migrate()` when ready. - -When `brain.migrate()` runs: - -1. **Backup (optional)** — with `backupTo`, a hard-link snapshot of the current generation is persisted before any transform runs. Rollback is `brain.restore(backupPath, { confirm: true })`. - -2. **Transform** — iterates all nouns/verbs in paginated batches. For each entity, calls the `transform` function. If it returns a new object, saves it. If it returns `null`, skips. Vectors are never touched. - -3. **Save state** — records each completed migration ID so it never re-runs. - -4. **Rebuild indexes** — if any entities were modified, rebuilds the MetadataIndex. - ---- - -## Writing Migrations - -### The Migration interface - -```typescript -interface Migration { - id: string // Unique ID, e.g. "7.17.0-rename-field" - version: string // Version that introduced this migration - description: string // Human-readable description - applies: 'nouns' | 'verbs' | 'both' - transform: (metadata: Record) => Record | null -} -``` - -### Transform rules - -- **Return a new object** to modify the entity's metadata. -- **Return `null`** to skip (no change needed). -- **Must be idempotent** — running the same transform twice on the same data should produce the same result (or return `null` the second time). This is required because interrupted runs resume and re-encounter already-migrated entities. -- **Must be pure** — no side effects, no async, no external state. -- Transforms only see metadata. Vectors, embeddings, and the `data` field stored inside metadata are available as properties on the metadata object. - -### Ordering - -Migrations run in array order. Add new migrations at the end of the `MIGRATIONS` array. Each migration runs independently per entity — migration 2 sees the output of migration 1. - -### Validation - -`MigrationRunner.validateMigrations()` checks migration definitions and will throw on: - -- Duplicate IDs -- Invalid `applies` values (must be `'nouns'`, `'verbs'`, or `'both'`) -- Non-function `transform` -- Missing or empty `id`, `version`, or `description` - ---- - -## API Reference - -### `brain.migrate(options?)` - -```typescript -// Dry-run: preview what would change without writing -const preview = await brain.migrate({ dryRun: true }) -// preview.pendingMigrations — array of { id, description } -// preview.affectedEntities — count of entities that would change -// preview.totalEntities — count of entities scanned -// preview.sampleChanges — up to 5 before/after samples -// preview.estimatedTime — rough time estimate string - -// Apply migrations (optionally with a pre-migration snapshot) -const result = await brain.migrate({ backupTo: '/backups/pre-migration' }) -// result.backupPath — snapshot path, or null when no backupTo was supplied -// result.migrationsApplied — array of migration IDs that ran -// result.entitiesProcessed — total entities scanned -// result.entitiesModified — entities actually changed -// result.errors — array of entity-level errors (non-fatal) -``` - -### Options - -```typescript -interface MigrateOptions { - dryRun?: boolean // Preview without writing (default: false) - maxErrors?: number // Bail out after N entity errors (default: 100) - onProgress?: (progress: { - migrationId: string - processed: number - modified: number - hasMore: boolean - }) => void -} -``` - ---- - -## Error Handling - -If a transform function throws on a specific entity, the error is recorded and migration continues to the next entity. The failed entity's metadata is left unchanged. - -```typescript -const result = await brain.migrate() - -if (result.errors.length > 0) { - for (const err of result.errors) { - console.warn(`Entity ${err.entityId} failed in ${err.migrationId}: ${err.error}`) - } -} -``` - -If errors exceed `maxErrors` (default: 100), the migration stops early and returns partial results. Successfully migrated entities keep their changes; failed entities are unchanged. - -```typescript -// Strict mode: fail fast on any error -const result = await brain.migrate({ maxErrors: 1 }) - -// Lenient mode: tolerate many errors -const result = await brain.migrate({ maxErrors: 10000 }) -``` - ---- - -## Backup and Rollback - -Pass `backupTo` and `brain.migrate()` persists a snapshot of the current generation **before any transform runs**. On filesystem storage the snapshot is a hard-link farm — created without copying entity data, and immune to later writes (see [Snapshots & Time Travel](./snapshots-and-time-travel.md)): - -```typescript -const result = await brain.migrate({ backupTo: '/backups/pre-migration-8.0' }) -console.log(result.backupPath) // '/backups/pre-migration-8.0' (null when no backupTo) -``` - -To roll back, restore the snapshot wholesale: - -```typescript -await brain.restore('/backups/pre-migration-8.0', { confirm: true }) -``` - -Without `backupTo`, no backup is taken — transforms are idempotent (they return `null` when already applied), but a pre-migration snapshot is the cheap insurance for anything destructive. - ---- - -## Progress Tracking - -For large datasets, use the `onProgress` callback: - -```typescript -await brain.migrate({ - onProgress: ({ migrationId, processed, modified, hasMore }) => { - console.log(`[${migrationId}] ${processed} scanned, ${modified} modified${hasMore ? '...' : ' (done)'}`) - } -}) -``` - -Progress is reported after each batch (batch size is determined by the storage adapter). - ---- - -## Examples - -### Rename a field - -```typescript -{ - id: '7.17.0-rename-state-to-status', - version: '7.17.0', - description: 'Rename metadata.state to metadata.status', - applies: 'nouns', - transform: (m) => { - if ('state' in m) { - const { state, ...rest } = m - return { ...rest, status: state } - } - return null - } -} -``` - -### Add a default value - -```typescript -{ - id: '7.18.0-add-priority-default', - version: '7.18.0', - description: 'Add priority field with default "normal"', - applies: 'both', - transform: (m) => { - if (!('priority' in m)) { - return { ...m, priority: 'normal' } - } - return null - } -} -``` - -### Remove a deprecated field - -```typescript -{ - id: '7.19.0-remove-legacy-flag', - version: '7.19.0', - description: 'Remove deprecated "legacy" field', - applies: 'nouns', - transform: (m) => { - if ('legacy' in m) { - const { legacy, ...rest } = m - return rest - } - return null - } -} -``` - -### Transform verb metadata - -```typescript -{ - id: '7.20.0-normalize-verb-weights', - version: '7.20.0', - description: 'Normalize verb weights from 0-100 to 0-1 scale', - applies: 'verbs', - transform: (m) => { - if (typeof m.weight === 'number' && m.weight > 1) { - return { ...m, weight: m.weight / 100 } - } - return null - } -} -``` - ---- - -## Storage Backend Compatibility - -Migrations work identically across all storage backends (Memory, FileSystem). The system uses `BaseStorage` methods (`getNouns`, `saveNounMetadata`, `getVerbs`, `saveVerbMetadata`) which are implemented by every adapter. - -Batch size and rate limiting are automatically configured per adapter — no tuning required. - ---- - -## What Migrations Don't Do - -- **Re-embedding** — migrations transform metadata only. If you change your embedding model or dimensions, that requires re-vectorizing data, which is a separate concern (not part of this system). -- **Vector modification** — the `vectors.json` files are never touched by migrations. -- **Schema enforcement** — migrations are opt-in transforms, not schema validators. Brainy's metadata is schemaless by design. diff --git a/docs/guides/snapshots-and-time-travel.md b/docs/guides/snapshots-and-time-travel.md deleted file mode 100644 index 490aecab..00000000 --- a/docs/guides/snapshots-and-time-travel.md +++ /dev/null @@ -1,449 +0,0 @@ ---- -title: Snapshots & Time Travel -slug: guides/snapshots-and-time-travel -public: true -category: guides -template: guide -order: 9 -description: Recipes for the Db API — instant backups with persist(), restore, time-travel debugging with asOf(), range queries over history (diff, history, since, log windows), persist-before-migrate, what-if analysis with with(), and audit trails via transaction metadata. -next: - - concepts/consistency-model - - guides/optimistic-concurrency - - guides/external-backups ---- - -# Snapshots & Time Travel - -Brainy 8.0 treats the database as a **value**: `brain.now()` pins the -current state as an immutable `Db`, `brain.transact()` commits an atomic -batch and hands you the resulting value, `brain.asOf()` opens past state, -and `db.persist()` cuts a self-contained snapshot. This guide is the recipe -book. The precise guarantees behind every recipe live in the -[consistency model](../concepts/consistency-model.md). - -## Instant backup - -Pin the current state, persist it, release: - -```typescript -const db = brain.now() -try { - await db.persist('/backups/2026-06-11') -} finally { - await db.release() -} -``` - -On filesystem storage the snapshot is built from **hard links**: every data -file in Brainy is immutable-by-rename, so the snapshot is created without -copying entity data and shares disk space with the live store. Later writes -can never alter it — a rewrite swaps the inode, the snapshot keeps the old -bytes. Cross-device targets fall back to per-file byte copies, and -persisting an in-memory brain serializes it to the same directory layout — -a real, durable store. - -> Archiving a brain directory with **external tools** (`tar`, `rsync`, `cp`)? -> Some index files are sparse and can explode to their apparent size under a -> naive copy — see [External Backups & Sparse Storage](/docs/guides/external-backups). - -Two things to know: - -- `persist()` requires the view to still be the store's **latest** - generation. If something committed after your pin, it throws - `GenerationConflictError` instead of snapshotting the wrong state — pin - and persist before further writes, or retry with a fresh `brain.now()`. -- The target directory must be empty or absent. - -For scheduled backups, this loop is the whole job: - -```typescript -const db = brain.now() -try { - await db.persist(`/backups/${new Date().toISOString().slice(0, 10)}`) -} finally { - await db.release() -} -``` - -## Restore - -`restore()` replaces the store's **entire** current state from a snapshot — -entities, relationships, indexes, history. It is deliberately loud about it: - -```typescript -await brain.restore('/backups/2026-06-11', { confirm: true }) -``` - -- `{ confirm: true }` is mandatory — current state is destroyed. -- The snapshot is copied in (never linked), so it stays independent and can - be restored again later. -- All indexes are rebuilt from the restored records. -- The generation counter is floored at its pre-restore value, so generation - numbers you observed before the restore are never reissued. -- Live `Db` pins do not survive a restore — release them first. - -## Open a snapshot read-only - -You do not have to restore to look inside a snapshot. `Brainy.load()` opens -it as a self-contained read-only store with the **full query surface**, -including vector search: - -```typescript -const db = await Brainy.load('/backups/2026-06-11') - -const hits = await db.find({ query: 'unpaid invoices from the spring campaign' }) -const orders = await db.find({ type: NounType.Document, subtype: 'order' }) - -await db.release() // closes the underlying read-only instance -``` - -`brain.asOf('/backups/2026-06-11')` does the same from an existing brain. -This is also the 8.0 answer to "named branches": a branch is a name → path -mapping your application keeps, where each path is a persisted snapshot. -Need a writable copy? Restore the snapshot into a fresh data directory and -open a writer on it — instead of switching a shared store between branches -in place, every line of code always sees exactly the store it opened. - -## Time-travel debugging - -When production data looks wrong, query the past directly — by wall-clock -time or by generation: - -```typescript -// What did this order look like yesterday? -const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000)) -const before = await yesterday.get(orderId) - -// Full queries work at any reachable generation — search, graph, filters: -const thenActive = await yesterday.find({ - type: NounType.Document, - subtype: 'order', - where: { status: 'active' } -}) - -await yesterday.release() -``` - -Pin two points in time and diff them: - -```typescript -const before = await brain.asOf(1041) -const after = brain.now() - -const changed = await after.since(before) -changed.nouns // entity ids touched by transactions in between -changed.verbs // relationship ids touched in between - -await before.release() -await after.release() -``` - -Three things to remember: - -- History granularity is per-write: EVERY write — `transact()` AND a - single-operation `add`/`update`/`remove`/`relate` — is its own immutable - generation, so a pin always freezes against later writes and every write is - individually addressable via `asOf()` (see the - [consistency model](../concepts/consistency-model.md)). Use `transact()` when - you want several operations to share ONE atomic generation. -- The first index-accelerated query (semantic search, traversal, cursors, - aggregation) at a historical generation builds an in-memory index - materialization — O(n at that generation), once per `Db`, freed on - `release()`. Metadata-level reads are free. -- Generations reclaimed by `compactHistory()` throw - `GenerationCompactedError` — persist anything you need to keep forever. - -## Range queries over history - -`asOf()` answers "what was the state AT a point". Four range verbs answer -"what happened BETWEEN two points" and "what is one entity's whole history". -They all build on the same generation records — no extra bookkeeping. - -### `diff(a, b)` — what changed, classified - -`since()` gives you the raw set of *touched* ids. `diff()` goes further: it -resolves each touched id at both endpoints and classifies it as **added**, -**removed**, or **modified** — split by entities (`nouns`) and relationships -(`verbs`). An id that was touched but ended up identical (changed then -reverted, or created and deleted within the interval) lands in **none** of the -buckets. Endpoints are a generation, a `Date`, or a `Db`, in either order: - -```typescript -const d = await brain.diff(1041, brain.generation()) - -d.added.nouns // entity ids created between the two states -d.removed.nouns // entity ids deleted -d.modified.nouns // entity ids whose stored value actually changed -d.added.verbs // …relationships, the same three ways -``` - -Orientation is `a → b`: `added` means "exists at `b`, not at `a`". The -comparison behind `modified` is key-order-insensitive, so a no-op re-write of -the same fields never shows up as a change. - -### `history(id, range?)` — one entity, every version - -`asOf()` is per-*generation*; `history()` is per-*entity*. It returns every -distinct version of one id over a range, oldest first — each `value` is the -materialized state at that version (and `null` marks a removal): - -```typescript -const h = await brain.history(invoiceId) - -for (const v of h.versions) { - console.log(v.generation, v.value?.metadata?.status ?? '(deleted)') -} -// 1041 'draft' -// 1043 'approved' -// 1050 'paid' -``` - -Every version ties to the trusted `asOf()` path — `v.value` equals -`(await brain.asOf(v.generation)).get(id)`. Pass `{ from, to }` (generation or -`Date`) to bound the range; a `from` below the compaction horizon is quietly -truncated to it rather than throwing (history is best-effort over surviving -records). - -### `since()` and `transactionLog()` take ranges too - -`since()` accepts a `Db`, a generation number, or a `Date` — all equivalent, -all an **exclusive** lower bound (`db.since(prior)` equals -`db.since(prior.generation)`): - -```typescript -await brain.now().since(1041) // ids changed after generation 1041 -await brain.now().since(new Date(Date.now() - 3_600_000)) // …in the last hour -``` - -`transactionLog({ from, to })` windows the commit log **inclusively** on both -ends (a log window names the commits it spans — the deliberate contrast to -`since`'s exclusive lower bound); `limit` applies after the window, newest -first: - -```typescript -const window = await brain.transactionLog({ from: 1041, to: 1050 }) // commits 1041…1050 -const recent = await brain.transactionLog({ from: lastHour, limit: 20 }) -``` - -### Composing them - -"Which orders changed in this window?" is `diff` ids intersected with an -`asOf` query — the two agree by construction: - -```typescript -const changed = await brain.diff(g1, g2) -const atG2 = await brain.asOf(g2) -const changedOrders = (await atG2.find({ type: NounType.Document, subtype: 'order' })) - .map(r => r.id) - .filter(id => changed.added.nouns.includes(id) || changed.modified.nouns.includes(id)) -await atG2.release() -``` - -One contrast to keep straight: `diff` and `since` **throw** -`GenerationCompactedError` for a bound below the horizon, while `history` -**truncates** to the horizon — diffs must be exact, history is best-effort. - -## Safe schema migration - -`brain.migrate()` integrates with snapshots directly: pass `backupTo` and a -hard-link snapshot of the current generation is persisted **before any -transform runs**: - -```typescript -const result = await brain.migrate({ backupTo: '/backups/pre-migration-8.0' }) -console.log(result.migrationsApplied, result.backupPath) - -// If the migration went wrong, roll the whole store back: -await brain.restore('/backups/pre-migration-8.0', { confirm: true }) -``` - -The same persist-before-mutate pattern works for any risky bulk operation, -not just migrations: - -```typescript -const pin = brain.now() -try { - await pin.persist('/backups/pre-bulk-edit') -} finally { - await pin.release() -} -await runRiskyBulkEdit(brain) -``` - -## What-if analysis - -`db.with(ops)` applies a transaction **speculatively, in memory** — nothing -touches disk, the generation counter, or the indexes. Ask "what would the -store look like if…", then commit the same operations for real: - -```typescript -const ops = [ - { op: 'update', id: employeeId, metadata: { team: 'platform' } }, - { op: 'relate', from: employeeId, to: milestoneId, type: VerbType.ParticipatesIn, subtype: 'assignment' } -] - -const base = brain.now() -const whatIf = await base.with(ops) - -await whatIf.get(employeeId) // sees the change -await whatIf.find({ where: { team: 'platform' } }) // metadata finds work -await whatIf.related(employeeId) // overlay relations included - -await whatIf.release() -await base.release() - -// Looks right — make it real, atomically: -await brain.transact(ops) -``` - -**The boundary:** speculative entities carry no embeddings (`with()` never -invokes the embedder), so semantic search, traversal, cursors, aggregation, -and `persist()` throw `SpeculativeOverlayError` on overlay views instead of -returning silently incomplete results. `get()`, metadata-filter `find()`, -and filter-based `related()` are fully supported. Overlays chain — calling -`with()` on an overlay stacks another layer. - -## Audit trails - -`transact()` reifies transaction metadata: whatever you pass as `meta` is -recorded durably alongside the committed generation and timestamp, readable -via `brain.transactionLog()`: - -```typescript -await brain.transact( - [{ op: 'update', id: invoiceId, metadata: { status: 'approved' } }], - { meta: { author: 'approvals-service', actor: 'jane@example.com', reason: 'PO-7741' } } -) - -const log = await brain.transactionLog({ limit: 20 }) // newest first -// [{ generation: 1042, timestamp: 1765432100000, meta: { author: 'approvals-service', ... } }] -``` - -Combine the log with `asOf()` to reconstruct exactly what any transaction -did: - -```typescript -const [entry] = await brain.transactionLog({ limit: 1 }) - -const after = await brain.asOf(entry.generation) -const before = await brain.asOf(entry.generation - 1) - -const touched = await after.since(before) -for (const id of touched.nouns) { - console.log(id, await before.get(id), '→', await after.get(id)) -} - -await before.release() -await after.release() -``` - -For per-entity write coordination (rather than whole-store history), the -`_rev` counter and `ifRev` CAS remain the right tool — see -[optimistic concurrency](./optimistic-concurrency.md). - -## Keeping history bounded - -Under Model-B every write is a generation, so history can grow quickly — -Brainy auto-compacts at `close()` (time-bounded per pass) under the -**`retention`** knob (configured on the constructor). Since 8.9.0, `flush()` -never compacts: flushing is durability work and costs only what the current -window's writes cost, regardless of history backlog. A long-lived writer that -never closes keeps its history until its next explicit `compactHistory()` — -schedule one in your maintenance window if you run bounded retention: - -```typescript -// Zero-config: ADAPTIVE — keep as much history as free disk/RAM allows, -// reclaiming oldest-first under pressure. (This is the default.) -new Brainy({ /* retention unset */ }) - -// Unbounded — never reclaim history (opt in explicitly): -new Brainy({ retention: 'all' }) - -// Explicit CAPS — reclaim oldest-unpinned generations while ANY cap is exceeded: -new Brainy({ retention: { maxGenerations: 1000, maxAge: 7 * 86_400_000, maxBytes: 512 * 1024 ** 2 } }) -``` - -Reclaim manually at any time (the same caps, plus an optional per-pass time -budget for maintenance windows — an early stop is a consistent prefix and the -next pass resumes): - -```typescript -await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 24 * 60 * 60 * 1000 }) -await brain.compactHistory({ maxBytes: 512 * 1024 ** 2, timeBudgetMs: 10_000 }) -``` - -Compaction never breaks a pinned read — record-sets are reclaimed only when -no live `Db` could need them (live pins are ALWAYS exempt). Release views you -are done with (including the ones `transact()` returns), and `persist()` any -generation you want to keep beyond the retention window: snapshots are -self-contained and unaffected by compaction. - -## Time travel for files (the VFS) - -Since 8.2.0, time travel covers Virtual Filesystem **content**, not just -entity records. File bytes are retention-protected: a content blob referenced -by any generation inside the retention window is never reclaimed, so reading -the past always returns the exact bytes — never a stale field or a -dangling hash. - -**`vfs.readFile(path, { asOf })`** takes a generation number or a `Date` and -returns the file's exact bytes as they stood then. It resolves the path's -current entity, then materializes its state at the target generation — so it -answers *"what did the file at this path hold at that point?"* It bypasses -the content cache; the `encoding` option still applies. Asking about a -generation before the file existed throws the usual not-found error, and -asking past the retention window's compaction horizon throws a -compacted-generation error. - -**`vfs.history(path)`** returns the file's versions inside the retention -window, oldest first — one `FileVersion` per generation that wrote the file, -the newest entry being the current state: - -```typescript -// A CMS page evolves… -await brain.vfs.writeFile('/pages/home.json', '{"title":"Launch"}') -await brain.vfs.writeFile('/pages/home.json', '{"title":"Launch v2"}') -await brain.vfs.writeFile('/pages/home.json', '{"title":""}') // bad deploy! - -// Every version is listed and readable: -const versions = await brain.vfs.history('/pages/home.json') -// → [{ generation, timestamp, hash, size, mimeType? }, …] ascending - -const good = versions[versions.length - 2] -const bytes = await brain.vfs.readFile('/pages/home.json', { - asOf: good.generation -}) - -// Restore = write the old bytes back. This is a NEW write (a new -// generation) — history is never rewritten, so the bad version stays -// visible in the audit trail. -await brain.vfs.writeFile('/pages/home.json', bytes) -``` - -Two lifecycle consequences worth stating plainly: - -- **Deleting or overwriting a file no longer frees its bytes immediately.** - Old content lives until history compaction reclaims the generations that - reference it — the same `retention` budget that bounds all Model-B history - (and pinned views are exempt, exactly as above). Size your `retention` for - the file-version depth you want; `retention: 'all'` keeps every version of - every file forever. -- **After `compactHistory()` reclaims a generation, its file versions are - gone** and their bytes are physically reclaimed. (This also fixed a - pre-8.2.0 defect where overwritten content was never reclaimed at all — an - unbounded silent leak.) - -## From branches to values - -If you used the pre-8.0 `fork`/`checkout`/`commit`/`versions` surface, every -use case maps to a sharper tool: - -| Pre-8.0 habit | 8.0 recipe | -|---|---| -| `fork()` to experiment safely | `db.with(ops)` for speculation in memory; a restored snapshot in a fresh directory for a long-lived writable copy | -| `commit()` checkpoints | `transact(ops, { meta })` — every batch is an atomic, logged, time-travelable commit | -| `checkout()` to switch branches | Open the snapshot you want — `Brainy.load(path)` read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store. | -| `getHistory()` | `brain.transactionLog()` + `db.since(priorDb)` | -| `versions.save()` per-entity snapshots | A pinned `Db` or persisted snapshot captures *every* entity at that moment; `asOf()` reads any entity's past state | -| `versions.restore()` | `brain.restore(snapshot, { confirm: true })` for the whole store, or read the old entity via `asOf()` and write it back with `transact()` | -| Backup branches | `db.persist(path)` — instant, hard-link-shared, self-contained | diff --git a/docs/guides/standard-import-progress.md b/docs/guides/standard-import-progress.md deleted file mode 100644 index 9f2e2e5b..00000000 --- a/docs/guides/standard-import-progress.md +++ /dev/null @@ -1,453 +0,0 @@ -# Standard Import Progress API - -## ✅ Build Once, Works for ALL Formats - -**Brainy provides a 100% standardized progress API** - write your UI/tool once, and it works for all 7 supported formats (CSV, PDF, Excel, JSON, Markdown, YAML, DOCX) with **zero format-specific code**. - ---- - -## 🎯 The Standard Interface - -### One Interface for Everything - -```typescript -import { Brainy } from '@soulcraft/brainy' - -const brain = await Brainy.create() - -// THIS CODE WORKS FOR ALL 7 FORMATS - NO FORMAT-SPECIFIC LOGIC NEEDED! -await brain.import(anyBuffer, { - onProgress: (progress) => { - // Standard fields - ALWAYS available regardless of format - console.log(progress.stage) // Current stage - console.log(progress.message) // Human-readable status - - // Optional fields - available when relevant - console.log(progress.processed) // Items processed so far - console.log(progress.total) // Total items (if known) - console.log(progress.entities) // Entities extracted - console.log(progress.relationships) // Relationships inferred - console.log(progress.throughput) // Items/sec (during extraction) - console.log(progress.eta) // Time remaining in ms - } -}) -``` - -### The Complete Interface - -```typescript -interface ImportProgress { - // === ALWAYS PRESENT === - - /** High-level stage (5 stages for all formats) */ - stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete' - - /** Human-readable status message */ - message: string - - // === AVAILABLE WHEN RELEVANT === - - /** Items processed (rows, pages, nodes, etc.) */ - processed?: number - - /** Total items to process (if known ahead of time) */ - total?: number - - /** Entities extracted so far */ - entities?: number - - /** Relationships inferred so far */ - relationships?: number - - /** Processing rate (items per second) */ - throughput?: number - - /** Estimated time remaining (milliseconds) */ - eta?: number - - /** Whether data is queryable at this point */ - queryable?: boolean -} -``` - ---- - -## 🎨 Generic UI Components - -### React Progress Component (Works for ALL Formats) - -```typescript -import { useState } from 'react' -import { Brainy } from '@soulcraft/brainy' - -function UniversalImportProgress({ file }: { file: File }) { - const [progress, setProgress] = useState({ - stage: 'idle', - message: 'Ready to import', - percent: 0, - entities: 0, - relationships: 0 - }) - - const handleImport = async () => { - const buffer = await file.arrayBuffer() - const brain = await Brainy.create() - - await brain.import(Buffer.from(buffer), { - // THIS WORKS FOR CSV, PDF, EXCEL, JSON, MARKDOWN, YAML, DOCX! - onProgress: (p) => { - setProgress({ - stage: p.stage, - message: p.message, - - // Calculate percentage from stage + processed/total - percent: calculatePercent(p), - - entities: p.entities || 0, - relationships: p.relationships || 0 - }) - } - }) - } - - // Helper: Calculate percentage from progress - function calculatePercent(p: ImportProgress): number { - // Use processed/total if available - if (p.processed && p.total) { - return Math.round((p.processed / p.total) * 100) - } - - // Otherwise estimate from stage - const stagePercents = { - detecting: 5, - extracting: 50, - 'storing-vfs': 80, - 'storing-graph': 90, - complete: 100 - } - return stagePercents[p.stage] || 0 - } - - return ( -
- {/* Stage Indicator */} -
- {['detecting', 'extracting', 'storing-vfs', 'storing-graph', 'complete'].map(s => ( - - {s} - - ))} -
- - {/* Progress Bar */} -
-
-
- - {/* Status Message (format-specific but always readable) */} -

{progress.message}

- - {/* Counts */} -
- Entities: {progress.entities} - Relationships: {progress.relationships} -
-
- ) -} -``` - -**This component works perfectly for:** -- ✅ CSV files with 10,000 rows -- ✅ PDF documents with 200 pages -- ✅ Excel workbooks with 5 sheets -- ✅ JSON files with nested structures -- ✅ Markdown documents with sections -- ✅ YAML configuration files -- ✅ DOCX documents with paragraphs - -**No format detection needed. No format-specific rendering. Just works.** - ---- - -### CLI Progress Indicator (Works for ALL Formats) - -```typescript -import ora from 'ora' -import { Brainy } from '@soulcraft/brainy' - -async function importWithProgress(filePath: string) { - const spinner = ora('Starting import...').start() - const brain = await Brainy.create() - - try { - await brain.import(filePath, { - // THIS WORKS FOR ALL 7 FORMATS! - onProgress: (p) => { - // Update spinner text with current message - spinner.text = p.message - - // Add counts if available - if (p.entities || p.relationships) { - spinner.text += ` (${p.entities || 0} entities, ${p.relationships || 0} relationships)` - } - - // Add throughput/ETA if available (during extraction) - if (p.throughput && p.eta) { - const etaSec = Math.round(p.eta / 1000) - spinner.text += ` [${p.throughput.toFixed(1)}/sec, ETA: ${etaSec}s]` - } - - // Change spinner when complete - if (p.stage === 'complete') { - spinner.succeed(p.message) - } - } - }) - } catch (error) { - spinner.fail(`Import failed: ${error.message}`) - } -} - -// Works for ANY format! -await importWithProgress('data.csv') -await importWithProgress('document.pdf') -await importWithProgress('workbook.xlsx') -await importWithProgress('config.yaml') -``` - -**CLI Output (same code, different formats):** - -```bash -# CSV Import -⠋ Detecting format... -⠙ Parsing CSV rows (delimiter: ",") -⠹ Extracting entities from csv (45 rows/sec, ETA: 120s) (150 entities, 45 relationships) -⠸ Extracting entities from csv (45 rows/sec, ETA: 60s) (750 entities, 223 relationships) -✔ Import complete (1350 entities, 401 relationships) - -# PDF Import -⠋ Detecting format... -⠙ Loading PDF document... -⠹ Processing page 5 of 23 -⠸ Extracting entities from pdf (2.5 pages/sec, ETA: 30s) (45 entities, 12 relationships) -✔ Import complete (156 entities, 89 relationships) - -# Excel Import -⠋ Detecting format... -⠙ Loading Excel workbook... -⠹ Reading sheet: Sales (2/5) -⠸ Extracting entities from excel (120 rows/sec, ETA: 45s) (500 entities, 234 relationships) -✔ Import complete (2340 entities, 892 relationships) -``` - -**Same code. Different formats. Perfect progress for all.** - ---- - -### Dashboard with Real-Time Stats (Works for ALL Formats) - -```typescript -function ImportDashboard() { - const [stats, setStats] = useState({ - stage: '', - message: '', - elapsed: 0, - entities: 0, - relationships: 0, - throughput: 0, - eta: 0 - }) - - const startTime = Date.now() - - const handleImport = async (file: File) => { - await brain.import(await file.arrayBuffer(), { - // UNIVERSAL PROGRESS HANDLER - WORKS FOR ALL FORMATS! - onProgress: (p) => { - setStats({ - stage: p.stage, - message: p.message, - elapsed: Date.now() - startTime, - entities: p.entities || 0, - relationships: p.relationships || 0, - throughput: p.throughput || 0, - eta: p.eta || 0 - }) - } - }) - } - - return ( -
-

Import Progress

- -
- - {stats.stage} -
- -
- - {stats.message} -
- -
- - {(stats.elapsed / 1000).toFixed(1)}s -
- -
- - {stats.entities.toLocaleString()} -
- -
- - {stats.relationships.toLocaleString()} -
- - {stats.throughput > 0 && ( -
- - {stats.throughput.toFixed(1)} items/sec -
- )} - - {stats.eta > 0 && ( -
- - {(stats.eta / 1000).toFixed(0)}s -
- )} -
- ) -} -``` - -**This dashboard shows live stats for ANY format** - CSV, PDF, Excel, JSON, Markdown, YAML, DOCX. - ---- - -## 📊 What Messages Look Like (Format-Specific Text, Standard Fields) - -While the **fields are standardized**, the **message text** varies by format to be most helpful: - -```typescript -// CSV Import Messages -"Detecting format..." -"Parsing CSV rows (delimiter: ",")" -"Extracted 1000 rows, inferring types..." -"Extracting entities from csv (45 rows/sec, ETA: 120s)..." -"Creating VFS structure..." -"Import complete" - -// PDF Import Messages -"Detecting format..." -"Loading PDF document..." -"Processing page 5 of 23" -"Extracting entities from pdf (2.5 pages/sec, ETA: 30s)..." -"Creating VFS structure..." -"Import complete" - -// Excel Import Messages -"Detecting format..." -"Loading Excel workbook..." -"Reading sheet: Sales (2/5)" -"Extracting entities from excel (120 rows/sec, ETA: 45s)..." -"Creating VFS structure..." -"Import complete" -``` - -**Key Point:** You can display `progress.message` directly in your UI **without parsing it**. It's always human-readable and contextually appropriate. - ---- - -## ✅ The 5 Standard Stages (Same for ALL Formats) - -Every import goes through these 5 stages in order: - -| Stage | Duration | Description | Fields Available | -|-------|----------|-------------|------------------| -| **detecting** | ~1% | Format detection | `stage`, `message` | -| **extracting** | ~70% | Parse file + AI extraction | `stage`, `message`, `processed`, `total`, `entities`, `relationships`, `throughput`, `eta` | -| **storing-vfs** | ~5% | Create file structure | `stage`, `message` | -| **storing-graph** | ~20% | Create graph nodes | `stage`, `message`, `entities`, `relationships` | -| **complete** | ~1% | Finalize | `stage`, `message`, `entities`, `relationships` | - -**These 5 stages are the same whether you're importing:** -- A 10MB CSV file with 50,000 rows -- A 200-page PDF document -- A 5-sheet Excel workbook -- A nested JSON structure -- A Markdown document -- A YAML configuration -- A DOCX document - ---- - -## 🎯 Why This Matters - -### Build Tools That Work for Everything - -```typescript -// ONE progress handler for your entire application -function universalProgressHandler(progress: ImportProgress) { - // Update UI (works for all formats) - updateProgressBar(progress) - updateStatusText(progress.message) - updateCounts(progress.entities, progress.relationships) - - // Log to analytics (works for all formats) - analytics.track('import_progress', { - stage: progress.stage, - processed: progress.processed, - total: progress.total - }) - - // Send to monitoring (works for all formats) - monitoring.gauge('import.entities', progress.entities) - monitoring.gauge('import.throughput', progress.throughput) -} - -// Use it everywhere -await brain.import(csvFile, { onProgress: universalProgressHandler }) -await brain.import(pdfFile, { onProgress: universalProgressHandler }) -await brain.import(excelFile, { onProgress: universalProgressHandler }) -await brain.import(jsonFile, { onProgress: universalProgressHandler }) -``` - -### No Format Detection Needed - -```typescript -// ❌ DON'T DO THIS (format-specific handling) -if (format === 'csv') { - // CSV-specific progress code -} else if (format === 'pdf') { - // PDF-specific progress code -} else if (format === 'excel') { - // Excel-specific progress code -} - -// ✅ DO THIS (universal handling) -onProgress: (p) => { - // Works for ALL formats! - updateUI(p.stage, p.message, p.entities, p.relationships) -} -``` - ---- - -## 📝 Summary - -✅ **100% Standardized** - Same `ImportProgress` interface for all 7 formats -✅ **Build Once** - Your progress UI works for CSV, PDF, Excel, JSON, Markdown, YAML, DOCX -✅ **No Format Detection** - No need to check file type in your progress handler -✅ **Human-Readable Messages** - Display `progress.message` directly, no parsing needed -✅ **Standard Fields** - `stage`, `processed`, `total`, `entities`, `relationships` work everywhere -✅ **Optional Enhancements** - `throughput`, `eta` available during extraction (all formats) - -**Developers can now build monitoring tools, dashboards, CLIs, and UIs that work perfectly for all import formats with zero format-specific code!** diff --git a/docs/guides/storage-adapters.md b/docs/guides/storage-adapters.md deleted file mode 100644 index 06ec9f3a..00000000 --- a/docs/guides/storage-adapters.md +++ /dev/null @@ -1,162 +0,0 @@ ---- -title: Storage Adapters -slug: guides/storage-adapters -public: true -category: guides -template: guide -order: 2 -description: "Two adapters cover every deployment: in-memory for tests + ephemeral workloads, filesystem for everything that needs to persist. Both share one on-disk contract, including generational history and snapshots. Cloud backup is operator tooling, not a built-in adapter." -next: - - guides/plugins - - concepts/consistency-model ---- - -# Storage Adapters - -Brainy 8.0 ships **two storage adapters**: - -- **`FileSystemStorage`** — persistent on-disk storage. The default for any - deployment that needs to survive a restart. Runs on Node.js, Bun, and Deno. -- **`MemoryStorage`** — in-memory only. The right choice for tests, ephemeral - workloads, and short-lived demos. - -Both implement the same `StorageAdapter` interface, support the full Db API -(generational history, snapshots, restore — see the -[consistency model](../concepts/consistency-model.md)), and use the same -on-disk layout (memory's "disk" is a JS Map). - -## Quick start - -```ts -import { Brainy } from '@soulcraft/brainy' - -// Filesystem (recommended for any persistent workload): -const brain = new Brainy({ - storage: { type: 'filesystem', path: './brainy-data' } -}) - -// Memory (tests, ephemeral): -const brainMem = new Brainy({ storage: { type: 'memory' } }) - -// Auto-detect (filesystem on Node-like runtimes, memory in browsers): -const brainAuto = new Brainy({ storage: { type: 'auto' } }) -``` - -## When to use which - -| Use case | Adapter | Why | -|---|---|---| -| Production app | `filesystem` | Durable, snapshot-able, mmap-able | -| Tests, CI | `memory` | No disk teardown; fast | -| Short-lived data pipeline | `memory` | No persistence needed | -| In-browser demo | `memory` | Filesystem unavailable in browsers | -| Cloud deployment | `filesystem` on local disk + operator backup | See "Cloud backup" below | - -## Cloud backup — operator tooling, not a built-in - -Brainy 8.0 deliberately ships **no cloud storage adapters**. Cloud backup is -handled at the operator layer with standard tooling, the same pattern every -production database uses (Postgres, SQLite, Redis): - -```bash -# After a brainy flush, sync the on-disk artefact to your cloud of choice. -gsutil rsync -r /var/lib/brainy gs://my-backup-bucket/brainy/ -# or: -aws s3 sync /var/lib/brainy s3://my-backup-bucket/brainy/ -# or: -rclone sync /var/lib/brainy remote:brainy-backups/ -# or: -azcopy sync /var/lib/brainy "https://account.blob.core.windows.net/brainy?sv=..." -``` - -Brainy's filesystem layout is sync-friendly: -- Atomic writes (temp + rename) — readers never see torn files -- Per-shard files — `rsync`-style incremental sync works well -- Immutable generation records (`_generations/`) — append-only, cache-friendly - -For point-in-time backups, take a filesystem snapshot (ZFS, btrfs, LVM, EBS, -etc.) or use `brain.now().persist(path)` to write a self-contained snapshot -you can sync independently of the live brain — see -[Snapshots & Time Travel](./snapshots-and-time-travel.md). - -## Why no cloud adapters in 8.0? - -Cloud storage adapters lived in Brainy 4.x-7.x. They were dropped in 8.0 -because: - -- Zero production consumers used them at scale — every known production - deployment ran on local filesystem. -- Cloud-storage HNSW / DiskANN doesn't perform — vector indexes need - low-latency random reads that S3 / GCS / R2 / Azure can't provide - consistently. -- Bundling cloud SDKs into the library cost ~3000-5000 LOC + 4-7 transitive - dependencies for a feature nobody used. -- Cloud backup via operator tooling is strictly more reliable than in-app - upload (better retry semantics, better observability, better cost control). - -Brainy 8.0 is smaller, faster to install, and clearer about what it does. - -## Configuration - -```ts -// BrainyConfig['storage'] — either a config object or a pre-constructed adapter: -storage?: - | { - // The adapter type. Optional — a top-level `path` implies 'filesystem', - // so { path: '/data' } works without it. Defaults to 'auto' - // (filesystem on Node-like runtimes, memory otherwise). - type?: 'auto' | 'memory' | 'filesystem' - - // CANONICAL directory for filesystem storage. The rest of the API already - // speaks `path` (persist(path), Brainy.load(path), asOf(path), - // restore(path)). Specifying it implies type: 'filesystem'. Passed through - // to storage factories, including plugin-provided ones. - path?: string - - // REMOVED in 8.0 — passing `rootDirectory` THROWS. Use `path`. - rootDirectory?: string - - // REMOVED in 8.0 — a nested `options.{path,rootDirectory}` THROWS. Use `path`. - options?: any - } - | StorageAdapter // e.g. storage: new MemoryStorage() -``` - -The canonical — and only — key is the top-level **`path`**. The pre-8.0 aliases -`rootDirectory`, `options.{path,rootDirectory}`, and -`fileSystemStorage.{path,rootDirectory}` were **removed in 8.0**: passing one now -throws with the exact rename (never a silent default that would misplace data on -upgrade). Because `path` implies filesystem, `{ path: '/data' }` is a complete -config; the `type` is optional. - -## Direct construction - -If you want to skip the factory: - -```ts -import { FileSystemStorage, MemoryStorage } from '@soulcraft/brainy' - -const fsStorage = new FileSystemStorage('./brainy-data') -const memStorage = new MemoryStorage() - -const brain = new Brainy({ storage: fsStorage }) -``` - -## Migration from 7.x cloud adapters - -7.x consumers of `OPFSStorage`, `GcsStorage`, `R2Storage`, `S3CompatibleStorage`, -or `AzureBlobStorage` need to migrate to `FileSystemStorage` plus operator -backup tooling. The recipe: - -1. On the host running Brainy, mount a local disk (NVMe recommended). Cloud - providers all expose persistent local disks: GCP Persistent Disk, AWS EBS, - Azure Managed Disks. -2. Set `storage: { type: 'filesystem', path: '/mnt/brainy-data' }`. -3. Run your existing data import once into the new local store. -4. Set up an operator backup job using `gsutil` / `aws s3` / `rclone` / - `azcopy` on a cron — hourly or whatever your RPO requires. Point it at - the brainy data dir. -5. For point-in-time backups, use filesystem snapshots or - `brain.now().persist(path)`. - -Same data, same APIs, no library-side cloud code. diff --git a/docs/guides/streaming-imports.md b/docs/guides/streaming-imports.md deleted file mode 100644 index 6fc80b44..00000000 --- a/docs/guides/streaming-imports.md +++ /dev/null @@ -1,395 +0,0 @@ -# 🌊 Streaming Imports - -> **All imports stream by default - query data as it's imported** - -Brainy imports always use streaming architecture with progressive index flushing, enabling you to query data while it's being imported. - ---- - -## How It Works - -Every import streams with adaptive flush intervals: - -```typescript -await brain.import(file, { - onProgress: async (progress) => { - // Query data during import - if (progress.queryable) { - const products = await brain.find({ type: 'product', limit: 1000 }) - console.log(`${products.length} products imported so far...`) - } - } -}) -``` - -**Benefits**: -- ✅ **Progressive queries**: Data queryable as import proceeds -- ✅ **Crash resilient**: Partial imports survive server restarts -- ✅ **Live monitoring**: Real-time progress with actual data counts -- ✅ **Zero configuration**: Works optimally out of the box - ---- - -## Progressive Flush Intervals - -Brainy automatically adjusts flush intervals **as the import progresses**, based on current entity count: - -| Current Count | Flush Interval | Reason | -|---------------|----------------|--------| -| 0-999 entities | Every 100 | Frequent early updates for better UX | -| 1K-9.9K entities | Every 1000 | Balanced performance/responsiveness | -| 10K+ entities | Every 5000 | Performance focused, minimal overhead | - -**Example**: Importing 5,000 entities -- Flushes at: 100, 200, ..., 900 (9 flushes with interval=100) -- At entity #1000: Interval adjusts to 1000 -- Flushes at: 1000, 2000, 3000, 4000, 5000 (5 more flushes) -- Total flushes: 14 -- Overhead: ~700ms (~0.14% of import time for 5K entities) - -**Why Progressive?** -- ✅ Works with known totals (file imports) -- ✅ Works with unknown totals (streaming APIs, database cursors) -- ✅ Adapts automatically as import grows -- ✅ No configuration needed - -### 🎯 Engineering Insight: Why This Is Advanced - -Most import systems use either: -1. **Fixed intervals** (simple but inefficient for large imports) -2. **Adaptive intervals** (efficient but requires knowing total count upfront) - -Brainy uses **progressive intervals** which combine the best of both: - -```typescript -// Traditional approach (requires total count) -const interval = total < 1000 ? 100 : (total < 10000 ? 1000 : 5000) - -// Brainy's approach (works with unknown totals) -const interval = getProgressiveInterval(currentCount) -// Adjusts dynamically: 100 → 1000 → 5000 as import grows -``` - -**Real-World Impact**: -- **Known totals** (files): Optimal performance automatically -- **Unknown totals** (APIs): Still works perfectly - adjusts on the fly -- **Growing datasets**: UX-focused early (frequent updates), performance-focused later -- **Zero overhead** decisions: Algorithm adapts, developer configures nothing - -This makes Brainy the **only import system** that: -- ✅ Optimizes automatically without configuration -- ✅ Works for both batch and streaming scenarios -- ✅ Balances UX and performance dynamically -- ✅ Scales from 10 to 10 million entities seamlessly - ---- - -## Architecture - -``` -Import Process (Always Streaming) -├─ For each entity: -│ ├─ Extract from source -│ ├─ Classify type (SmartExtractor) -│ ├─ Write to storage ← IMMEDIATE -│ ├─ Update in-memory indexes -│ └─ entitiesSinceFlush++ -│ -├─ When entitiesSinceFlush >= interval: -│ ├─ brain.flush() ← Write indexes to disk -│ ├─ onProgress({ queryable: true }) -│ └─ entitiesSinceFlush = 0 -│ -└─ Final flush at end -``` - -### Key Insight - -Entities write to storage **immediately** on creation. Flushing only writes the search indexes: - -- **Metadata Index** → Fast filtering by type, fields -- **Graph Adjacency Index** → Fast relationship traversal -- **Storage Counts** → Type statistics - -**Without flush**: Entities exist but queries are slow (full table scans) -**With periodic flush**: Entities exist AND queries are fast (index lookups) - ---- - -## Use Cases - -### Use Case 1: Live Import Dashboard - -Show real-time progress with queryable data: - -```typescript -const stats = { - total: 0, - byType: {} as Record -} - -await brain.import(largeCSV, { - onProgress: async (progress) => { - stats.total = progress.entities || 0 - - // Only query after flush - if (progress.queryable) { - const products = await brain.find({ type: 'product', limit: 10000 }) - const people = await brain.find({ type: 'person', limit: 10000 }) - - stats.byType = { - product: products.length, - person: people.length - } - - // Update UI - websocket.send({ stage: progress.stage, stats }) - } - } -}) -``` - -**Output**: -``` -Importing products.csv... -━━━━━━━━━━━━━━━━░░░░░░░░░░ 60% - Products: 12,453 - People: 2,871 - Queryable: ✅ -``` - ---- - -### Use Case 2: Progress Bar with Live Counts - -```typescript -import { ProgressBar } from 'cli-progress' - -const progressBar = new ProgressBar.SingleBar({ - format: 'Importing |{bar}| {percentage}% | {stats}' -}) - -await brain.import(file, { - onProgress: async (progress) => { - if (progress.stage === 'storing-graph' && progress.total) { - if (!progressBar.getProgress()) { - progressBar.start(progress.total, 0, { stats: '' }) - } - - let stats = `${progress.entities || 0} entities` - - // Add queryable count after flush - if (progress.queryable) { - const all = await brain.find({ limit: 100000 }) - stats += ` (${all.length} queryable)` - } - - progressBar.update(progress.processed || 0, { stats }) - } - - if (progress.stage === 'complete') { - progressBar.stop() - } - } -}) -``` - ---- - -### Use Case 3: Conditional Processing - -Make decisions during import based on imported data: - -```typescript -let shouldImportPricing = false - -await brain.import(catalogCSV, { - onProgress: async (progress) => { - if (progress.queryable && progress.processed! > 1000) { - // Check if we have enough products - const products = await brain.find({ type: 'product', limit: 20000 }) - - if (products.length > 10000 && !shouldImportPricing) { - console.log(`Found ${products.length} products - will import pricing next`) - shouldImportPricing = true - } - } - } -}) - -// Conditionally import related data -if (shouldImportPricing) { - await brain.import(pricingCSV) -} -``` - ---- - -## Performance - -### Benchmarks - -| Import Size | Total Time | Flush Overhead | % Overhead | -|-------------|------------|----------------|------------| -| 1K entities | 1.5s | +5ms | 0.3% | -| 10K entities | 15s | +50ms | 0.3% | -| 100K entities | 150s | +500ms | 0.3% | -| 1M entities | 1500s | +5s | 0.3% | - -**Conclusion**: Streaming overhead is negligible (~0.3%) for the benefits gained. - -### Performance Tips - -**1. Limit Query Results** - -```typescript -// ❌ Bad: Fetch all entities (slow for large imports) -onProgress: async (p) => { - if (p.queryable) { - const all = await brain.find({}) // Could be 100K+ entities! - } -} - -// ✅ Good: Limit results or query specific types -onProgress: async (p) => { - if (p.queryable) { - const count = await brain.find({ type: 'product', limit: 10000 }).then(r => r.length) - } -} -``` - -**2. Only Query When Needed** - -```typescript -// ❌ Bad: Query on every progress event -onProgress: async (p) => { - const all = await brain.find({ limit: 10000 }) // Runs 100+ times! -} - -// ✅ Good: Only query after flush -onProgress: async (p) => { - if (p.queryable) { - const all = await brain.find({ limit: 10000 }) // Runs ~10 times - } -} -``` - -**3. Disable Features You Don't Need** - -```typescript -await brain.import(file, { - enableNeuralExtraction: false, // 10x faster - enableRelationshipInference: false, // 5x faster - enableConceptExtraction: false // 2x faster -}) -``` - ---- - -## API Reference - -### ImportProgress - -```typescript -interface ImportProgress { - stage: 'detecting' | 'extracting' | 'storing-vfs' | 'storing-graph' | 'complete' - message: string - processed?: number // Current item number - total?: number // Total items - entities?: number // Entities extracted so far - relationships?: number // Relationships inferred so far - - /** - * Whether data is queryable - * - * true = Indexes flushed, queries will be fast and complete - * false/undefined = Data in storage but indexes not flushed yet - */ - queryable?: boolean -} -``` - -### brain.flush() - -Manually flush indexes to disk: - -```typescript -// Add many entities -for (const entity of entities) { - await brain.add(entity) -} - -// Flush indexes to make queryable -await brain.flush() - -// Now queries will be fast -const results = await brain.find({ type: 'product', limit: 1000 }) -``` - -**Performance**: ~5-50ms per flush (depends on index size) - -**What Gets Flushed**: -- Metadata index (field indexes + EntityIdMapper) -- Graph adjacency index (relationship cache) -- Storage adapter counts (type statistics) - -**What Doesn't Get Flushed** (already persisted): -- Entities (written immediately on `add()`) -- Relationships (written immediately on `relate()`) - ---- - -## Troubleshooting - -### Q: Queries during import are slow - -**A:** Only query when `queryable === true`: - -```typescript -onProgress: async (p) => { - // ✅ Good - if (p.queryable) { - const results = await brain.find({ type: 'product', limit: 1000 }) - } - - // ❌ Bad - queries before flush are slow - const results = await brain.find({ type: 'product', limit: 1000 }) -} -``` - -### Q: How often does data flush? - -**A:** Progressively adjusts based on current entity count: -- 0-999 entities: Every 100 entities -- 1K-9.9K: Every 1000 entities -- 10K+: Every 5000 entities - -The interval increases automatically as more data is imported. Check console output to see when intervals adjust. - ---- - -## Migration from v3.x/v4.0/v4.1 - -No changes required! Streaming is now always enabled with optimal defaults: - -```typescript -// Before (v3.x, v4.0, v4.1): Works the same -await brain.import(file) - -// After: Streaming always on, zero config -await brain.import(file) -``` - -The `flushInterval` option has been removed in favor of automatic progressive intervals that adjust dynamically as the import proceeds. - ---- - -## Further Reading - -- [Import Flow Guide](./import-flow.md) - Complete import pipeline explanation -- [Import Quick Reference](./import-quick-reference.md) - API cheat sheet -- [VFS Guide](./vfs-guide.md) - Virtual file system organization - ---- - -**Questions?** Check the [FAQ](../faq.md) or [open an issue](https://github.com/soulcraft/brainy/issues)! diff --git a/docs/guides/subtypes-and-facets.md b/docs/guides/subtypes-and-facets.md deleted file mode 100644 index ff5de320..00000000 --- a/docs/guides/subtypes-and-facets.md +++ /dev/null @@ -1,575 +0,0 @@ ---- -title: Subtypes & Facets -slug: guides/subtypes-and-facets -public: true -category: guides -template: guide -order: 7 -description: Use the top-level `subtype` field to sub-classify entities within a NounType, track other metadata facets like status or role, and migrate field names without downtime. -next: - - guides/aggregation - - api/reference ---- - -# Subtypes & Facets - -> Sub-classify entities within a NounType, track arbitrary metadata facets, and migrate field names without downtime. - -## Why this exists - -Brainy's `NounType` (Person, Document, Event, Concept, Task, …) is a stable, flat 42-type taxonomy. It's deliberately coarse — every product needs a way to further classify entities *within* a type. Is this `Person` an employee or a customer? Is this `Document` an invoice or a contract? Is this `Event` a meeting or a milestone? - -Three layers solve this: - -| Layer | Use it for | Shape | -|---|---|---| -| **`subtype`** | The primary sub-classification of an entity, one value per entity | Top-level standard field | -| **`trackField()`** | Other facets you want to count or filter on (`status`, `source`, `role`, `paradigm`) | Registered metadata field | -| **`migrateField()`** | Renaming or restructuring fields across an existing dataset | One-shot stream-and-rewrite | - -## Layer 1 — `subtype` - -`subtype` is a top-level standard field on every entity, alongside `type` / `confidence` / `weight`. Flat string, no hierarchy. The vocabulary is your choice — Brainy stores and counts, never validates. - -### Write - -```typescript -import { Brainy, NounType } from '@soulcraft/brainy' - -const brain = new Brainy() -await brain.init() - -await brain.add({ - data: 'Avery Brooks — runs the AI lab', - type: NounType.Person, - subtype: 'employee', // top-level write param - metadata: { department: 'ai-lab' } -}) -``` - -### Read - -```typescript -// Top-level filter — standard-field fast path, no `where` wrapper: -const employees = await brain.find({ type: NounType.Person, subtype: 'employee' }) - -// Set membership: -const internal = await brain.find({ - type: NounType.Person, - subtype: ['employee', 'contractor'] -}) - -// Operator-form predicates use `where`: -const typed = await brain.find({ - type: NounType.Person, - where: { subtype: { exists: true } } -}) -``` - -### What it looks like - -```json -{ - "id": "01HZK3M7TW...", - "type": "person", - "subtype": "employee", - "data": "Avery Brooks — runs the AI lab", - "metadata": { "department": "ai-lab" } -} -``` - -`subtype` lives at the **top level** — NOT inside `metadata`, NOT inside `data`. That's what makes the fast path possible: queries on `subtype` hit the column-store index directly, never the metadata fallback. - -### Counts (O(1)) - -```typescript -// All subtypes for a NounType -brain.counts.bySubtype(NounType.Person) -// → { employee: 12, customer: 847, vendor: 34 } - -// Point count -brain.counts.bySubtype(NounType.Person, 'employee') -// → 12 - -// Top N -brain.counts.topSubtypes(NounType.Person, 3) -// → [['customer', 847], ['employee', 12], ['vendor', 34]] - -// Distinct subtypes for a NounType -brain.subtypesOf(NounType.Person) -// → ['customer', 'employee', 'vendor'] -``` - -These are O(1) lookups backed by `_system/subtype-statistics.json` — no scan, no storage round-trip. The rollup is incrementally maintained as entities are added, updated, and deleted. - -### Aggregation - -`subtype` is a first-class group-by dimension: - -```typescript -const rows = await brain.find({ - aggregate: { - groupBy: ['type', 'subtype'], - metrics: { count: { op: 'COUNT' } } - } -}) -// [ -// { groupKey: { type: 'person', subtype: 'customer' }, count: 847 }, -// { groupKey: { type: 'person', subtype: 'employee' }, count: 12 }, -// { groupKey: { type: 'document', subtype: 'invoice' }, count: 2103 }, -// ... -// ] -``` - -## Layer 2 — `trackField()` - -Some metadata fields aren't *the* sub-classification of an entity but are still worth counting: `status`, `source`, `role`, `paradigm`. Promoting each one to top-level would clutter the contract. `trackField()` registers a metadata field for cardinality + per-NounType breakdown stats without the contract growth. - -### Register and query - -```typescript -// Track a single facet -brain.trackField('status') - -await brain.add({ data: 'Ship subtype', type: NounType.Task, metadata: { status: 'todo' } }) -await brain.add({ data: 'Write docs', type: NounType.Task, metadata: { status: 'done' } }) - -await brain.counts.byField('status') -// → { todo: 1, done: 1 } -``` - -### Per-NounType breakdown - -```typescript -brain.trackField('status', { perType: true }) - -await brain.counts.byField('status', { type: NounType.Task }) -// → { todo: 1, done: 1 } -``` - -### Vocabulary whitelist (opt-in validation) - -```typescript -brain.trackField('priority', { values: ['low', 'medium', 'high'] }) - -// Throws — 'urgent' isn't in the vocabulary: -await brain.add({ - data: 'Fix bug', - type: NounType.Task, - metadata: { priority: 'urgent' } -}) -``` - -`trackField` piggybacks on the existing aggregation engine (see the [Aggregation guide](./aggregation.md) for the underlying mechanism). Backfill-on-define means the first call to `counts.byField()` scans existing entities; subsequent calls are O(groups). - -### subtype vs trackField — when to use which - -- **`subtype`** when there's one primary sub-classification per entity. Limit yourself to one per NounType. Examples: Person→employee/customer/vendor; Document→invoice/contract/policy. -- **`trackField`** for anything else you want to count or filter on. No limit on how many you register. Examples: status, source, role, paradigm. - -## Layer 3 — `migrateField()` - -Use this when you need to rename or restructure a field across an entire dataset — for example, moving a `metadata.kind` convention up to the top-level `subtype` standard field. - -### One-shot rewrite - -```typescript -// Starting state: every entity has metadata.kind -const result = await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype' -}) - -console.log(result) -// { -// scanned: 1500, -// migrated: 1500, -// skipped: 0, -// errors: [] -// } -``` - -After this returns, every entity has `subtype` populated from the old `metadata.kind` value, and `metadata.kind` is cleared. - -### Deprecation window — keep both fields readable - -When you can't coordinate all readers and the migration in a single deploy, use `readBoth: true` to preserve the source field alongside the new one: - -```typescript -// Phase 1: dual-populate (existing readers still work against metadata.kind): -await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype', - readBoth: true -}) - -// ... readers migrate to query subtype at their own pace ... - -// Phase 2: clear the source field when ready: -await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) -``` - -### Supported paths - -| Path form | Refers to | -|---|---| -| `'subtype'`, `'type'`, `'confidence'` | Top-level standard fields | -| `'metadata.X'` | A key under `entity.metadata` | -| `'data.X'` | A key under `entity.data` (when `data` is an object) | -| `'X'` (bare, non-standard) | Shorthand for `metadata.X` | - -### Idempotent - -`migrateField` is safe to re-run. Entities where the source is absent, or where the destination already holds the same value, are skipped. This makes it safe to use in a deploy-once-then-cleanup workflow. - -### Progress reporting - -```typescript -await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype', - batchSize: 500, - onProgress: ({ scanned, migrated }) => { - console.log(`${scanned} scanned, ${migrated} migrated`) - } -}) -``` - -## Putting it together - -A realistic adoption sequence for a brain that started without these primitives: - -```typescript -import { Brainy, NounType } from '@soulcraft/brainy' - -const brain = new Brainy({ storage: { type: 'filesystem', path: './brain-data' } }) -await brain.init() - -// 1. Migrate any existing metadata.kind convention to the new top-level subtype -await brain.migrateField({ from: 'metadata.kind', to: 'subtype', readBoth: true }) - -// 2. Register the other facets you want counted -brain.trackField('status', { perType: true }) -brain.trackField('source') - -// 3. Use subtype on every new write -await brain.add({ - data: 'Quarterly review', - type: NounType.Event, - subtype: 'milestone', - metadata: { status: 'todo', source: 'planning-session' } -}) - -// 4. Query the breakdowns -brain.counts.bySubtype(NounType.Event) -// → { milestone: 14, meeting: 203, deadline: 7 } - -await brain.counts.byField('status', { type: NounType.Event }) -// → { todo: 12, done: 212 } - -// 5. Once all readers are on the new field, drop the source: -await brain.migrateField({ from: 'metadata.kind', to: 'subtype' }) -``` - -## Layer V — `subtype` on relationships (verbs) - -Relationships are first-class citizens too. Every verb (`VerbType`) gets the same `subtype` primitive: a `ReportsTo` relationship might carry `subtype: 'direct'` vs `'dotted-line'`; a `RelatedTo` edge might carry `'spouse'` / `'sibling'` / `'colleague'`. Same shape as the noun side: flat string, no hierarchy, top-level standard field, indexed on the fast path. - -### Write - -```typescript -const ceoId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Avery' }) -const vpId = await brain.add({ type: NounType.Person, subtype: 'employee', data: 'Jordan' }) -const matrixId = await brain.add({ type: NounType.Person, subtype: 'contractor', data: 'Sam' }) - -await brain.relate({ - from: ceoId, - to: vpId, - type: VerbType.ReportsTo, - subtype: 'direct' -}) - -await brain.relate({ - from: ceoId, - to: matrixId, - type: VerbType.ReportsTo, - subtype: 'dotted-line' -}) -``` - -### Read & filter - -```typescript -// Direct reports — fast path filter (column-store hit, not metadata fallback) -const direct = await brain.related({ - from: ceoId, - type: VerbType.ReportsTo, - subtype: 'direct' -}) - -// Set membership -const allReports = await brain.related({ - from: ceoId, - type: VerbType.ReportsTo, - subtype: ['direct', 'dotted-line'] -}) -``` - -### Update (new — `updateRelation()`) - -Verbs previously had no update method — the only way to change a relationship was delete-then-recreate. 7.30 closes that gap: - -```typescript -// Promote a dotted-line report to direct without losing the edge id -await brain.updateRelation({ id: relationId, subtype: 'direct' }) - -// Or change weight/confidence -await brain.updateRelation({ id: relationId, weight: 0.5, confidence: 0.9 }) -``` - -### Traversal filter - -`find({ connected, subtype })` filters traversal edges by their subtype. Composes with `via` (verb-type filter): - -```typescript -// All direct reports two hops deep (will be supported in Cor; for now depth-1) -const directChain = await brain.find({ - connected: { - from: ceoId, - via: VerbType.ReportsTo, - subtype: 'direct', - depth: 1 - } -}) -``` - -Multi-hop subtype filtering (`depth > 1`) lights up on the Cor native path; the JS path throws today rather than return incorrect partial results. - -### Counts - -Same shape as the noun-side counts API: - -```typescript -brain.counts.byRelationshipSubtype(VerbType.ReportsTo) -// → { direct: 12, 'dotted-line': 3 } - -brain.counts.byRelationshipSubtype(VerbType.ReportsTo, 'direct') // O(1) point -// → 12 - -brain.counts.topRelationshipSubtypes(VerbType.ReportsTo, 3) -// → [['direct', 12], ['dotted-line', 3]] - -brain.relationshipSubtypesOf(VerbType.ReportsTo) -// → ['direct', 'dotted-line'] -``` - -These are O(1) lookups backed by the persisted `_system/verb-subtype-statistics.json` rollup — same self-heal machinery as the noun-side rollup. - -### Migrate verb fields - -`migrateField()` now walks verbs too: - -```typescript -// Migrate verb-side metadata.kind → top-level subtype -await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype', - entityKind: 'verb' -}) - -// Or migrate both nouns and verbs in one pass -await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype', - entityKind: 'both' -}) -``` - -Default is `entityKind: 'noun'` (backward-compatible with 7.29). - -## Enforcement — `requireSubtype()` + brain-wide strict mode - -By default (7.30) subtype is optional. Two complementary opt-in mechanisms let you enforce the pairing of type + subtype on every write: - -### Per-type registration - -Mark a specific `NounType` or `VerbType` as requiring a subtype, optionally with a fixed vocabulary: - -```typescript -brain.requireSubtype(NounType.Person, { - values: ['employee', 'customer', 'vendor'], - required: true -}) - -brain.requireSubtype(VerbType.ReportsTo, { - values: ['direct', 'dotted-line'], - required: true -}) - -// Now this throws — Person requires a subtype: -await brain.add({ type: NounType.Person, data: 'no subtype' }) - -// And this throws — 'matrix' isn't in the registered vocabulary: -await brain.relate({ - from: a, - to: b, - type: VerbType.ReportsTo, - subtype: 'matrix' -}) -``` - -### Brain-wide strict mode - -Enforce on every write across the whole brain: - -```typescript -const brain = new Brainy({ requireSubtype: true }) - -// Allow specific types to omit subtype (e.g. catch-all `Thing`) -const brain2 = new Brainy({ - requireSubtype: { except: [NounType.Thing, NounType.Custom] } -}) -``` - -When strict mode is on: - -- Every `add()` / `addMany()` / `update()` / `relate()` / `relateMany()` / `updateRelation()` rejects writes missing a subtype on a non-exempt type. -- `addMany()` and `relateMany()` validate every item BEFORE any storage write — atomic-fail semantics, no partial writes. -- Per-type rules registered via `requireSubtype()` compose with the brain-wide flag; specific rules win when both apply. -- Brainy's own internal writes (VFS root, VFS directories, VFS file entities) bypass enforcement via the `metadata.isVFSEntity: true` infrastructure marker. - -### Migrating to strict mode on an existing brain - -`brain.migrateField()` is your friend — populate `subtype` from an existing convention before flipping strict mode on: - -```typescript -// Step 1: backfill subtype from your existing metadata.kind convention -await brain.migrateField({ - from: 'metadata.kind', - to: 'subtype', - entityKind: 'both' -}) - -// Step 2: register the vocabulary -brain.requireSubtype(NounType.Person, { - values: ['employee', 'customer', 'vendor'], - required: true -}) - -// Step 3: future writes must have a subtype matching the vocabulary -await brain.add({ type: NounType.Person, subtype: 'employee', data: '...' }) -``` - -## Strict mode in practice (for SDK-style vocabulary consumers) - -When a platform layer like the Soulcraft SDK registers `requireSubtype()` rules on behalf of every consumer's brain, every downstream product that calls `brain.add()` / `brain.relate()` against those types must pass a matching `subtype`. Skipping the field — or passing one outside the registered vocabulary — throws at the boundary. - -This pattern is powerful but surfaces a class of latent bug: any `brain.add()` call site that was written before strict-mode adoption starts rejecting writes. A representative production incident: a booking flow started returning 500 on every request because a service method called `brain.add({ type: NounType.Event, ... })` without subtype, and an SDK layer had just registered `requireSubtype()` for `NounType.Event` on every brain instance. - -The fix is a four-step migration recipe — and Brainy 7.30.1+ ships diagnostic tools to make it deterministic. - -### Migration recipe - -1. **Inventory the gap with `brain.audit()`** — returns the deterministic list of which NounTypes and VerbTypes have entities/relationships missing subtype, grouped by type: - - ```typescript - const report = await brain.audit() - // { - // entitiesWithoutSubtype: { event: 24, document: 3, ... }, - // relationshipsWithoutSubtype: { relatedTo: 1402 }, - // total: 1429, - // scanned: 8400, - // recommendation: 'Found 1429 entries without subtype. ...' - // } - ``` - - By default, VFS infrastructure entities are excluded (they bypass enforcement anyway via the `metadata.isVFSEntity` marker). Pass `{ includeVFS: true }` to surface them too. - -2. **Bulk-migrate any existing convention** with `brain.migrateField()` if a legacy field can be lifted: - - ```typescript - // Common pattern: subtype mirrors a discriminator field already in metadata - await brain.migrateField({ - from: 'metadata.entityType', - to: 'subtype', - readBoth: true // safety: keep the source field readable during cutover - }) - ``` - -3. **Hand-fix the remaining call sites.** The exact list is in `report.entitiesWithoutSubtype`. For each call site, add `subtype: ''` to the `brain.add()` / `brain.relate()` params. Choose a stable convention (e.g. mirror `metadata.entityType` if you have one; any rule that's deterministic from the data works). - -4. **Verify with `brain.audit()` again.** Re-run; total should be `0`. If you turn on brain-wide strict mode at this point, all future writes are protected. - -### Brainy's own infrastructure subtype labels (reference) - -Brainy's internal write paths set subtype on every entity and edge they create. Consumers don't need to do anything for these — they're documented here so you understand the data shape: - -| Code path | NounType / VerbType | Subtype label | -|---|---|---| -| VFS root directory `/` | `NounType.Collection` | `'vfs-root'` | -| VFS subdirectories | `NounType.Collection` | `'vfs-directory'` | -| VFS files | (mime-driven, e.g. `Document`/`Code`/`Image`) | `'vfs-file'` | -| VFS symlinks | `NounType.File` | `'vfs-symlink'` | -| VFS Contains edges | `VerbType.Contains` | `'vfs-contains'` | -| Aggregation materialized output | `NounType.Measurement` | `'materialized-aggregate'` | -| Import-document provenance entity | `NounType.Document` | `'import-source'` | -| Importer-extracted entities (no caller default) | extractor-driven | `'imported'` | -| Importer placeholder targets | `NounType.Thing` | `'import-placeholder'` | -| Neural extraction (no caller default) | extractor-driven | `'extracted'` | -| GoogleSheets API entity writes | request-driven | `'imported-from-sheets'` | -| OData API entity writes | request-driven | `'imported-from-odata'` | -| MCP client message storage | `NounType.Message` | `'mcp-message'` | -| `brainy add` CLI (no `--subtype` flag) | user-supplied type | `'cli-add'` | -| `brainy relate` CLI (no `--subtype` flag) | user-supplied verb | `'cli-relate'` | - -You can query these directly: `await brain.find({ subtype: 'vfs-file' })` returns every VFS-managed file regardless of NounType. `await brain.counts.bySubtype(NounType.Document)` shows you the import-source / imported / extracted / vfs-file breakdown. - -Importer and extraction paths accept a caller-supplied `defaultSubtype` option so you can tag a whole batch with your own provenance label (e.g. `'customer-upload-2026q2'`) instead of the Brainy default `'imported'` / `'extracted'`. - -### Looking ahead — Brainy 8.0 - -Brainy 8.0 ships: - -- **`brain.fillSubtypes(rules)`** — the bulk migration helper that pairs with `audit()`. Given caller-supplied rules per NounType / VerbType, it walks the brain and fills in missing subtypes via `update()`. Pre-8.0 brains run this once before upgrading to clear migration debt. -- **`subtype: string` (non-optional)** on `AddParams` and `RelateParams`. TypeScript catches missing subtype at compile time, not just runtime. -- **`new Brainy({ requireSubtype: true })` becomes the default.** Consumers explicitly opt out with `{ requireSubtype: false }` during migration. - -7.30.1's `audit()` is the diagnostic; 8.0's `fillSubtypes()` is the bulk fixer. Together they close the migration gap deterministically. - -## Reference - -### Layer 1 — `subtype` (nouns) - -- `brain.add({ ..., subtype: 'value' })` — write the field -- `brain.update({ id, subtype: 'value' })` — change the field -- `brain.find({ type, subtype })` — filter (fast path) -- `brain.find({ subtype: ['a', 'b'] })` — set membership -- `brain.counts.bySubtype(type, subtype?)` — O(1) counts -- `brain.counts.topSubtypes(type, n?)` — top N by count -- `brain.subtypesOf(type)` — distinct subtype list - -### Layer V — `subtype` (verbs / relationships) - -- `brain.relate({ ..., subtype: 'value' })` — write the field -- `brain.updateRelation({ id, subtype, type?, weight?, ... })` — change a relationship -- `brain.related({ subtype })` — filter (fast path) -- `brain.related({ subtype: ['a', 'b'] })` — set membership -- `brain.find({ connected: { via, subtype, depth } })` — traversal filter -- `brain.counts.byRelationshipSubtype(verb, subtype?)` — O(1) counts -- `brain.counts.topRelationshipSubtypes(verb, n?)` — top N by count -- `brain.relationshipSubtypesOf(verb)` — distinct subtype list - -### Layer 2 — generic facets - -- `brain.trackField(name, { perType?, values? })` — register a facet -- `brain.counts.byField(name, { type? })` — facet counts - -### Layer 3 — migration - -- `brain.migrateField({ from, to, readBoth?, batchSize?, onProgress?, entityKind? })` — rewrite a field (nouns, verbs, or both) - -### Enforcement - -- `brain.requireSubtype(type, { values?, required })` — per-`NounType` / `VerbType` rule -- `new Brainy({ requireSubtype: true })` — brain-wide strict mode -- `new Brainy({ requireSubtype: { except: [type, ...] } })` — strict with exemptions diff --git a/docs/guides/upgrading-7-to-8.md b/docs/guides/upgrading-7-to-8.md deleted file mode 100644 index a3c64fb9..00000000 --- a/docs/guides/upgrading-7-to-8.md +++ /dev/null @@ -1,186 +0,0 @@ ---- -title: Upgrading from 7.x to 8.0 -slug: guides/upgrading-7-to-8 -public: true -category: guides -template: guide -order: 10 -description: What the one-time 7→8 on-disk migration does, how 8.0 automatically recovers Virtual Filesystem content that older layouts stored in the removed copy-on-write area, and how to verify (or force) that recovery. -next: - - guides/storage-adapters - - guides/inspection ---- - -# Upgrading from 7.x to 8.0 - -Opening a 7.x on-disk store with Brainy 8.0 runs a **one-time, in-place layout -migration** the first time the store is opened. It is automatic (`autoMigrate` -defaults to `true`), it runs once, and it stamps a marker so every later open is -a no-op. - -Almost everything about the upgrade is transparent: your entities, relationships, -metadata, and indexes migrate and rebuild without any action on your part. This -guide covers the **one case that needs attention** — Virtual Filesystem (VFS) -content — and how 8.0 recovers it for you. - -## TL;DR - -- **Just upgrade to `@soulcraft/brainy@8.0.12` (or later) and open the store.** - If a previous upgrade left VFS content stranded, 8.0.12 **heals it on open**, - with no operator action. -- Want to force or script it? Call **`await brain.vfs.adoptOrphanedBlobs()`**. -- The recovery is **non-destructive and idempotent** — it copies, never moves, - and running it twice is a no-op. - -## What the migration does - -The 7.x on-disk layout stored entities under a per-branch path -(`branches//entities/...`). 8.0 uses a flat layout (`entities/...`). On -first open, Brainy: - -1. Collapses `branches//entities/*` into the flat `entities/*` layout. -2. Rebuilds the derived indexes (vector, graph, metadata) and count rollups from - the canonical entities. -3. Stamps `_system/migration-layout.json` so re-opening is a no-op. -4. Takes an automatic **pre-upgrade backup** of the directory before it starts, - and removes it once the upgrade is verified complete (see - [The safety net](#the-safety-net-pre-upgrade-backup) below). - -The log line to expect (once per store): - -``` -[brainy] Migrating a 7.x branch layout (branches/main) to the 8.0 flat layout - in place — 13175 entity files. This runs once; back up the directory first if - you need a rollback (8.0 does not keep the old layout). -``` - -## The one thing that needs recovery: VFS content - -If your application uses the Virtual Filesystem (VFS) to store file content (for -example, a CMS that keeps page documents at paths like -`/pages/homepage/page.json`), that content is held as **content blobs**. - -7.x kept those blobs in the branch system's **copy-on-write area** (`_cow/`). -8.0 removed the branch/copy-on-write system, and its content blobs live in the -content-addressed store (`_cas/`). The layout migration moves entities — it does -**not** move the VFS content blobs. Left alone, a 7.x store's VFS blobs would -remain in `_cow/`, and a read of one would throw: - -``` -VFS: Cannot read blob for /pages/homepage/page.json: - Blob metadata not found: 6b87cfb71ad2f04602a5c157214dc42000... -``` - -### 8.0.12 recovers them automatically - -Brainy 8.0.12 adds an **on-open recovery pass** that runs right after the layout -migration. It scans `_cow/`, and for every content blob whose `blob:` + -`blob-meta:` pair is not already in `_cas/`, it copies **both** across into the -8.0 store. It: - -- **Heals a fresh 7→8 upgrade** (where `_cow/` still holds the blobs), **and** -- **Heals a store already upgraded by an earlier 8.0.x** that stranded them — - the recovery is gated on the presence of `_cow/` and its own marker, not on - the layout-migration marker, so an already-migrated store is still healed. -- Is a **cheap no-op** on a native-8.0 or fresh store (there is no `_cow/`), and - on a store already healed (a marker records completion so later opens skip the - scan). - -So the operator action for a stranded store is simply: **upgrade to 8.0.12 and -open it.** - -```ts -import { Brainy } from '@soulcraft/brainy' - -// Opening the store is all that is required — recovery runs during init(). -const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/my-store' } }) -await brain.init() - -// The previously-failing read now succeeds. -const page = await brain.vfs.readFile('/pages/homepage/page.json') -``` - -On a store that needed recovery you will see: - -``` -[brainy] Recovered 337 VFS content blob(s) stranded by a 7→8 upgrade - (adopted _cow/ → _cas/ in place). -``` - -### Forcing recovery explicitly - -If you would rather run the recovery deliberately (for example, in an upgrade -script that asserts a clean result before flipping traffic), call it directly: - -```ts -const result = await brain.vfs.adoptOrphanedBlobs() -// → { cowBlobs, adopted, alreadyPresent, incomplete } -console.log(`adopted ${result.adopted}, already present ${result.alreadyPresent}`) -if (result.incomplete > 0) { - // One or more _cow/ blobs are missing their bytes or metadata — investigate - // _cow/ before discarding your own backup. See "The safety net" below. -} -``` - -`adoptOrphanedBlobs()` self-initializes the brain, so it is safe to call -immediately after construction. It returns: - -| Field | Meaning | -| ---------------- | ---------------------------------------------------------------- | -| `cowBlobs` | Distinct content-blob hashes found in `_cow/`. | -| `adopted` | Newly copied into `_cas/` on this call. | -| `alreadyPresent` | Already in `_cas/` (a prior open or run adopted them). | -| `incomplete` | A `_cow/` blob missing its bytes *or* its metadata — **skipped** rather than half-adopted. | - -## Verifying recovery - -After opening under 8.0.12, confirm a previously-failing path reads: - -```ts -const content = await brain.vfs.readFile('/pages/homepage/page.json') -console.log(content.toString().slice(0, 80)) -``` - -If you scripted it with `adoptOrphanedBlobs()`, a clean result is -`incomplete === 0` and `adopted + alreadyPresent === cowBlobs`. - -## The safety net: pre-upgrade backup - -Brainy takes an automatic pre-upgrade backup and removes it once the upgrade is -**verified complete**. In 8.0.12 that verification includes VFS content: if the -blob recovery reports `incomplete > 0`, the completion marker is **not** stamped -(the next open retries) and the **pre-upgrade backup is retained** so you still -have a rollback while blobs remain unaccounted for. You will see: - -``` -[brainy] VFS blob recovery adopted N blob(s) but M orphaned _cow/ blob(s) are - missing their bytes or metadata and were left in place. The pre-upgrade backup - is being retained; inspect _cow/ before discarding it. -``` - -The recovery never deletes anything from `_cow/`, so the original blobs stay in -place for inspection or a manual rollback regardless. - -## Who is affected - -**Any 7.x store that used the VFS to store file content** (so it has a `_cow/` -area) is a candidate for stranded blobs on a 7→8 upgrade. Stores that never used -the VFS have no `_cow/` content blobs and are unaffected. - -Because the recovery is **gated on the presence of `_cow/`**, it is -self-selecting and safe to roll out everywhere: - -- On a store with stranded blobs → it adopts them. -- On a native-8.0, fresh, or non-VFS store → it is a no-op on the existence - check. - -There is no configuration to set and nothing to opt into. Upgrading to 8.0.12 -and opening each store is sufficient. - -## Rollback - -The recovery is copy-only, so no rollback of the recovery itself is ever needed. -If you need to roll back the **whole** 7→8 upgrade, restore the directory from -your pre-upgrade backup (retained automatically while recovery is incomplete, or -your own snapshot) and pin `@soulcraft/brainy@7.x`. 8.0 does not keep the old -branch layout in place, so a directory-level restore is the rollback path. diff --git a/docs/guides/vue-integration.md b/docs/guides/vue-integration.md index 7f7c6a06..a38c92ed 100644 --- a/docs/guides/vue-integration.md +++ b/docs/guides/vue-integration.md @@ -2,8 +2,6 @@ Complete guide to integrating Brainy with Vue.js applications, covering Vue 3, Nuxt.js, Composition API, Options API, and advanced patterns. -> **Runtime**: Brainy 8.0 runs on Node.js 22+ and Bun, server-side only — it is not a browser library. In Vue apps, Brainy lives behind an HTTP endpoint (a Nuxt server route, or any Node/Bun backend), and your components call that endpoint. The composables, plugin, and store below are wrappers around `fetch`; the single Brainy instance is created once on the server (see [Nuxt Server Route](#nuxt-server-route)). - ## 🚀 Quick Start ### Installation @@ -17,47 +15,34 @@ npm install @soulcraft/brainy ### Basic Setup -The composable talks to a Brainy-backed endpoint (see [Nuxt Server Route](#nuxt-server-route)). It is always "ready" because there is no client-side initialization — the server owns the Brainy instance. - ```javascript // src/composables/useBrainy.js -import { ref } from 'vue' +import { ref, onMounted } from 'vue' +import { Brainy } from '@soulcraft/brainy' -export function useBrainy(endpoint = '/api/brain') { - const error = ref(null) +const brain = ref(null) +const isReady = ref(false) +const error = ref(null) - const search = async (query, options = {}) => { - error.value = null +export function useBrainy() { + onMounted(async () => { try { - const res = await fetch(`${endpoint}/search`, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ query, options }) + brain.value = new Brainy({ + storage: { type: 'opfs' } // Browser storage }) - if (!res.ok) throw new Error(`Search failed: ${res.status}`) - return (await res.json()).results + await brain.value.init() + isReady.value = true } catch (err) { error.value = err.message - console.error('Brainy search failed:', err) - return [] + console.error('Brainy initialization failed:', err) } - } + }) - const add = async (data, type, metadata) => { - const res = await fetch(`${endpoint}/add`, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ data, type, metadata }) - }) - return (await res.json()).id + return { + brain: readonly(brain), + isReady: readonly(isReady), + error: readonly(error) } - - const stats = async () => { - const res = await fetch(`${endpoint}/stats`) - return await res.json() - } - - return { search, add, stats, error: readonly(error) } } ``` @@ -69,36 +54,43 @@ export function useBrainy(endpoint = '/api/brain') {