Commit graph

5 commits

Author SHA1 Message Date
c0d326b36d feat: verb subtype + updateRelation + requireSubtype enforcement
Brings verbs to first-class parity with nouns. The 7.29.0 subtype primitive
shipped for entities only; this release ships the symmetric verb mirror plus
the enforcement layer for ensuring every entity AND every relationship has
both type AND subtype.

Layer V1 — verb subtype mirror
- HNSWVerbWithMetadata.subtype + STANDARD_VERB_FIELDS set + resolveVerbField()
- Relation<T>.subtype, RelateParams<T>.subtype, UpdateRelationParams<T> extended,
  GetRelationsParams.subtype, GraphConstraints.subtype (for find connected)
- relate() persists subtype on verbMetadata + GraphVerb + transaction ops
- getRelations({ type, subtype }) fast-path filter with set membership
- find({ connected: { via, subtype, depth } }) traversal filter (depth-1 on
  the JS path; explicit error on depth > 1 pointing at Cortex native)
- verbsToRelations + storage destructure sites surface subtype to top-level
- All three graph-index fast-path queries (getVerbsBySource/ByTarget) enrich
  with subtype from metadata

Layer V2 — updateRelation() closes a pre-7.30 gap
- New first-class verb update method (parallel to update() for nouns)
- Changes subtype/type/weight/confidence/data/metadata in place
- Re-indexes in graph adjacency when verb type changes; id preserved
- validateUpdateRelationParams enforces id + at-least-one-field-to-update

Layer V3 — verb subtype storage rollup
- verbSubtypeCountsByType: Map<number, Map<string, number>> on BaseStorage
- verbSubtypeByIdCache for self-heal during update/delete
- incrementVerbSubtypeCount + decrementVerbSubtypeCount maintain state
- loadVerbSubtypeStatistics + saveVerbSubtypeStatistics persist to
  _system/verb-subtype-statistics.json (mirrors noun-side shape)
- rebuildVerbSubtypeCounts for poison recovery / explicit repair
- getVerbSubtypeCountsByType accessor for the public counts API
- Wired into init() / flushCounts() / saveVerbMetadata / deleteVerbMetadata

Layer V4 — verb counts API + relationshipSubtypesOf
- brain.counts.byRelationshipSubtype(verb, subtype?) — O(1) breakdown or point
- brain.counts.topRelationshipSubtypes(verb, n) — top N by count
- brain.relationshipSubtypesOf(verb) — sorted distinct subtypes

Layer V5 — migrateField extended to verbs
- New entityKind?: 'noun' | 'verb' | 'both' option (default 'noun')
- Mirror verb iteration via storage.getVerbs() with same path semantics
- verbToRelationLike + buildRelationMigrationUpdate helpers project the
  storage verb shape onto the Entity<T>-shaped surface readPath understands
- Routes through new updateRelation() for the verb-side rewrite

Enforcement (opt-in in 7.30, default in 8.0)
- brain.requireSubtype(type, options) — unified API for NounType OR VerbType.
  Registers per-type rules with optional values whitelist; composes with the
  brain-wide flag.
- new Brainy({ requireSubtype: true }) — brain-wide strict mode. Every public
  write path validates the pairing guarantee.
- { except: [NounType.Thing, ...] } form for catch-all type exemptions
- Atomic-fail semantics on addMany / relateMany — pre-validate every item
  before any storage write, throw on first failure with item index
- Per-type rules + brain-wide flag both throw with descriptive messages
- VFS infrastructure bypass via metadata.isVFSEntity / isVFS markers so
  brain's own VFS writes don't get rejected when strict mode is on

VFS labeling — concrete subtypes for infrastructure entities
- VFS root: NounType.Collection + subtype: 'vfs-root' (was bare Collection)
- VFS directories: subtype: 'vfs-directory'
- VFS files: subtype: 'vfs-file' (NounType still mime-based)
- VFS containment edges: VerbType.Contains + subtype: 'vfs-contains'
- Lets consumers cleanly enumerate VFS state via find({ subtype: 'vfs-file' })
  and distinguish Brainy's VFS Collections from user-created Collections

Docs
- docs/guides/subtypes-and-facets.md extended with Layer V (Verbs) section +
  Enforcement section. New full reference at the bottom split into Layer 1
  (nouns), Layer V (verbs), Layer 2 (facets), Layer 3 (migration), Enforcement.
- docs/api/README.md adds updateRelation(), getRelations({ subtype }), the
  three verb-side counts methods, requireSubtype(), and the brain-wide
  constructor option. relate() params include subtype.
- docs/DATA_MODEL.md adds a Subtype-for-VerbType section + STANDARD_VERB_FIELDS
- docs/architecture/finite-type-system.md extends Principle 1a to verbs
- docs/QUERY_OPERATORS.md adds a verb-subtype filter section covering
  getRelations and find({connected, subtype}) traversal
- README.md "Subtypes" section now shows both noun + verb in one example +
  the enforcement APIs
- RELEASES.md v7.30.0 entry with the full noun/verb capability parity matrix

Tests
- tests/integration/verb-subtype-and-enforcement.test.ts — 30 new tests
  covering V1 round-trips, V1 set membership, updateRelation in place,
  updateRelation preservation, V2 counts breakdown + point + topN + distinct,
  V2 decrements on unrelate, V2 re-routes on updateRelation, V3 depth-1
  traversal filter, V3 depth>1 explicit error, V4 verb migration, V4 both
  entity kinds, V4 readBoth preservation, V5 per-type required rejection,
  V5 vocabulary rejection, V5 on-vocab acceptance, V5 verb-side enforcement,
  V5 addMany atomic-fail, V5 relateMany atomic-fail, V5 update enforcement,
  V5 updateRelation enforcement, V5 brain-wide strict mode, V5 except clause.

Verification
- Unit suite: 1468/1468 passing
- Noun subtype integration (7.29 carryover): 26/26 passing
- Verb subtype + enforcement integration: 30/30 passing
- Type-check: clean
- Build: clean
- Public closed-source reference audit: clean

Internal 8.0 spec
- .strategy/BRAINY-8.0-SUBTYPE-CONTRACT.md (gitignored, not in npm artifact)
  documents the contract upgrade Cortex 3.0 implements against: required-by-
  default subtype, SubtypeRegistry typing hook, native simplification,
  multi-hop traversal native fast path, brain.fillSubtypes() migration helper.
  Coordinated via PLATFORM-HANDOFF rows CTX-SUBTYPE-PARITY-V2 (7.30 parallel
  work) and CTX-SUBTYPE-8.0-CONTRACT (8.0 spec).
2026-06-05 11:15:52 -07:00
2cdf70ee0f feat: subtype top-level field + trackField + migrateField
Promotes `subtype?: string` to a top-level standard field on every entity,
alongside `type` / `confidence` / `weight`. Flat string, no hierarchy — the
consumer-chosen vocabulary for sub-classifying entities within a NounType
(Person → employee/customer, Document → invoice/contract, etc.).

Layer 1 — subtype field + rollup
- HNSWNounWithMetadata.subtype + STANDARD_ENTITY_FIELDS entry
- Entity / Result / AddParams / UpdateParams / FindParams threading
- add()/update() persist subtype on storageMetadata + entityForIndexing
- get()/find() route through the standard-field fast path
- subtypeCountsByType (Map<NounTypeIdx, Map<subtype, count>>) on
  BaseStorage, mirrored after nounCountsByType with the same self-heal
  rebuild and persisted to _system/subtype-statistics.json
- brain.counts.bySubtype(type, subtype?) — O(1) point + breakdown
- brain.counts.topSubtypes(type, n) — top-N by count
- brain.subtypesOf(type) — distinct subtypes seen
- find({ type, subtype }) and find({ subtype: ['a','b'] }) on the fast path

Layer 2 — trackField for other facets
- brain.trackField(name, { perType?, values? }) registers a field for
  cardinality + per-NounType breakdown stats. Backed by the aggregation
  engine (auto-defines __fieldCounts__<name>), backfill-on-define applies.
- brain.counts.byField(name, { type? }) returns value frequencies
- Optional vocabulary whitelist rejects off-vocabulary writes at add/update

Layer 3 — generic migrateField
- brain.migrateField({ from, to, readBoth?, batchSize?, onProgress? })
  streams every entity, copies the value from one path to another, and
  (unless readBoth) clears the source. Supports top-level standard fields,
  metadata.X, and data.X paths. Idempotent — safe to re-run.

Docs
- New guide: docs/guides/subtypes-and-facets.md (Layer 1 + 2 + 3)
- README, DATA_MODEL, QUERY_OPERATORS, api/README, finite-type-system,
  quick-start all treat subtype as a core primitive with anonymous example
  vocabularies (employee/customer/invoice/milestone).

Tests
- 26 new integration tests covering write/read/update/delete round-trips,
  counts rollup decrement + re-route on mutation, trackField + byField
  with and without perType, vocabulary whitelist enforcement, and
  migrateField for metadata.X → subtype and data.X → subtype paths
  including readBoth deprecation-window semantics.

Unit suite: 1468/1468 passing. Type-check + build clean.
2026-06-04 17:25:28 -07:00
da7d2ed29d feat: migrate embeddings to Candle WASM + remove semantic type inference
Major architectural changes:

1. EMBEDDINGS ENGINE (ONNX → Candle WASM):
   - Replace ONNX Runtime with Rust Candle compiled to WASM
   - Embedded model in WASM binary (no external downloads)
   - Quantized Q8 precision with <50MB memory footprint
   - Zero-download, offline-first operation
   - Same embedding quality (all-MiniLM-L6-v2)

2. REMOVE SEMANTIC TYPE INFERENCE:
   - Delete embeddedKeywordEmbeddings.ts (14MB of pre-computed embeddings)
   - Remove typeAwareQueryPlanner.ts and semanticTypeInference.ts
   - Remove VerbExactMatchSignal (uses keyword embeddings)
   - Update SmartRelationshipExtractor to 3 signals (55%/30%/15% weights)

API CHANGES (requires v7.0.0):
- Removed: inferTypes(), inferNouns(), inferVerbs(), inferIntent()
- Removed: getSemanticTypeInference(), SemanticTypeInference class
- Removed: TypeInference, SemanticTypeInferenceOptions types

Users can still use natural language queries in find() - they just
need to specify type explicitly for type-optimized searches.

PACKAGE SIZE IMPACT:
- Compressed: 90.1 MB → 86.2 MB (-4.3%)
- Uncompressed: 114.4 MB → 100.3 MB (-12%)
- ~448K lines of code removed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-06 12:52:34 -08:00
52e961760d docs: label all performance claims as MEASURED vs PROJECTED (NO FAKE CODE compliance)
Fixed 10 evidence violations across 5 files per NO FAKE CODE policy:
- All billion-scale claims now labeled as PROJECTED (not yet benchmarked)
- Distinguishes calculated projections from empirical measurements
- Maintains architectural honesty about what's tested vs theoretical

Files updated:
- src/hnsw/typeAwareHNSWIndex.ts (2 claims)
- src/utils/metadataIndex.ts (1 claim)
- src/query/typeAwareQueryPlanner.ts (1 claim)
- docs/architecture/finite-type-system.md (2 claims)
- CHANGELOG.md (4 claims)

Changes:
- 87% HNSW memory reduction → PROJECTED (calculated from architecture)
- 86% metadata memory reduction → PROJECTED (calculated from chunking)
- 385x type tracking reduction → PROJECTED (calculated from Uint32Array)
- 40% query latency reduction → PROJECTED (calculated from graph reduction)

All claims remain architecturally sound but are now honestly labeled.
Future TIER 4 work will add benchmarks to upgrade PROJECTED → MEASURED.

Audit document: .strategy/EVIDENCE_VIOLATIONS_AUDIT.md
2025-11-14 08:26:45 -08:00
92c96246fb feat(v4.0.0): Complete metadata/vector separation architecture with Azure support
This commit completes the core v4.0.0 architecture changes for billion-scale
performance with metadata/vector separation. NO RELEASE YET - remaining optimizations
and testing required before production release.

## Core v4.0.0 Architecture Changes

### Type System Updates
- Fixed all TypeScript compilation errors (zero errors achieved)
- Updated HNSWNoun/HNSWVerb to separate core fields from metadata
- Implemented HNSWNounWithMetadata/HNSWVerbWithMetadata for API boundaries
- Added required 'noun' field to NounMetadata for semantic structure
- Renamed verb.type to verb.verb for consistency

### Storage Adapter Updates
**All adapters updated for v4.0.0 two-file storage pattern:**
- memoryStorage: Proper metadata/vector separation
- fileSystemStorage: Two-file pattern with sharding
- opfsStorage: Browser persistent storage updated
- s3CompatibleStorage: AWS/MinIO/DigitalOcean support
- r2Storage: Cloudflare R2 optimization
- gcsStorage: Google Cloud with ADC support
- **azureBlobStorage: NEW - Full Azure Blob Storage support**

### Storage Features
- BaseStorage: Internal vs public method separation (_getNoun vs getNoun)
- Two-file storage: Vectors in one file, metadata in another
- Change tracking: getChangesSince return type updated
- Pagination: getNounsWithPagination returns WithMetadata types

### Azure Blob Storage Integration (NEW)
- Native @azure/storage-blob SDK integration
- Four authentication methods:
  * DefaultAzureCredential (Managed Identity) - recommended
  * Connection String - simplest setup
  * Account Name + Key - traditional auth
  * SAS Token - delegated access
- High-volume mode with write buffering
- Adaptive backpressure for throttling
- UUID-based sharding for billion-scale
- Full HNSW support with graph persistence

### Utility Updates
- EmbeddingManager: Updated to accept Record<string, unknown>
- LSMTree: Wrapped data in NounMetadata structure with 'noun' field
- EntityIdMapper: Fixed nested metadata.data structure access
- MetadataIndex: Fixed field type inference integration
- PeriodicCleanup: Updated for new metadata structure

### Core API Updates
- Brainy: Updated verb property access from v.type to v.verb
- ConfigAPI: Fixed NounMetadata access patterns
- DataAPI: Updated metadata handling

### Documentation Updates
- CREATING-AUGMENTATIONS.md: v4.0.0 breaking changes guide
- DEVELOPER-GUIDE.md: Migration checklist and examples
- COMPLETE-REFERENCE.md: v4.0.0 architecture improvements
- **finite-type-system.md: NEW - Revolutionary type system benefits**

### Build & Dependencies
- Zero TypeScript compilation errors
- Added @azure/storage-blob and @azure/identity
- 591 tests passing (23 timeout in long-running neural tests)

## What's NOT in This Release
This is a work-in-progress commit. Before v4.0.0 release we need:
- Storage adapter optimizations (batch operations, compression)
- Azure blob tier management (Hot/Cool/Archive)
- Cost optimization implementations
- Additional performance testing at billion-scale
- Migration guides for v3.x users

## Testing
- Clean build: 
- Type checking:  (zero errors)
- Test suite:  (591/614 passing, timeouts in neural tests only)

🔐 Generated with Claude Code
https://claude.com/claude-code

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-17 12:29:27 -07:00