The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).
Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
readWriteSeparation, queryPlanner, healthMonitor, configManager,
hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
(slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
src/config/extensibleConfig.ts (config/augmentation registry built on removed
cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
enableDistributedSearch (dead config flag); the metadata partition field;
the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
storage-architecture; reframed scale prose to the 8.0 model.
Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.
RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
Final cleanup pass for Brainy 8.0. Catches three categories of debt:
A. STEP-7 FOLLOW-THROUGH (rebuild-path collapse)
Step 7's bisect-reset (debugging a flaky test) lost the in-source edits
to three rebuild paths even though the commit message claimed they
shipped. Re-applied now:
- src/utils/metadataIndex.ts — collapsed the `isLocalStorage` /
cloud-pagination branching. Local-load-all-at-once is the only path
in 8.0. Removed ~150 LOC of paginated-cloud branching for both nouns
and verbs, plus the safety counters (`consecutiveEmptyBatches`,
`MAX_ITERATIONS`, etc.).
- src/hnsw/hnswIndex.ts — same simplification for HNSW rebuild. The
paginated cloud path is gone; HNSW now loads all nodes at once.
Removed ~85 LOC.
- src/graph/graphAdjacencyIndex.ts — same simplification for graph
adjacency rebuild. Removed ~50 LOC.
The collapse is safe because cloud adapters were deleted in step 7;
`storageType === 'OPFSStorage'` (and similar) can never match now.
B. CLOUD-ONLY DOCS DELETED
- docs/operations/cost-optimization-aws-s3.md
- docs/operations/cost-optimization-azure.md
- docs/operations/cost-optimization-cloudflare-r2.md
- docs/operations/cost-optimization-gcs.md
- docs/operations/cloud-run-filestore-guide.md
(docs/deployment/* contained no cloud-specific files that needed deletion.)
C. STORAGE-ADAPTERS GUIDE REWRITTEN FOR 8.0
docs/guides/storage-adapters.md → fresh content reflecting the 8.0
reality:
- Two adapters: FileSystemStorage + MemoryStorage. Quick-start matrix.
- Cloud backup section explains the operator-tooling pattern (gsutil /
aws s3 / rclone / azcopy) with the exact commands consumers will run.
- "Why no cloud adapters in 8.0?" section documents the four reasons
per BR-BRAINY-80-STORAGE-SIMPLIFY.
- Migration recipe for 7.x cloud-adapter consumers: mount local disk →
filesystem storage → operator backup cron.
Updated frontmatter description so soulcraft.com/docs renders the
correct preview.
NOT IN THIS COMMIT (deliberate, lower-priority)
- src/storage/cacheManager.ts still references StorageType.S3 /
REMOTE_API / OPFS as dead branches (23 sites). The branches are never
reached in 8.0, but cleaning them would cascade through 5 consumers.
Defer to a follow-up if the dead code surfaces as a real maintenance
issue.
- src/config/storageAutoConfig.ts keeps its StorageType enum + autodetect
for 7.x compat surface. Same reason: rewriting cascades through
zeroConfig, extensibleConfig, sharedConfigManager. Defer.
- docs/MIGRATION-V3-TO-V4.md and docs/DEVELOPER_LEARNING_PATH.md still
reference cloud adapters as historical artefacts. That's accurate —
they describe how things used to be. Left as-is.
- @deprecated audit in src/ (10 files) deferred — audit each individually
in a future polish pass.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding from
step 7; no regressions from this cleanup)
brain.add() was generating 26-40 immediate cloud writes per call, causing
HTTP 429 rate limit errors and high latency on GCS/S3/R2/Azure. Three-layer
fix: (1) deferred metadata writes with dirty-marking, (2) MetadataWriteBuffer
for write coalescing, (3) retry/backoff on all cloud storage adapters.
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:
1. validateConsistency() to falsely detect corruption on every startup,
triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
and report inflated totalEntries/totalIds stats
Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
