chore(8.0): step-7 follow-through — collapse remaining cloud branches + docs sweep (scaffold step 13)

Final cleanup pass for Brainy 8.0. Catches three categories of debt:

A. STEP-7 FOLLOW-THROUGH (rebuild-path collapse)

Step 7's bisect-reset (debugging a flaky test) lost the in-source edits
to three rebuild paths even though the commit message claimed they
shipped. Re-applied now:

- src/utils/metadataIndex.ts — collapsed the `isLocalStorage` /
  cloud-pagination branching. Local-load-all-at-once is the only path
  in 8.0. Removed ~150 LOC of paginated-cloud branching for both nouns
  and verbs, plus the safety counters (`consecutiveEmptyBatches`,
  `MAX_ITERATIONS`, etc.).
- src/hnsw/hnswIndex.ts — same simplification for HNSW rebuild. The
  paginated cloud path is gone; HNSW now loads all nodes at once.
  Removed ~85 LOC.
- src/graph/graphAdjacencyIndex.ts — same simplification for graph
  adjacency rebuild. Removed ~50 LOC.

The collapse is safe because cloud adapters were deleted in step 7;
`storageType === 'OPFSStorage'` (and similar) can never match now.

B. CLOUD-ONLY DOCS DELETED

- docs/operations/cost-optimization-aws-s3.md
- docs/operations/cost-optimization-azure.md
- docs/operations/cost-optimization-cloudflare-r2.md
- docs/operations/cost-optimization-gcs.md
- docs/operations/cloud-run-filestore-guide.md

(docs/deployment/* contained no cloud-specific files that needed deletion.)

C. STORAGE-ADAPTERS GUIDE REWRITTEN FOR 8.0

docs/guides/storage-adapters.md → fresh content reflecting the 8.0
reality:
- Two adapters: FileSystemStorage + MemoryStorage. Quick-start matrix.
- Cloud backup section explains the operator-tooling pattern (gsutil /
  aws s3 / rclone / azcopy) with the exact commands consumers will run.
- "Why no cloud adapters in 8.0?" section documents the four reasons
  per BR-BRAINY-80-STORAGE-SIMPLIFY.
- Migration recipe for 7.x cloud-adapter consumers: mount local disk →
  filesystem storage → operator backup cron.

Updated frontmatter description so soulcraft.com/docs renders the
correct preview.

NOT IN THIS COMMIT (deliberate, lower-priority)

- src/storage/cacheManager.ts still references StorageType.S3 /
  REMOTE_API / OPFS as dead branches (23 sites). The branches are never
  reached in 8.0, but cleaning them would cascade through 5 consumers.
  Defer to a follow-up if the dead code surfaces as a real maintenance
  issue.
- src/config/storageAutoConfig.ts keeps its StorageType enum + autodetect
  for 7.x compat surface. Same reason: rewriting cascades through
  zeroConfig, extensibleConfig, sharedConfigManager. Defer.
- docs/MIGRATION-V3-TO-V4.md and docs/DEVELOPER_LEARNING_PATH.md still
  reference cloud adapters as historical artefacts. That's accurate —
  they describe how things used to be. Left as-is.
- @deprecated audit in src/ (10 files) deferred — audit each individually
  in a future polish pass.

VERIFICATION

- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding from
  step 7; no regressions from this cleanup)
This commit is contained in:
David Snelling 2026-06-09 15:05:02 -07:00
parent 780fb6444b
commit 9f9a41599e
13 changed files with 162 additions and 5288 deletions

View file

@ -628,105 +628,43 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
// Note: LSM-trees will be recreated from storage via their own initialization
// Verb data will be loaded on-demand via UnifiedCache
// Adaptive loading strategy based on storage type
// Brainy 8.0: storage is always local (filesystem or memory) per
// BR-BRAINY-80-STORAGE-SIMPLIFY. Load all verbs at once.
const storageType = this.storage?.constructor.name || ''
const isLocalStorage =
storageType === 'FileSystemStorage' ||
storageType === 'MemoryStorage' ||
storageType === 'OPFSStorage'
let totalVerbs = 0
if (isLocalStorage) {
// Local storage: Load all verbs at once to avoid repeated getAllShardedFiles() calls
prodLog.info(
`GraphAdjacencyIndex: Using optimized strategy - load all verbs at once (${storageType})`
)
prodLog.info(`GraphAdjacencyIndex: Load all verbs at once (${storageType})`)
const result = await this.storage.getVerbs({
pagination: { limit: 10000000 } // Effectively unlimited for local development
})
const result = await this.storage.getVerbs({
pagination: { limit: 10000000 } // Effectively unlimited for local storage
})
// Add each verb to index
for (const verb of result.items) {
// Convert HNSWVerbWithMetadata to GraphVerb format
const graphVerb: GraphVerb = {
id: verb.id,
sourceId: verb.sourceId,
targetId: verb.targetId,
vector: verb.vector,
source: verb.sourceId,
target: verb.targetId,
verb: verb.verb,
createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 },
updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 },
createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' },
service: verb.service,
data: verb.data,
embedding: verb.vector,
confidence: verb.confidence,
weight: verb.weight
}
await this.addVerb(graphVerb)
totalVerbs++
for (const verb of result.items) {
const graphVerb: GraphVerb = {
id: verb.id,
sourceId: verb.sourceId,
targetId: verb.targetId,
vector: verb.vector,
source: verb.sourceId,
target: verb.targetId,
verb: verb.verb,
createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 },
updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 },
createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' },
service: verb.service,
data: verb.data,
embedding: verb.vector,
confidence: verb.confidence,
weight: verb.weight
}
prodLog.info(
`GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs at once (local storage)`
)
} else {
// Cloud storage: Use pagination with native cloud APIs (efficient)
prodLog.info(
`GraphAdjacencyIndex: Using cloud pagination strategy (${storageType})`
)
let hasMore = true
let cursor: string | undefined = undefined
const batchSize = 1000
while (hasMore) {
const result = await this.storage.getVerbs({
pagination: { limit: batchSize, cursor }
})
// Add each verb to index
for (const verb of result.items) {
// Convert HNSWVerbWithMetadata to GraphVerb format
const graphVerb: GraphVerb = {
id: verb.id,
sourceId: verb.sourceId,
targetId: verb.targetId,
vector: verb.vector,
source: verb.sourceId,
target: verb.targetId,
verb: verb.verb,
createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 },
updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 },
createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' },
service: verb.service,
data: verb.data,
embedding: verb.vector,
confidence: verb.confidence,
weight: verb.weight
}
await this.addVerb(graphVerb)
totalVerbs++
}
hasMore = result.hasMore
cursor = result.nextCursor
// Progress logging
if (totalVerbs % 10000 === 0) {
prodLog.info(`GraphAdjacencyIndex: Indexed ${totalVerbs} verbs...`)
}
}
prodLog.info(
`GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs via pagination (cloud storage)`
)
await this.addVerb(graphVerb)
totalVerbs++
}
prodLog.info(
`GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs (${storageType})`
)
const rebuildTime = Date.now() - this.rebuildStartTime
const memoryUsage = this.calculateMemoryUsage()