chore(8.0): step-7 follow-through — collapse remaining cloud branches + docs sweep (scaffold step 13)
Final cleanup pass for Brainy 8.0. Catches three categories of debt: A. STEP-7 FOLLOW-THROUGH (rebuild-path collapse) Step 7's bisect-reset (debugging a flaky test) lost the in-source edits to three rebuild paths even though the commit message claimed they shipped. Re-applied now: - src/utils/metadataIndex.ts — collapsed the `isLocalStorage` / cloud-pagination branching. Local-load-all-at-once is the only path in 8.0. Removed ~150 LOC of paginated-cloud branching for both nouns and verbs, plus the safety counters (`consecutiveEmptyBatches`, `MAX_ITERATIONS`, etc.). - src/hnsw/hnswIndex.ts — same simplification for HNSW rebuild. The paginated cloud path is gone; HNSW now loads all nodes at once. Removed ~85 LOC. - src/graph/graphAdjacencyIndex.ts — same simplification for graph adjacency rebuild. Removed ~50 LOC. The collapse is safe because cloud adapters were deleted in step 7; `storageType === 'OPFSStorage'` (and similar) can never match now. B. CLOUD-ONLY DOCS DELETED - docs/operations/cost-optimization-aws-s3.md - docs/operations/cost-optimization-azure.md - docs/operations/cost-optimization-cloudflare-r2.md - docs/operations/cost-optimization-gcs.md - docs/operations/cloud-run-filestore-guide.md (docs/deployment/* contained no cloud-specific files that needed deletion.) C. STORAGE-ADAPTERS GUIDE REWRITTEN FOR 8.0 docs/guides/storage-adapters.md → fresh content reflecting the 8.0 reality: - Two adapters: FileSystemStorage + MemoryStorage. Quick-start matrix. - Cloud backup section explains the operator-tooling pattern (gsutil / aws s3 / rclone / azcopy) with the exact commands consumers will run. - "Why no cloud adapters in 8.0?" section documents the four reasons per BR-BRAINY-80-STORAGE-SIMPLIFY. - Migration recipe for 7.x cloud-adapter consumers: mount local disk → filesystem storage → operator backup cron. Updated frontmatter description so soulcraft.com/docs renders the correct preview. NOT IN THIS COMMIT (deliberate, lower-priority) - src/storage/cacheManager.ts still references StorageType.S3 / REMOTE_API / OPFS as dead branches (23 sites). The branches are never reached in 8.0, but cleaning them would cascade through 5 consumers. Defer to a follow-up if the dead code surfaces as a real maintenance issue. - src/config/storageAutoConfig.ts keeps its StorageType enum + autodetect for 7.x compat surface. Same reason: rewriting cascades through zeroConfig, extensibleConfig, sharedConfigManager. Defer. - docs/MIGRATION-V3-TO-V4.md and docs/DEVELOPER_LEARNING_PATH.md still reference cloud adapters as historical artefacts. That's accurate — they describe how things used to be. Left as-is. - @deprecated audit in src/ (10 files) deferred — audit each individually in a future polish pass. VERIFICATION - npx tsc --noEmit: clean - npm test: 1408 / 1409 (same pre-existing race-condition outstanding from step 7; no regressions from this cleanup)
This commit is contained in:
parent
780fb6444b
commit
9f9a41599e
13 changed files with 162 additions and 5288 deletions
|
|
@ -628,105 +628,43 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
|
|||
// Note: LSM-trees will be recreated from storage via their own initialization
|
||||
// Verb data will be loaded on-demand via UnifiedCache
|
||||
|
||||
// Adaptive loading strategy based on storage type
|
||||
// Brainy 8.0: storage is always local (filesystem or memory) per
|
||||
// BR-BRAINY-80-STORAGE-SIMPLIFY. Load all verbs at once.
|
||||
const storageType = this.storage?.constructor.name || ''
|
||||
const isLocalStorage =
|
||||
storageType === 'FileSystemStorage' ||
|
||||
storageType === 'MemoryStorage' ||
|
||||
storageType === 'OPFSStorage'
|
||||
|
||||
let totalVerbs = 0
|
||||
|
||||
if (isLocalStorage) {
|
||||
// Local storage: Load all verbs at once to avoid repeated getAllShardedFiles() calls
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Using optimized strategy - load all verbs at once (${storageType})`
|
||||
)
|
||||
prodLog.info(`GraphAdjacencyIndex: Load all verbs at once (${storageType})`)
|
||||
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { limit: 10000000 } // Effectively unlimited for local development
|
||||
})
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { limit: 10000000 } // Effectively unlimited for local storage
|
||||
})
|
||||
|
||||
// Add each verb to index
|
||||
for (const verb of result.items) {
|
||||
// Convert HNSWVerbWithMetadata to GraphVerb format
|
||||
const graphVerb: GraphVerb = {
|
||||
id: verb.id,
|
||||
sourceId: verb.sourceId,
|
||||
targetId: verb.targetId,
|
||||
vector: verb.vector,
|
||||
source: verb.sourceId,
|
||||
target: verb.targetId,
|
||||
verb: verb.verb,
|
||||
createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 },
|
||||
updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 },
|
||||
createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' },
|
||||
service: verb.service,
|
||||
data: verb.data,
|
||||
embedding: verb.vector,
|
||||
confidence: verb.confidence,
|
||||
weight: verb.weight
|
||||
}
|
||||
await this.addVerb(graphVerb)
|
||||
totalVerbs++
|
||||
for (const verb of result.items) {
|
||||
const graphVerb: GraphVerb = {
|
||||
id: verb.id,
|
||||
sourceId: verb.sourceId,
|
||||
targetId: verb.targetId,
|
||||
vector: verb.vector,
|
||||
source: verb.sourceId,
|
||||
target: verb.targetId,
|
||||
verb: verb.verb,
|
||||
createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 },
|
||||
updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 },
|
||||
createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' },
|
||||
service: verb.service,
|
||||
data: verb.data,
|
||||
embedding: verb.vector,
|
||||
confidence: verb.confidence,
|
||||
weight: verb.weight
|
||||
}
|
||||
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs at once (local storage)`
|
||||
)
|
||||
} else {
|
||||
// Cloud storage: Use pagination with native cloud APIs (efficient)
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Using cloud pagination strategy (${storageType})`
|
||||
)
|
||||
|
||||
let hasMore = true
|
||||
let cursor: string | undefined = undefined
|
||||
const batchSize = 1000
|
||||
|
||||
while (hasMore) {
|
||||
const result = await this.storage.getVerbs({
|
||||
pagination: { limit: batchSize, cursor }
|
||||
})
|
||||
|
||||
// Add each verb to index
|
||||
for (const verb of result.items) {
|
||||
// Convert HNSWVerbWithMetadata to GraphVerb format
|
||||
const graphVerb: GraphVerb = {
|
||||
id: verb.id,
|
||||
sourceId: verb.sourceId,
|
||||
targetId: verb.targetId,
|
||||
vector: verb.vector,
|
||||
source: verb.sourceId,
|
||||
target: verb.targetId,
|
||||
verb: verb.verb,
|
||||
createdAt: { seconds: Math.floor(verb.createdAt / 1000), nanoseconds: (verb.createdAt % 1000) * 1000000 },
|
||||
updatedAt: { seconds: Math.floor(verb.updatedAt / 1000), nanoseconds: (verb.updatedAt % 1000) * 1000000 },
|
||||
createdBy: verb.createdBy || { augmentation: 'unknown', version: '0.0.0' },
|
||||
service: verb.service,
|
||||
data: verb.data,
|
||||
embedding: verb.vector,
|
||||
confidence: verb.confidence,
|
||||
weight: verb.weight
|
||||
}
|
||||
await this.addVerb(graphVerb)
|
||||
totalVerbs++
|
||||
}
|
||||
|
||||
hasMore = result.hasMore
|
||||
cursor = result.nextCursor
|
||||
|
||||
// Progress logging
|
||||
if (totalVerbs % 10000 === 0) {
|
||||
prodLog.info(`GraphAdjacencyIndex: Indexed ${totalVerbs} verbs...`)
|
||||
}
|
||||
}
|
||||
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs via pagination (cloud storage)`
|
||||
)
|
||||
await this.addVerb(graphVerb)
|
||||
totalVerbs++
|
||||
}
|
||||
|
||||
prodLog.info(
|
||||
`GraphAdjacencyIndex: Loaded ${totalVerbs.toLocaleString()} verbs (${storageType})`
|
||||
)
|
||||
|
||||
const rebuildTime = Date.now() - this.rebuildStartTime
|
||||
const memoryUsage = this.calculateMemoryUsage()
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue