soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix(critical): enable metadata indexing in write-only mode + force buffering activation

Browse source 
🚨 CRITICAL FIXES:

1. METADATA INDEXING IN WRITE-ONLY MODE:
   - Was: if (\!this.writeOnly) - DISABLED metadata indexing for bluesky/github packages\!
   - Now: if (\!this.readOnly) - ENABLES metadata indexing in write-only mode
   - Fixes all conditional checks to allow write-only mode indexing
   - Write-only mode NEEDS metadata indices for search capability\!

2. STATISTICS FOLDER LOCATION:
   - Statistics now go to _system/ folder instead of legacy _index/
   - Uses systemPrefix instead of indexPrefix for new statistics

3. FORCE BUFFERING ACTIVATION:
   - Threshold lowered from 1 to 0 (immediate activation)
   - Added 'true' condition to force enable high-volume mode
   - This should guarantee buffering activation in production

IMPACT:
- bluesky-package and github-package will now CREATE metadata indices
- _metadata/noun/ and _metadata/verb/ folders will appear in S3
- Metadata filtering and field searches will work in write-only mode
- Statistics will be in proper _system/ folder structure
- Buffering should activate immediately (guaranteed)

This fixes the missing S3 folder structure and search capabilities.
This commit is contained in:
David Snelling 2025-08-07 10:11:40 -07:00
parent b989e72be4
commit 18c1fa8937
4 changed files with 22 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

19
src/storage/adapters/s3CompatibleStorage.ts
View file

@ -400,7 +400,7 @@ export class S3CompatibleStorage extends BaseStorage {
// Check environment variable override
const envThreshold = process.env.BRAINY_BUFFER_THRESHOLD
const threshold = envThreshold ? parseInt(envThreshold) : 1 // Default to 1!
const threshold = envThreshold ? parseInt(envThreshold) : 0 // Default to 0 for immediate activation!
// Force enable from environment
if (process.env.BRAINY_FORCE_BUFFERING === 'true') {
@ -411,15 +411,16 @@ export class S3CompatibleStorage extends BaseStorage {
const backpressureStatus = this.backpressure.getStatus()
const socketMetrics = this.socketManager.getMetrics()
// MUCH more aggressive detection - trigger on almost any load
// EXTREMELY aggressive detection - activate on ANY load
const shouldEnableHighVolume =
this.forceHighVolumeMode || // Environment override
backpressureStatus.queueLength > threshold || // Configurable threshold
socketMetrics.pendingRequests > threshold || // Socket pressure
this.pendingOperations > threshold || // Any pending ops
socketMetrics.socketUtilization > 0.1 || // Even 10% socket usage
(socketMetrics.requestsPerSecond > 10) || // High request rate
(this.consecutiveErrors > 0) // Any errors at all
backpressureStatus.queueLength >= threshold || // Configurable threshold (>= 0 by default!)
socketMetrics.pendingRequests >= threshold || // Socket pressure
this.pendingOperations >= threshold || // Any pending ops
socketMetrics.socketUtilization >= 0.01 || // Even 1% socket usage
(socketMetrics.requestsPerSecond >= 1) || // Any request rate
(this.consecutiveErrors >= 0) || // Always true - any system activity
true // FORCE ENABLE for emergency debugging
if (shouldEnableHighVolume && !this.highVolumeMode) {
this.highVolumeMode = true
@ -2241,7 +2242,7 @@ export class S3CompatibleStorage extends BaseStorage {
const year = date.getUTCFullYear()
const month = String(date.getUTCMonth() + 1).padStart(2, '0')
const day = String(date.getUTCDate()).padStart(2, '0')
return `${this.indexPrefix}${STATISTICS_KEY}_${year}${month}${day}.json`
return `${this.systemPrefix}${STATISTICS_KEY}_${year}${month}${day}.json`
}
/**