soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

335 releases 524 tags

RSS feed
  • v3.50.2 0f57ee6bb4

    v3.50.2 - Emergency Hotfix: Complete Metadata Explosion Fix Stable

    dpsifr released this 2025-10-17 01:31:10 +02:00 | 759 commits to main since this release

    🚨 Emergency Hotfix for v3.50.1

    Critical fix for incomplete v3.50.1 release - numeric field names still being indexed

    Problem

    v3.50.1 prevented vector fields by name ('vector', 'embedding') but missed vectors stored as objects with numeric keys.

    Studio Team Diagnostics Showed:

    • 212,531 chunk files with NUMERIC field names still being created
    • Files like: __chunk__54716_0.json with "field": "54716"
    • 424,837 total files (expected ~1,200)
    • Root cause: Vectors stored as objects {0: 0.1, 1: 0.2, ...} bypassed v3.50.1's field name check

    Solution

    Added regex check in extractIndexableFields():

    if (/^\d+$/.test(key)) continue  // Skip purely numeric field names

    This catches:

    • Array indices as object keys: "0", "1", "2", "100"
    • High-dimensional indices: "54716", "100000", "100001"
    • Works with v3.50.1's semantic checks: 'vector', 'embedding', 'embeddings'

    Impact

    • ✅ File reduction: 424,837 → ~1,200 files (354x reduction)
    • ✅ Prevents 212K+ chunk files from being created
    • ✅ Fixes server hangs during initialization
    • ✅ Completes the fix started in v3.50.1

    Test Results

    • ✅ Added new test: "should NOT index objects with numeric keys (v3.50.2 fix)"
    • ✅ 8/8 integration tests passing
    • ✅ Verifies NO chunk files have numeric field names

    Migration for Studio Team

    After upgrading to v3.50.2:

    1. Delete corrupted chunk files:

      cd brainy-data
rm -rf _system/

    2. Restart server - metadata index will rebuild correctly

    3. Verify file count:

      find brainy-data -type f | wc -l
# Should be ~1,200 (was 424,837)

    4. Verify no numeric field names:

      find _system -name '__chunk__*' -exec grep -l '"field":"[0-9]\+' {} \; | wc -l
# Should be 0 (was 212,531)

    Files Modified

    • src/utils/metadataIndex.ts (line 1106) - Added numeric field name exclusion
    • tests/integration/metadata-vector-exclusion.test.ts - Added v3.50.2 test case

    Install

    npm install @soulcraft/brainy@3.50.2

    Thank you to the Soulcraft Studio team for the detailed diagnostics that made this fix possible!

    Downloads