soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

test(8.0): integration rot pass — 77→17 failures (parallel per-file hardening)

Browse source 
Cleared ~60 rotted-test failures across 17 integration files: get() now passes
{includeVectors:true} where the vector is used; close() teardown added (cures
heartbeat-bleed timeouts); removed-API call-sites rewritten to the 8.0 surface
(addRelationship→relate, COW internals dropped); Result/Entity shape assertions
updated; deterministic-embedder semantic assertions rewritten as self-retrieval
(or moved to Tier-2 where irreducible); perf thresholds relaxed; 384-dim fixtures.

Adds the Tier-2 (real-model) scaffolding: tests/setup-semantic.ts +
tests/configs/vitest.semantic.config.ts + test:semantic; test:ci now runs
unit+integration (anti-rot gate, goes live once green).

Remaining 17 failures are REAL 8.0 library bugs the pass surfaced (fixed next,
not papered over): dual-bound where-filter dropping a bound; counts not
rehydrating after restart; related() offset pagination; relate() non-idempotent
updatedAt; unscoped VFS path-cache. Plus find-unified finish + a few stragglers.
This commit is contained in:
David Snelling 2026-06-17 13:11:41 -07:00
parent c600468bb5
commit e5997a1516
20 changed files with 1187 additions and 789 deletions
Download patch file Download diff file Expand all files Collapse all files

272
tests/integration/metadata-vector-exclusion.test.ts
View file

@ -1,18 +1,31 @@
/**
* Integration test for metadata explosion fix (v3.50.1)
* Integration test for metadata explosion fix
*
* Validates that vector embeddings are NEVER indexed in metadata,
* while preserving legitimate small array indexing (tags, categories).
* Validates that vector embeddings (and array-index-as-object-key payloads) are
* NEVER indexed as metadata fields, while legitimate small arrays and scalar
* fields ARE registered for `where` filtering.
*
* Bug: 825,924 chunk files created for 1,144 entities (721 files per entity)
* Fix: NEVER_INDEX field name check + array length safety check
* Original bug class: 825,924 chunk files created for 1,144 entities (721 files
* per entity) because each of the 384 vector dimensions was being indexed as its
* own numeric metadata field.
*
* 8.0 architecture note: the per-vector-dimension chunk explosion is structurally
* impossible now — the metadata index is a column store + sparse field index
* (`_column_index/<field>/…` and `_system/idx/**`), not one `__chunk__*` file per
* field-value. So this suite no longer counts `_system/__chunk__*` files (there
* are none). Instead it asserts the real invariant directly via the public
* `getAvailableFields()` surface: no numeric (vector-dimension) field names, no
* `vector`/`embedding` field, but the legitimate semantic fields ARE indexed.
*
* Field placement (8.0 contract — see AddParams JSDoc): `data` is the content that
* gets embedded; `metadata` holds the structured, queryable fields used in `where`
* filters. Queryable fields therefore live under `metadata` here.
*/
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { Brainy } from '../../src/brainy.js'
import { NounType } from '../../src/types/graphTypes.js'
import { readFileSync, readdirSync, existsSync, rmSync } from 'fs'
import { join } from 'path'
import { existsSync, rmSync } from 'fs'
describe('Metadata Vector Exclusion Fix', () => {
let brainy: Brainy
@ -41,54 +54,50 @@ describe('Metadata Vector Exclusion Fix', () => {
}
})
it('should NOT index vector embeddings in metadata chunks', async () => {
// Add entity with vector embedding
const entity = await brainy.add({
type: 'person' as any,
data: {
it('should NOT index vector embeddings as metadata fields', async () => {
// Add entity with queryable metadata + a small array (tags SHOULD be indexed)
await brainy.add({
type: NounType.Person,
data: 'Alice the developer',
metadata: {
name: 'Alice',
email: 'alice@example.com',
tags: ['developer', 'typescript'] // Small array - SHOULD be indexed
}
})
// Wait for async operations
await new Promise(resolve => setTimeout(resolve, 100))
// The indexed-field set is the canonical record of what the metadata index
// actually tracks. The vector explosion bug manifested as hundreds of numeric
// field names (one per dimension) appearing here.
const fields = await brainy.getAvailableFields()
// Check _system directory for chunk files
const systemDir = join(testDir, '_system')
// CRITICAL: no purely-numeric field names (vector dimension indices like
// "0", "1", "54716") and no nested numeric dimension keys.
const numericFields = fields.filter(f => /(^|\.)\d+$/.test(f))
expect(numericFields).toEqual([])
// No _system dir means no chunk files at all — the invariant below
// (no vector-dimension chunks) holds trivially on an empty list.
const chunkFiles = existsSync(systemDir)
? readdirSync(systemDir).filter(f => f.startsWith('__chunk__'))
: []
// CRITICAL: the raw vector must never be indexed as a metadata field.
expect(fields).not.toContain('vector')
expect(fields).not.toContain('embedding')
expect(fields).not.toContain('embeddings')
// Should have at most a few chunk files (name, email, tags)
// NOT hundreds of files from vector dimensions
expect(chunkFiles.length).toBeLessThan(10)
// The legitimate semantic fields ARE indexed.
expect(fields).toContain('name')
expect(fields).toContain('email')
expect(fields).toContain('tags')
// Verify NO chunk files have numeric field names (vector dimension indices)
for (const file of chunkFiles) {
const content = JSON.parse(readFileSync(join(systemDir, file), 'utf-8'))
const fieldName = content.field
// CRITICAL: Field should be a semantic name, NOT a number
// This catches both "vector" fields AND numeric keys like "0", "1", "54716"
expect(fieldName).not.toMatch(/^\d+$/)
// Field should NOT be 'vector', 'embedding', 'embeddings'
expect(fieldName).not.toBe('vector')
expect(fieldName).not.toBe('embedding')
expect(fieldName).not.toBe('embeddings')
}
// Field count stays sane (semantic fields + Brainy's own system/VFS fields),
// nowhere near the hundreds a per-dimension explosion would produce.
expect(fields.length).toBeLessThan(50)
})
it('should NOT index objects with numeric keys (v3.50.2 fix)', async () => {
// Add entity with object that has numeric keys (simulates vector-as-object)
it('should NOT index objects with numeric keys (array-as-object guard)', async () => {
// Add entity with an object that has numeric keys (simulates a vector
// accidentally serialized as { "0": 0.1, "1": 0.2, ... }).
await brainy.add({
type: 'person' as any,
data: {
type: NounType.Person,
data: 'NumericTest entity',
metadata: {
name: 'NumericTest',
numericObject: {
'0': 0.1,
@ -99,44 +108,45 @@ describe('Metadata Vector Exclusion Fix', () => {
}
})
await new Promise(resolve => setTimeout(resolve, 100))
const fields = await brainy.getAvailableFields()
const systemDir = join(testDir, '_system')
// CRITICAL: numeric keys (whether top-level or nested under a field like
// "numericObject.0") must NOT become indexed fields. This is the guard that
// prevents vectors-as-objects from exploding the index.
const numericFields = fields.filter(f => /(^|\.)\d+$/.test(f))
expect(numericFields).toEqual([])
// No _system dir means no chunk files — the assertions below hold on [].
const chunkFiles = existsSync(systemDir)
? readdirSync(systemDir).filter(f => f.startsWith('__chunk__'))
: []
// CRITICAL: Check that NO chunk files have numeric field names
// This is the v3.50.2 fix - prevents vectors-as-objects from being indexed
for (const file of chunkFiles) {
const content = JSON.parse(readFileSync(join(systemDir, file), 'utf-8'))
const fieldName = content.field
// Should NOT index purely numeric field names (array indices as object keys)
// This catches: "0", "1", "2", "100", "54716", "100000", etc.
expect(fieldName).not.toMatch(/^\d+$/)
}
// Verify we DID create chunk files for legitimate fields
expect(chunkFiles.length).toBeGreaterThan(0)
// The legitimate scalar field IS indexed.
expect(fields).toContain('name')
})
it('should still index small arrays (tags, categories)', async () => {
// Add entity with tags
// Add entity with a small (<= 10 element) array of tags.
await brainy.add({
type: 'person' as any,
data: {
type: NounType.Person,
data: 'Bob the engineer',
metadata: {
name: 'Bob',
tags: ['javascript', 'react', 'nodejs']
}
})
// Wait for indexing
await new Promise(resolve => setTimeout(resolve, 100))
// Small arrays must be registered as a multi-value field (not skipped, not
// exploded). 'tags' should appear in the indexed-field set.
const fields = await brainy.getAvailableFields()
expect(fields).toContain('tags')
// Verify metadata filtering works on tags
// Behavioral check: filtering by a tag value should return the entity. The
// array was ['javascript', 'react', 'nodejs'] — matching ANY member must work.
//
// REAL LIBRARY BUG (left failing on purpose): the ColumnStore ingest path in
// src/utils/metadataIndex.ts (addToIndex, ~line 1407) does
// `fieldsMap[field] = value` for every extracted pair, so for a multi-value
// field it keeps only the LAST array element ('nodejs') and drops the rest.
// Only the '__words__' field gets the array-accumulation treatment. As a
// result `where: { tags: 'react' }` (and 'javascript') returns [], while
// `where: { tags: 'nodejs' }` returns 1. Correct input → wrong output, so
// this assertion stays as-is to keep the bug visible.
const results = await brainy.find({
where: { tags: 'react' }
})
@ -146,71 +156,75 @@ describe('Metadata Vector Exclusion Fix', () => {
})
it('should skip indexing large arrays (>10 elements)', async () => {
// Add entity with large array (not a vector, just bulk data)
// Add entity with a large array (not a vector, just bulk data).
const largeArray = Array.from({ length: 100 }, (_, i) => `item${i}`)
await brainy.add({
type: 'document' as any,
data: {
type: NounType.Document,
data: 'Doc with large array',
metadata: {
name: 'Doc with large array',
items: largeArray
}
})
// Wait for indexing
await new Promise(resolve => setTimeout(resolve, 100))
// Large arrays (> 10 elements) are deliberately skipped to avoid indexing
// bulk/vector-like payloads: 'items' must NOT appear, and the 100 elements
// must NOT have produced 100 indexed fields.
const fields = await brainy.getAvailableFields()
expect(fields).not.toContain('items')
const numericFields = fields.filter(f => /(^|\.)\d+$/.test(f))
expect(numericFields).toEqual([])
// Check chunk count - should NOT create 100 chunk files
const systemDir = join(testDir, '_system')
// No _system dir means no chunk files — the assertions below hold on [].
const chunkFiles = existsSync(systemDir)
? readdirSync(systemDir).filter(f => f.startsWith('__chunk__'))
: []
// Should have minimal chunk files (just 'name' field)
expect(chunkFiles.length).toBeLessThan(5)
// The scalar 'name' field IS indexed.
expect(fields).toContain('name')
})
it('should preserve HNSW vector search functionality', async () => {
// Add entities with semantic content
const id1 = await brainy.add({
type: 'concept' as any,
data: {
type: NounType.Concept,
data: 'AI algorithms that learn from data',
metadata: {
name: 'Machine Learning',
description: 'AI algorithms that learn from data'
}
})
const id2 = await brainy.add({
type: 'concept' as any,
data: {
type: NounType.Concept,
data: 'Neural networks with multiple layers',
metadata: {
name: 'Deep Learning',
description: 'Neural networks with multiple layers'
}
})
// Wait for vector indexing
await new Promise(resolve => setTimeout(resolve, 200))
// Verify entities were created (vector indexing happened)
const entity1 = await brainy.get(id1)
const entity2 = await brainy.get(id2)
// Vectors are metadata-only by default; request them explicitly to verify the
// embedding pipeline actually produced a stored vector for each entity.
const entity1 = await brainy.get(id1, { includeVectors: true })
const entity2 = await brainy.get(id2, { includeVectors: true })
expect(entity1).toBeDefined()
expect(entity2).toBeDefined()
expect(entity1?.vector).toBeDefined()
expect(entity2?.vector).toBeDefined()
expect(Array.isArray(entity1?.vector)).toBe(true)
expect(Array.isArray(entity2?.vector)).toBe(true)
expect(entity1!.vector!.length).toBe(384)
expect(entity2!.vector!.length).toBe(384)
// Verify vectors are not in metadata chunks (already validated by first test)
// Mock AI may not support semantic search, so we just verify vectors exist
// Self-retrieval confirms the HNSW index is wired: querying an entity by its
// OWN content text returns it (deterministic embedder → cosine 1.0).
const selfHits = await brainy.find({ query: 'AI algorithms that learn from data', limit: 5 })
expect(selfHits.length).toBeGreaterThan(0)
expect(selfHits.some(r => r.id === id1)).toBe(true)
})
it('should preserve metadata field filtering', async () => {
// Add entities with various metadata
// Add entities with various queryable metadata
await brainy.add({
type: 'person' as any,
data: {
type: NounType.Person,
data: 'Charlie the engineer',
metadata: {
name: 'Charlie',
email: 'charlie@example.com',
role: 'engineer'
@ -218,18 +232,16 @@ describe('Metadata Vector Exclusion Fix', () => {
})
await brainy.add({
type: 'person' as any,
data: {
type: NounType.Person,
data: 'Dana the designer',
metadata: {
name: 'Dana',
email: 'dana@example.com',
role: 'designer'
}
})
// Wait for indexing
await new Promise(resolve => setTimeout(resolve, 100))
// Verify metadata filtering works
// Verify scalar metadata filtering works
const engineers = await brainy.find({
where: { role: 'engineer' }
})
@ -248,8 +260,9 @@ describe('Metadata Vector Exclusion Fix', () => {
it('should handle nested object metadata correctly', async () => {
// Add entity with nested metadata
await brainy.add({
type: 'person' as any,
data: {
type: NounType.Person,
data: 'Eve in New York',
metadata: {
name: 'Eve',
address: {
city: 'New York',
@ -258,10 +271,8 @@ describe('Metadata Vector Exclusion Fix', () => {
}
})
// Wait for indexing
await new Promise(resolve => setTimeout(resolve, 100))
// Verify nested field filtering works
// Nested fields are flattened to dot-notation field names; filter by the
// flattened path.
const results = await brainy.find({
where: { 'address.city': 'New York' }
})
@ -270,12 +281,13 @@ describe('Metadata Vector Exclusion Fix', () => {
expect(results[0].entity.metadata?.name).toBe('Eve')
})
it('should NOT create exponential chunk files for multiple entities', async () => {
// Add 10 entities (each with vector embedding)
it('should NOT create exponential index fields for multiple entities', async () => {
// Add 10 entities (each with a vector embedding + a few metadata fields)
for (let i = 0; i < 10; i++) {
await brainy.add({
type: 'person' as any,
data: {
type: NounType.Person,
data: `Person ${i}`,
metadata: {
name: `Person ${i}`,
email: `person${i}@example.com`,
tags: ['user']
@ -283,21 +295,17 @@ describe('Metadata Vector Exclusion Fix', () => {
})
}
// Wait for all indexing
await new Promise(resolve => setTimeout(resolve, 500))
// Check total chunk files
const systemDir = join(testDir, '_system')
// No _system dir means no chunk files — the assertions below hold on [].
const chunkFiles = existsSync(systemDir)
? readdirSync(systemDir).filter(f => f.startsWith('__chunk__'))
: []
// Should have reasonable number of chunks (not 7,210 for 10 entities!)
// Expected: ~30 chunks (name, email, tags fields across 10 entities)
expect(chunkFiles.length).toBeLessThan(100)
console.log(`✅ Created ${chunkFiles.length} chunk files for 10 entities (expected <100)`)
// The indexed-field set is keyed by field NAME, not by entity — so it must
// stay bounded regardless of entity count. The original bug turned this into
// ~721 fields per entity (one per vector dimension); here it must remain a
// small handful of semantic fields plus Brainy's own system/VFS fields.
const fields = await brainy.getAvailableFields()
const numericFields = fields.filter(f => /(^|\.)\d+$/.test(f))
expect(numericFields).toEqual([])
expect(fields).toContain('name')
expect(fields).toContain('email')
expect(fields).toContain('tags')
// Nowhere near the thousands a per-dimension explosion would produce.
expect(fields.length).toBeLessThan(50)
})
})