Compare commits

...

2 commits

Author SHA1 Message Date
d7444ae804 test(metadata): the three large-metadata cases pin the bound, not a magic length
Some checks are pending
CI / Node 24 (push) Waiting to run
CI / Node 22 (push) Waiting to run
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
get(), relate() and update() each carried a "very large metadata" case that
parked an array of 1000 (or 100) elements in the metadata bag and asserted it
came back. The indexable-array bound refuses that shape at the write door now
— an array field mints one posting per element, so an unbounded array is an
unbounded write — and the three cases were failing on the refusal they should
have been pinning.

Each is rewritten to the law that replaced it, in two halves:

  - a large SCALAR payload still round-trips whole through the door: a
    10,000-character string, 100 sibling fields, a ten-deep nest walked to the
    bottom, and an array sitting exactly ON the bound, checked first element
    to last;
  - an array one element OVER the bound refuses with MetadataArrayTooLargeError
    carrying the field, the length and the bound, on the error object AND in
    the message. update()'s refusal additionally proves the row is unchanged,
    and relate()'s that no relation was written — refused means not written,
    not written-then-skipped.

Every length is derived from the imported MAX_INDEXED_ARRAY_LENGTH; none is
typed as a number. That is what made the old cases fragile: 100 read as "over
the bound" and 1000 as "large", and both meanings changed under them when the
constant moved. These follow the constant instead.
2026-09-02 15:41:21 -07:00
e435da787d fix(metadata): the indexable-array bound is 256 — a keyword list is not a vector
64 cleared tags, authors and labels, but not the shape that actually turns up
in production metadata: a long keyword or participant list. 256 clears those
and still refuses every embedding this engine will ever meet — the narrowest
model it ships is 384-dimensional, so the two populations still do not overlap
and nobody has to tune anything. A vector parked in metadata throws by name;
a 200-keyword list writes and indexes.

The number lives in ONE place, `MAX_INDEXED_ARRAY_LENGTH`, and every message,
warning and pin derives it from there. Two pins still carried a literal:
metadata-vector-exclusion refused an array of exactly 100 — which sits UNDER
the new bound, so the case would have asserted a refusal that no longer
happens — and the array-bound suite named "all 64 elements" in a title and
picked its middle element as a hardcoded 't31'. Both derive from the constant
now, so the pins follow it wherever it goes rather than silently inverting the
next time it moves.
2026-09-02 15:41:11 -07:00
6 changed files with 184 additions and 37 deletions

View file

@ -412,18 +412,23 @@ export class MigrationInProgressError extends BrainyError {
* embedding parked in the metadata bag would mint 384 postings for one row.
* The bound exists to keep that out of the index.
*
* 64 is hardcoded on purpose (the zero-config law: no knob). It sits far above
* 256 is hardcoded on purpose (the zero-config law: no knob). It sits far above
* every legitimate multi-value field the engine has seen tags, authors,
* categories, labels, participant lists and far below any real embedding
* width, so the two populations do not overlap and no caller has to tune it.
* categories, labels, keyword lists, participant lists and still below the
* narrowest embedding this engine will ever meet (384 dimensions, the smallest
* model it ships), so the two populations do not overlap and no caller has to
* tune it. A vector parked in metadata is refused; a long keyword list is not.
*
* It replaces a limit of 10 that was applied SILENTLY: a row whose `tags` array
* held eleven entries had that field skipped entirely and dropped out of every
* filtered search on it, with no error, no warning and no way to tell the
* difference from "no row matches". A rule this consequential is a law with a
* name and a refusal, not a `continue`.
*
* This is the ONE place the number lives. Every message, warning, doc line and
* pin derives it from here never a literal.
*/
export const MAX_INDEXED_ARRAY_LENGTH = 64
export const MAX_INDEXED_ARRAY_LENGTH = 256
/**
* A metadata field carries an array longer than {@link MAX_INDEXED_ARRAY_LENGTH}.

View file

@ -161,7 +161,8 @@ describe('Metadata Vector Exclusion Fix', () => {
// silence at a bound of 10 — the field simply vanished from the index and
// the row dropped out of every `where` on it, indistinguishably from "no
// row matches". The bound is now MAX_INDEXED_ARRAY_LENGTH and it REFUSES.
const largeArray = Array.from({ length: 100 }, (_, i) => `item${i}`)
const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1
const largeArray = Array.from({ length: overTheBound }, (_, i) => `item${i}`)
const err = await brainy
.add({
@ -176,7 +177,7 @@ describe('Metadata Vector Exclusion Fix', () => {
expect(err).toBeInstanceOf(MetadataArrayTooLargeError)
expect(err.field).toBe('items')
expect(err.length).toBe(100)
expect(err.length).toBe(overTheBound)
expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH)
// Nothing was indexed from the refused write — no 'items' field, and above

View file

@ -5,7 +5,8 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { Brainy } from '../../../src/brainy'
import {
import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../../src/errors/brainyError'
import {
createAddParams,
generateTestVector,
createTestConfig,
@ -268,32 +269,75 @@ describe('Brainy.get()', () => {
expect(entity!.id).toBe(id)
})
it('should get entity with very large metadata', async () => {
// Arrange
// THE INDEXABLE-ARRAY BOUND, from get()'s side. This case used to park a
// 1000-element array in the metadata bag and assert it came back. That
// shape is refused at the write door now — an array field mints one
// posting per element, so an unbounded array is an unbounded write — so
// the case pins BOTH halves of the law that replaced it: a large SCALAR
// payload still round-trips whole, and an array over the bound refuses by
// name. Every length derives from MAX_INDEXED_ARRAY_LENGTH so the pin
// follows the constant wherever it moves.
it('should get an entity with a large scalar metadata payload', async () => {
// Arrange — large in every dimension EXCEPT array length: a long string,
// many fields, deep nesting, and an array sitting exactly ON the bound.
const largeMetadata = {
bigArray: new Array(1000).fill('item'),
atTheBound: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`),
bigObject: Object.fromEntries(
Array.from({ length: 100 }, (_, i) => [`key${i}`, `value${i}`])
),
longString: 'x'.repeat(10_000),
deepNesting: Array(10).fill(null).reduce(
(acc) => ({ nested: acc }),
{ value: 'deep' }
)
}
const id = await brain.add(createAddParams({
data: 'Large metadata',
type: 'thing',
metadata: largeMetadata
}))
// Act
const entity = await brain.get(id)
// Assert
// Assert — the payload comes back whole, first element to last
expect(entity).not.toBeNull()
expect(entity!.metadata.bigArray).toHaveLength(1000)
expect(entity!.metadata.atTheBound).toHaveLength(MAX_INDEXED_ARRAY_LENGTH)
expect(entity!.metadata.atTheBound[0]).toBe('item0')
expect(entity!.metadata.atTheBound[MAX_INDEXED_ARRAY_LENGTH - 1])
.toBe(`item${MAX_INDEXED_ARRAY_LENGTH - 1}`)
expect(Object.keys(entity!.metadata.bigObject)).toHaveLength(100)
expect(entity!.metadata.longString).toHaveLength(10_000)
// ...including the deep nest, walked to the bottom.
let cursor: any = entity!.metadata.deepNesting
for (let depth = 0; depth < 10; depth++) cursor = cursor.nested
expect(cursor.value).toBe('deep')
})
it('should refuse a metadata array over the indexing bound, by name', async () => {
// Arrange
const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1
// Act
const err = await brain
.add(createAddParams({
data: 'Large metadata',
type: 'thing',
metadata: { bigArray: new Array(overTheBound).fill('item') }
}))
.catch((e: any) => e)
// Assert — the field, the length and the bound, on the error and in the
// message, so a handler can report or repair without parsing prose.
expect(err).toBeInstanceOf(MetadataArrayTooLargeError)
expect(err.field).toBe('bigArray')
expect(err.length).toBe(overTheBound)
expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH)
expect(err.message).toContain('bigArray')
expect(err.message).toContain(String(overTheBound))
expect(err.message).toContain(String(MAX_INDEXED_ARRAY_LENGTH))
})
})

View file

@ -5,7 +5,8 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { Brainy } from '../../../src/brainy'
import {
import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../../src/errors/brainyError'
import {
createAddParams,
createTestConfig,
} from '../../helpers/test-factory'
@ -248,16 +249,23 @@ describe('Brainy.relate()', () => {
expect(matches.length).toBe(1) // Only one relationship should exist
})
it('should handle very long metadata', async () => {
// Arrange
// THE INDEXABLE-ARRAY BOUND, from relate()'s side. This case used to pass a
// 100-element array through relate() and assert it came back — a length
// hardcoded either side of a bound it never named, so it read green or red
// purely by where the constant happened to sit. Both halves of the law are
// pinned here instead, and every length derives from
// MAX_INDEXED_ARRAY_LENGTH so the pin follows the constant.
it('should handle a large scalar metadata payload on a relation', async () => {
// Arrange — large in every dimension EXCEPT array length: a long string,
// many fields, and an array sitting exactly ON the bound.
const largeMetadata = {
bigArray: new Array(100).fill('item'),
atTheBound: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`),
bigObject: Object.fromEntries(
Array.from({ length: 50 }, (_, i) => [`key${i}`, `value${i}`])
),
longString: 'x'.repeat(1000)
longString: 'x'.repeat(10_000)
}
// Act
await brain.relate({
from: entity1Id,
@ -265,12 +273,46 @@ describe('Brainy.relate()', () => {
type: 'relatedTo',
metadata: largeMetadata
})
// Assert
// Assert — the payload comes back whole, first element to last
const relations = await brain.related({ from: entity1Id })
const relation = relations.find(r => r.to === entity2Id)
expect(relation).toBeDefined()
expect(relation!.metadata?.bigArray).toHaveLength(100)
expect(relation!.metadata?.atTheBound).toHaveLength(MAX_INDEXED_ARRAY_LENGTH)
expect(relation!.metadata?.atTheBound[0]).toBe('item0')
expect(relation!.metadata?.atTheBound[MAX_INDEXED_ARRAY_LENGTH - 1])
.toBe(`item${MAX_INDEXED_ARRAY_LENGTH - 1}`)
expect(Object.keys(relation!.metadata?.bigObject)).toHaveLength(50)
expect(relation!.metadata?.longString).toHaveLength(10_000)
})
it('should refuse a relation metadata array over the indexing bound, by name', async () => {
// Arrange
const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1
// Act
const err = await brain
.relate({
from: entity1Id,
to: entity3Id,
type: 'relatedTo',
metadata: { bigArray: new Array(overTheBound).fill('item') }
})
.catch((e: any) => e)
// Assert — the field, the length and the bound, on the error and in the
// message, so a handler can report or repair without parsing prose.
expect(err).toBeInstanceOf(MetadataArrayTooLargeError)
expect(err.field).toBe('bigArray')
expect(err.length).toBe(overTheBound)
expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH)
expect(err.message).toContain('bigArray')
expect(err.message).toContain(String(overTheBound))
expect(err.message).toContain(String(MAX_INDEXED_ARRAY_LENGTH))
// Refused means not written: no relation of this shape exists.
const relations = await brain.related({ from: entity1Id })
expect(relations.some(r => r.to === entity3Id && r.metadata?.bigArray)).toBe(false)
})
it('should handle special characters in metadata', async () => {

View file

@ -5,7 +5,8 @@
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { Brainy } from '../../../src/brainy'
import {
import { MetadataArrayTooLargeError, MAX_INDEXED_ARRAY_LENGTH } from '../../../src/errors/brainyError'
import {
createAddParams,
createTestConfig,
} from '../../helpers/test-factory'
@ -355,36 +356,88 @@ describe('Brainy.update()', () => {
expect(final!.metadata.counter).toBeLessThanOrEqual(10)
})
it('should handle very large metadata updates', async () => {
// THE INDEXABLE-ARRAY BOUND, from update()'s side. This case used to write
// a 1000-element array through update() and assert it came back. That
// shape is refused at the write door now — an array field mints one
// posting per element, so an unbounded array is an unbounded write — so
// the case pins BOTH halves of the law that replaced it. Every length
// derives from MAX_INDEXED_ARRAY_LENGTH so the pin follows the constant.
it('should handle a large scalar metadata update', async () => {
// Arrange
const id = await brain.add(createAddParams({
data: 'Large metadata test',
type: 'thing'
}))
// Large in every dimension EXCEPT array length: a long string, many
// fields, deep nesting, and an array sitting exactly ON the bound.
const largeMetadata = {
bigArray: new Array(1000).fill('item'),
atTheBound: Array.from({ length: MAX_INDEXED_ARRAY_LENGTH }, (_, i) => `item${i}`),
bigObject: Object.fromEntries(
Array.from({ length: 100 }, (_, i) => [`key${i}`, `value${i}`])
),
longString: 'x'.repeat(10_000),
deepNesting: Array(10).fill(null).reduce(
(acc) => ({ nested: acc }),
{ value: 'deep' }
)
}
// Act
await brain.update({
id,
metadata: largeMetadata,
merge: false
})
// Assert
// Assert — the payload comes back whole, first element to last
const updated = await brain.get(id)
expect(updated).not.toBeNull()
expect(updated!.metadata.bigArray).toHaveLength(1000)
expect(updated!.metadata.atTheBound).toHaveLength(MAX_INDEXED_ARRAY_LENGTH)
expect(updated!.metadata.atTheBound[0]).toBe('item0')
expect(updated!.metadata.atTheBound[MAX_INDEXED_ARRAY_LENGTH - 1])
.toBe(`item${MAX_INDEXED_ARRAY_LENGTH - 1}`)
expect(Object.keys(updated!.metadata.bigObject)).toHaveLength(100)
expect(updated!.metadata.longString).toHaveLength(10_000)
// ...including the deep nest, walked to the bottom.
let cursor: any = updated!.metadata.deepNesting
for (let depth = 0; depth < 10; depth++) cursor = cursor.nested
expect(cursor.value).toBe('deep')
})
it('should refuse an update whose metadata array is over the indexing bound, by name', async () => {
// Arrange
const id = await brain.add(createAddParams({
data: 'Large metadata test',
type: 'thing',
metadata: { keep: 'me' }
}))
const overTheBound = MAX_INDEXED_ARRAY_LENGTH + 1
// Act
const err = await brain
.update({
id,
metadata: { bigArray: new Array(overTheBound).fill('item') },
merge: false
})
.catch((e: any) => e)
// Assert — the field, the length and the bound, on the error and in the
// message, so a handler can report or repair without parsing prose.
expect(err).toBeInstanceOf(MetadataArrayTooLargeError)
expect(err.field).toBe('bigArray')
expect(err.length).toBe(overTheBound)
expect(err.limit).toBe(MAX_INDEXED_ARRAY_LENGTH)
expect(err.message).toContain('bigArray')
expect(err.message).toContain(String(overTheBound))
expect(err.message).toContain(String(MAX_INDEXED_ARRAY_LENGTH))
// Refused means unchanged: the row still carries what it had before.
const unchanged = await brain.get(id)
expect(unchanged!.metadata.keep).toBe('me')
expect(unchanged!.metadata.bigArray).toBeUndefined()
})
it('should preserve entity ID during update', async () => {

View file

@ -15,9 +15,10 @@
* and the caller had no way to tell that from "no row matches". Eleven tags is
* not an exotic shape; the eleventh tag made the row invisible.
*
* THE LAW. Arrays of scalars index up to {@link MAX_INDEXED_ARRAY_LENGTH} = 64,
* THE LAW. Arrays of scalars index up to {@link MAX_INDEXED_ARRAY_LENGTH},
* hardcoded (the zero-config law: no knob), which clears every legitimate
* multi-value field and stays far below any embedding width. Above it the WRITE
* multi-value field tags, authors, keyword lists and stays below the
* narrowest embedding this engine meets (384 dimensions). Above it the WRITE
* IS REFUSED by name `MetadataArrayTooLargeError`, carrying the field, the
* length and the bound at `add`, `update`, `relate` and `updateRelation`
* alike. Nothing is skipped in silence.
@ -65,7 +66,7 @@ describe('the indexable-array bound', () => {
}
})
it('indexes right up to the bound — all 64 elements', async () => {
it('indexes right up to the bound — every element of it', async () => {
await brain.add({
id: 'at-bound',
data: 'a row at the bound',
@ -74,8 +75,9 @@ describe('the indexable-array bound', () => {
vector: []
})
// The first, the last, and one in the middle.
for (const tag of ['t0', `t${MAX_INDEXED_ARRAY_LENGTH - 1}`, 't31']) {
// The first, the last, and one in the middle — all derived from the
// bound, so the case follows the constant wherever it moves.
for (const tag of ['t0', `t${MAX_INDEXED_ARRAY_LENGTH - 1}`, `t${Math.floor(MAX_INDEXED_ARRAY_LENGTH / 2)}`]) {
const hits = await brain.find({ where: { tags: tag }, limit: 10 } as any)
expect(hits.map((r: any) => r.id)).toContain(resolveEntityId('at-bound'))
}