soulcraftlabs/open-brainy
Archived
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 345 Packages Wiki Activity

feat(find): field projection — fields resolve from the column store, not the record

Browse source 
A list view that shows a title and a slug hydrates the whole record for every
row, document bodies included, and discards almost all of it. find/get({ fields })
names what is wanted; the column store serves it; the canonical record is opened
only for fields the index cannot supply.

The provider grows an optional getScalarsForIds(ids, fields) door, batched: it
walks each column ONCE and picks out every requested id, rather than re-walking
per row. The column store grows the primitive that was missing — valuesForIds —
because every other read door there answers which entities have a value, and a
projection asks the opposite.

It reads the COLUMN store, never the sparse index: the column keeps raw values,
the sparse index keeps a bucketed form built for range queries, and a projection
served from the latter would return a value that differs from the record's. A
field the column cannot serve is omitted rather than approximated — omission
costs a read, a wrong value is a wrong answer nobody can see.

Two laws the pins hold: fields absent is byte-identical to today, and a missing
field is simply absent rather than an error — so this path deliberately avoids
the strict address resolver, whose UnresolvableFieldError is right for orderBy
and wrong here.

related() takes no fields: a Relation carries from/to as ids and hydrates no
record, so the param would be decorative.
This commit is contained in:
David Snelling 2026-09-02 13:36:33 -07:00
parent 6597c146f7
commit ad0f493f7a
7 changed files with 639 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

208
src/brainy.ts
View file

@ -4193,6 +4193,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
// Route to metadata-only or full entity based on options
// A PROJECTED get goes through the same seam every list page uses, so a
// detail read of two scalars costs an index read rather than a record read.
// It is checked before `includeVectors` because the two are incompatible by
// construction: a projection returns the named fields, and a vector is not
// one of them unless it was named.
if (options?.fields !== undefined && options.fields.length > 0) {
const page = await this.hydratePage([id], options.fields)
return page.get(id) ?? null
}
const includeVectors = options?.includeVectors ?? false // Default: metadata-only (fast)
if (includeVectors) {
@ -4239,6 +4249,170 @@ export class Brainy<T = any> implements BrainyInterface<T> {
* const children = childIds.map(id => childrenMap.get(id)).filter(Boolean)
* ```
*/
/**
* **The projection seam** — hydrate a page of ids under an optional `fields`
* projection, opening the canonical record only when the index cannot serve
* what was asked for.
*
* Without a projection this is exactly `batchGet`, byte for byte: the whole
* point is that `fields` absent changes nothing.
*
* With one, the order is: ask the index for the named scalars in a single
* batched door; see which requested fields it actually served; and read
* records ONLY if something is still missing — and only to fill those fields.
* A page whose every requested field is index-served performs zero canonical
* reads, which is the whole reason the door exists.
*
* `guardFields` are fetched ALONGSIDE the projection and trimmed off before
* the caller sees them. find()'s index-integrity guard re-validates every row
* against its own predicate, and it reads the entity to do so — so a row
* projected down to `title` would fail a `where: { kind }` it genuinely
* matches, and the whole page would vanish. The fields a filter names are
* fields the index can serve by definition, so carrying them costs nothing
* and keeps the guard honest.
*
* A field nothing can supply is simply absent from the row. That is the
* permissive law: a projection asks "these, if you have them", and an
* optional field must not turn a list into an exception. It deliberately does
* NOT route through the strict address resolver, which throws
* `UnresolvableFieldError` for an unknown key — that strictness is right for
* `orderBy`, where a typo silently changes the order, and wrong here, where
* the honest answer is "this row does not have that".
*
* @param ids - Canonical ids for the page.
* @param fields - The projection, or undefined for the full record.
* @returns `id → entity`, projected when `fields` was given.
*/
/**
* The index keys find()'s integrity guard reads when it re-validates a row.
*
* The guard calls `entityMatchesFind(entity, params)`, so a projected entity
* must still carry whatever the params constrain — otherwise a row that
* genuinely matches is dropped for lacking the evidence. These are fetched
* with the projection and trimmed off before the caller sees them.
*
* @param params - The find params.
* @returns Index keys to carry through hydration.
*/
private guardFieldsFor(params: FindParams<T>): string[] {
const keys: string[] = []
if (params.where && typeof params.where === 'object') {
// Top-level where keys only: nested `anyOf`/`allOf` branches are carried
// by their own keys when the guard walks them, and a filter whose
// evidence is missing keeps the row (the guard's own catch) rather than
// dropping it.
for (const key of Object.keys(params.where as Record<string, unknown>)) {
if (key === 'anyOf' || key === 'allOf' || key === 'not') continue
keys.push(key)
}
}
if (params.type !== undefined) keys.push('system.type')
if (params.subtype !== undefined) keys.push('system.subtype')
if (params.service !== undefined) keys.push('system.service')
if (params.excludeVFS === true) keys.push('vfsType', 'isVFSEntity')
return keys
}
private async hydratePage(
ids: string[],
fields?: readonly string[],
guardFields: readonly string[] = []
): Promise<Map<string, Entity<T>>> {
if (fields === undefined || fields.length === 0) return this.batchGet(ids)
const wanted = [...new Set([...fields, ...guardFields])]
const provider = this.metadataIndex as unknown as MetadataIndexProvider
let served = new Map<string, Record<string, unknown>>()
if (typeof provider.getScalarsForIds === 'function') {
served = await provider.getScalarsForIds(ids, wanted)
}
// Which ids still owe a field? Only those cost a record read, and a page
// that owes nothing costs none at all.
const owing: string[] = []
for (const id of ids) {
const row = served.get(id)
if (row === undefined || wanted.some((f) => !(f in row))) owing.push(id)
}
// The records are read for the OWED fields only; everything the index
// already served is used as-is, so a body field pulls its own record and
// no more than that.
const records = owing.length > 0 ? await this.batchGet(owing) : new Map<string, Entity<T>>()
const out = new Map<string, Entity<T>>()
for (const id of ids) {
const fromIndex = served.get(id)
const record = records.get(id)
// An id neither the index nor storage knows is not a row.
if (fromIndex === undefined && record === undefined) continue
out.set(id, this.projectEntity(id, wanted, fromIndex, record))
}
return out
}
/**
* Build one projected entity: `id`, plus exactly the requested fields that
* something could supply.
*
* Values come from the index first and the record second, and they must agree
* — the index only reports what it can serve exactly, so a field it served is
* the record's value. A field neither has is omitted rather than set to
* `undefined`: absent and present-and-undefined are different answers, and a
* caller checking `'slug' in row.metadata` deserves the true one.
*
* @param id - The entity id, always present on the result.
* @param fields - The requested index keys.
* @param fromIndex - What the index served for this id, if anything.
* @param record - The canonical entity, if one had to be read.
* @returns The projected entity.
*/
private projectEntity(
id: string,
fields: readonly string[],
fromIndex: Record<string, unknown> | undefined,
record: Entity<T> | undefined
): Entity<T> {
const projected: Record<string, unknown> = { id }
const metadata: Record<string, unknown> = {}
let sawMetadata = false
for (const field of fields) {
let value: unknown
let found = false
if (fromIndex !== undefined && field in fromIndex) {
value = fromIndex[field]
found = true
} else if (record !== undefined) {
if (field.startsWith('system.')) {
const inner = field.slice('system.'.length)
const bag = record as unknown as Record<string, unknown>
if (inner in bag && bag[inner] !== undefined) {
value = bag[inner]
found = true
}
} else {
const bag = (record.metadata ?? {}) as Record<string, unknown>
if (field in bag) {
value = bag[field]
found = true
}
}
}
if (!found) continue
if (field.startsWith('system.')) {
projected[field.slice('system.'.length)] = value
} else {
metadata[field] = value
sawMetadata = true
}
}
if (sawMetadata) projected.metadata = metadata
return projected as unknown as Entity<T>
}
async batchGet(ids: string[], options?: GetOptions): Promise<Map<string, Entity<T>>> {
// Canonical read (see get): resolves by id from storage, no derived index.
await this.ensureInitialized({ needs: [] })
@ -8037,7 +8211,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Batch-load entities for 10x faster cloud storage performance
// GCS: 10 entities = 1×50ms vs 10×50ms = 500ms (10x faster)
const entitiesMap = await this.batchGet(pageIds)
const entitiesMap = await this.hydratePage(pageIds, params.fields, this.guardFieldsFor(params))
for (const id of pageIds) {
const entity = entitiesMap.get(id)
if (entity) {
@ -8074,7 +8248,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
if (hiddenIds.size > 0) allUuids = allUuids.filter((id) => !hiddenIds.has(id))
const pageIds = allUuids.slice(offset, offset + limit)
const entitiesMap = await this.batchGet(pageIds)
const entitiesMap = await this.hydratePage(pageIds, params.fields, this.guardFieldsFor(params))
for (const id of pageIds) {
const entity = entitiesMap.get(id)
if (entity) {
@ -8102,7 +8276,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const pageIds = filteredIds.slice(offset, offset + limit)
// Batch-load entities for 10x faster cloud storage performance
const entitiesMap = await this.batchGet(pageIds)
const entitiesMap = await this.hydratePage(pageIds, params.fields, this.guardFieldsFor(params))
for (const id of pageIds) {
const entity = entitiesMap.get(id)
if (entity) {
@ -8337,7 +8511,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Batch-load entities for current page - O(page_size) instead of O(total_results)
// GCS: 10 entities = 1×50ms vs 10×50ms = 500ms (10x faster)
const entitiesMap = await this.batchGet(pageIds)
const entitiesMap = await this.hydratePage(pageIds, params.fields, this.guardFieldsFor(params))
for (const id of pageIds) {
const entity = entitiesMap.get(id)
if (entity) {
@ -8365,7 +8539,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Batch-load entities for paginated results (10x faster on GCS)
const sortedResults: Result<T>[] = []
const entitiesMap = await this.batchGet(pageIds)
const entitiesMap = await this.hydratePage(pageIds, params.fields, this.guardFieldsFor(params))
for (const id of pageIds) {
const entity = entitiesMap.get(id)
if (entity) {
@ -8470,6 +8644,28 @@ export class Brainy<T = any> implements BrainyInterface<T> {
})
}
// PROJECTION TRIM — applied once, here, AFTER the integrity guard, so every
// find() path is trimmed uniformly and the guard still saw the evidence it
// needs. Hydration carried the guard's fields alongside the projection;
// this removes them, leaving exactly what the caller named.
//
// Rows that reached here from a path the seam does not hydrate (a vector or
// text leg builds its own entities) are trimmed from what they already
// hold, so the ANSWER is the same everywhere — only the cost differs, and
// only on the paths that still read a record.
if (params.fields !== undefined && params.fields.length > 0 && result.length > 0) {
const named = [...new Set(params.fields)]
result = result.map((r) => {
const projected = this.projectEntity(
r.id,
named,
undefined,
r.entity as unknown as Entity<T>
)
return { ...r, entity: projected } as typeof r
})
}
// includeVectors — opt-in vector hydration. Default (false) keeps the perf
// contract: every result path above builds entities via the metadata-only
// fast path, so `entity.vector` is the empty stub. When requested, fetch the
@ -16842,7 +17038,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
ordered = valued.map((v) => v.id)
}
const pageIds = ordered.slice(offset, offset + limit)
const entitiesMap = await this.batchGet(pageIds)
const entitiesMap = await this.hydratePage(pageIds, params.fields, this.guardFieldsFor(params))
const results: Result<T>[] = []
for (const id of pageIds) {
const entity = entitiesMap.get(id)