feat(8.0): brain.fillSubtypes migration helper + pre-RC1 gap closure

- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
  One rule per NounType/VerbType (literal default or per-entry function);
  fills only entries still missing a subtype through the public update()/
  updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
  Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
  was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
  opaque storage sharding failure; CLI --threshold without --near applies a
  plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
  explicitly; delete the unmaintained interactive REPL; replace the cloud-era
  storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
  recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
  README storage section reflects filesystem+memory and snapshots; eli5 and
  SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
  .strategy references scrubbed from published files.
This commit is contained in:
David Snelling 2026-06-11 10:42:34 -07:00
parent 9b0f4acd5b
commit c44678390e
30 changed files with 1517 additions and 3226 deletions

View file

@ -748,9 +748,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
*
* If neither the column store nor a sparse index covers the field, the
* function throws `BrainyError(FIELD_NOT_INDEXED)`. Returning `[]` for a
* genuinely unindexed field was the bug class `BR-FIND-WHERE-ZERO`
* tracked a silent empty result indistinguishable from "the data
* really isn't there."
* genuinely unindexed field was a long-standing silent-empty bug class
* an empty result indistinguishable from "the data really isn't there."
*/
private async getIdsFromChunks(field: string, value: any): Promise<string[]> {
// Load sparse index via UnifiedCache (lazy loading)
@ -1200,7 +1199,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// - Chunked Sparse Index: ~50 values per chunk, lazy-loaded
// - UnifiedCache LRU: Only hot chunks in memory
//
// Future: Bloom filter hybrid for unlimited words (see .strategy/BILLION-SCALE-PLAN.md)
// A Bloom-filter hybrid could lift the per-entity word cap entirely if
// full-document indexing at billion-entity scale ever becomes a need.
const textContent = this.extractTextContent(data)
if (textContent) {
const MAX_WORDS_PER_ENTITY = 5000 // Handles articles/chapters, memory-safe at scale
@ -1600,8 +1600,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* Throws `BrainyError(FIELD_NOT_INDEXED)` if the field has no entries in
* either store. Callers in find()-evaluation catch this and translate to
* an empty result with a logged warning. The throw aligns the production
* `find()` path with the `brain.explain()` diagnostic, so the silent-
* empty bug class (BR-FIND-WHERE-ZERO) is no longer possible.
* `find()` path with the `brain.explain()` diagnostic, so a silently
* empty result for an unindexed field is no longer possible.
*/
async getIds(field: string, value: any): Promise<string[]> {
// Track exact query for field statistics
@ -2866,8 +2866,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
*
* Prior implementation read from `this.fieldIndexes` + lazy-loaded sparse
* indices, which silently returned `0` entries for any workspace written
* after sparse-index writes were deleted in commit `11be039`. That defect
* is what `BR-FIND-WHERE-ZERO` tracked.
* after sparse-index writes were deleted in commit `11be039`. That
* silent-zero defect is why this reads the column store first.
*/
async getStats(): Promise<MetadataIndexStats> {
const entityCount = this.idMapper.size