CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
Delta Gate / Delta gate — candidate vs control (push) Failing after 7s
The metadata index fixed a field's value type from the first value it saw.
Every later value of another kind was coerced to that type, and when coercion
failed — `Number('electronics')` is NaN — the value was dropped from the index
with no error at all. The row stayed readable by id and by vector search and
vanished only from equality filters on that one field, which is what made it so
quiet: writing `category: 'electronics'` rows and then `category: 5` rows left
`where { category: 5 }` returning nothing, while the same rows in a
numbers-only corpus answered correctly.
The column store now keeps one posting column per (field, kind), where a kind
is a JavaScript typeof class. The first kind a field sees keeps the historical
`_column_index/<field>/` layout, so a single-kind field is byte-identical to
what earlier versions wrote and an index written before this opens unchanged;
each later kind takes its own column at `_column_index/<field>/k/<kind>/`.
Equality reads the column matching the query value's own kind, so `{c: 5}` and
`{c: '5'}` match different rows and neither is coerced into the other. Ranges
route by the kind of their bounds, and an unbounded range — the "has any value"
probe behind `exists` — reads every kind. A mixed field orders by kind first,
then by value, because a number and a string have no order between them. A
value that cannot be encoded for the column its own kind selected now raises
instead of being skipped: that path is unreachable by construction, and if it
is ever reached it is the silent drop this change exists to end.
Two neighbours fell out of the same routing. A boolean query value is now
encoded to the 1/0 the column stores, so boolean equality matches at all. And
an integer column widens to f64 the first time a non-integer arrives, so 4.5 is
stored as itself rather than rounded to 5 and answering the wrong query.
Field type inference reports every kind a field holds beside its dominant
reading, rather than leaving callers to treat one type as the whole answer.
Pins: mixed-kind equality in both write orders, `5` vs `'5'`, booleans mixed in,
a numeric range over a mixed field's numbers, close/reopen keeping every typed
posting, and an index in the pre-existing on-disk shape still reading.
`tests/critical-neural-validation.test.ts` — which writes `category` as strings
in one test and as numbers in another against one shared brain — passes whole
for the first time.
(cherry picked from commit a128f0eda5)
Prepares the repo for its new home at soulcraftlabs/open-brainy ahead
of the Forgejo transfer: package name, publish registry, release
script, and every install/import reference across docs, src, tests,
examples, and integrations now point at @soulcraftlabs/brainy on
The Source. The npmjs storefront leg and byte-identity pair
verification are stripped from the release script — The Source is
now the only publish target. README gains an Open Brainy explainer
and a registry note for consumers.
@soulcraft/brainy 10.4.2 was the last release under the old name.
- data-storage-architecture.md described transact() as 'the unit of history'
(rc.2 era). Single-op retention has been the model since 8.0 — every write
gets its own generation; transact() only groups several into one. Corrected.
- RELEASE-GUIDE.md now records the hard ordering constraint: no embedding-model
change ships before vector model-version stamping + hard-error-on-mismatch
lands.
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
One rule per NounType/VerbType (literal default or per-entry function);
fills only entries still missing a subtype through the public update()/
updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
opaque storage sharding failure; CLI --threshold without --near applies a
plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
explicitly; delete the unmaintained interactive REPL; replace the cloud-era
storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
README storage section reflects filesystem+memory and snapshots; eli5 and
SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
.strategy references scrubbed from published files.
The __words__ keyword index stores 50-5000 entries per entity (one per
word), which inflated avg entries/entity well above the corruption
threshold of 100. This caused:
1. validateConsistency() to falsely detect corruption on every startup,
triggering unnecessary clearAllIndexData() + rebuild() cycles
2. getStats() to log false "Metadata index may be corrupted" warnings
and report inflated totalEntries/totalIds stats
Both methods now skip __words__ when counting, so stats and health
checks reflect metadata fields only (noun, type, createdAt, etc.).
Keyword search is unaffected since the __words__ field index itself
is not modified.
Critical fix for GCS/S3 native storage adapters crashing on metadata index keys.
Problem:
- GCS and S3 adapters crashed with "Invalid UUID format" errors
- System keys like __metadata_field_index__status are NOT UUIDs
- Adapters incorrectly tried to shard all metadata keys as UUIDs
Solution:
- Move sharding/routing logic from adapters to BaseStorage class
- Add analyzeKey() method to detect system keys vs entity UUIDs
- System keys route to _system/ directory (no sharding)
- Entity UUIDs route to sharded directories (256 shards)
- All adapters now implement 4 primitive operations:
* writeObjectToPath(path, data)
* readObjectFromPath(path)
* deleteObjectFromPath(path)
* listObjectsUnderPath(prefix)
Benefits:
- Impossible for future adapters to repeat this mistake
- Zero breaking changes, full backward compatibility
- No data migration required
- Cleaner architecture with better separation of concerns
Updated adapters: GcsStorage, S3CompatibleStorage, OPFSStorage,
FileSystemStorage, MemoryStorage
Added: docs/architecture/data-storage-architecture.md
Updated: README.md with architecture docs link
