The class-killer for the lost-main.dkann incident (ADR-004 §7). A provider can
declare a derived-index blob FAMILY (a set of members that are load-bearing
together, e.g. vector-base = main.dkann + main.slotmap + main.slotrev). Once
declared:
- deleteBinaryBlob / removeRawPrefix REFUSE to remove a declared member, throwing
the new ProtectedArtifactError — an in-process GC / sweeper is now INCAPABLE of
deleting a load-bearing index file (COLD != DEAD). Intentional retirement is an
explicit unregisterDerivedFamily(name) first.
- The declaration persists to _system/derived-artifacts.json, so protection
survives a reopen; clear() resets it with the rest of the derived footprint.
- checkDerivedFamiliesPresent() names any member missing on open (the catch for an
EXTERNAL deleter that bypasses the in-process refusal) → rebuild from canonical.
- Transients (*.tmp.*, *.rebuild-tmp, *.rotate-tmp) are never protected; a
namespace family protects a growing prefix (seg-*).
New StorageAdapter surface (optional): registerDerivedFamily / unregisterDerivedFamily
/ listDerivedFamilies + DerivedFamilyDeclaration; new exported errors
ProtectedArtifactError / DerivedArtifactMissingError. Enforcement is inert until a
provider declares a family (no regression). Cor declares its 6 families + does the
atomic set-swap in 3.0.15 (M2); brainy builds the contract now. 8 tests.
Closes the "dishonest readiness proxy" anti-pattern (Pattern A): size()>0 /
isInitialized were treated as "this index serves queries", but a cold native
index can load its COUNT before its SERVING structure, so a query returned a
silent [] indistinguishable from "no such data". A shared assessIndexReadiness()
now reads only the provider's honest isReady() signal (never size()), applied at
every site:
- Vector: a one-shot verifyVectorLive() guard on the semantic/proximity search
path (a pure semantic find({query}) has no filter, so nothing guarded it). It
prefers isReady(), else a known-vector self-match probe; self-heals via rebuild
or throws the new VectorIndexNotReadyError instead of a silent [].
- Graph: getVerbsBySource/ByTarget skip the fast path when the provider reports
not-ready (falling to the canonical shard scan), plus a one-shot probe that
self-heals a no-isReady provider whose adjacency did not cold-load.
- getIndexStatus(): folds in per-index honest `ready` (making `populated`
honest) + rebuildFailed/rebuildError/degradedIds, so a readiness probe never
200s a brain that is still warming up or degraded.
Unblocked by the native providers now reporting serving-truth (graph via
SSTable-residency readiness, vector via durableBaseLoadFailed). New export:
VectorIndexNotReadyError. getIndexStatus gains additive fields. No breaking API.
13 new tests; existing readiness guards green.
`find({ where })` had two gaps that could silently return nothing. The in-memory
matcher (`matchesQuery`, used for egress re-validation and historical reads) was
missing several documented operators — `in`, `greaterThanOrEqual`,
`lessThanOrEqual` — so it disagreed with the index path, which does handle them.
And an unknown operator key (a typo, or `notIn` written where `not: { in }` was
meant) fell through to a nested-object-field interpretation and matched nothing.
- Align `matchesQuery` to the full documented operator set (add the missing
aliases; the default branch now throws instead of treating an unknown key as a
nested field — dotted paths remain the supported nested form).
- Validate the `where` filter up front (`validateWhereFilter`), throwing a typed
`BrainyError('INVALID_QUERY')` that names the unknown operator, recursing into
`allOf` / `anyOf` / `not`.
- Stop excluding a user field literally named `level` from the metadata index
(it collided with an internal never-index key), so numeric/exact filters on it
resolve instead of returning 0.
Adds unit coverage for the operator set and the throw-on-unknown behavior.
A downstream deployment reported find({where}) returning a silent [] on a
freshly-opened brain (a native metadata provider that reports data but hasn't
loaded its field postings), blanking filtered pages after every restart. Brainy
had a cold-read guard for GRAPH reads (verifyGraphAdjacencyLive → self-heal or a
loud GraphIndexNotReadyError) but no equivalent for metadata `where`.
Add verifyMetadataLive() — the field-index counterpart, one-shot per brain on the
first filtered find(): probe a known persisted entity's plain field value; if the
index serves it, live (the only cost on a warm brain — one O(1) probe). If not,
the postings didn't load: rebuild from canonical + re-probe; if still unserved,
throw the new exported MetadataIndexNotReadyError rather than let a silent [] pass.
Inconclusive cases (empty store, no plain field, shared-store foreign entity,
migrating provider) resolve to live — never a false rebuild.
The 8.0 open-core JS index already cold-loads correctly (verified: cold where 3/3,
cold related 2/2) — this guards the NATIVE path, where the durable cold-load cure
is cortex-side (same shape as the graph 2.7.8 cure). 4 unit tests (warm no-rebuild,
self-heal, loud-fail, no-filter-no-probe). Gates: typecheck 0, build 0, test:unit
1757/1757.
Replaces rc.8's no-freeze deference with an automatic, observable, coordinated
migration LOCK (David's reversal: "unknown/halfway states are more dangerous
than blocking"). While a native provider runs its one-time 7.x→8.0
rebuild-from-canonical (`isMigrating()===true`), brainy blocks/queues data-plane
reads AND writes so no operation ever touches a half-built index — closing the
three seam-map gaps (lost mid-rebuild writes, degraded reads, read-time rebuild).
Mechanism (one choke point):
- awaitMigrationLock() at ensureInitialized() covers all ~40 data-plane methods;
a non-migrating brain pays one boolean check. Polls isMigrating() at 250ms;
after migrationWaitTimeoutMs (default 30s) throws a retryable, exported
MigrationInProgressError. The timeout bounds the CALLER'S WAIT, not the
migration — the rebuild is unbounded and never interrupted.
- init() awaits the lock before VFS bootstrap + before serving ("not ready until
upgraded"); a rebuild past the budget surfaces MigrationInProgressError at init
(raise the budget, or run the offline migrator).
Observability (readiness-probe correct — never gated):
- getIndexStatus() gains `migrating` + `migration` (MigrationProgress); a probe
maps migrating→HTTP 503+Retry-After, not 500. health()/checkHealth() lock-exempt
(health() reports a `warn` migration check). stampBrainFormat()/close() are
ungated so cor can clear the lock — no deadlock.
- Optional provider migrationStatus() is relayed verbatim for a live %.
verifyGraphAdjacencyLive() honors the lock (no self-rebuild mid-migration); the
stamp is still withheld while any provider migrates (cor stamps after verify).
Adversarially verified: fixed a critical init/VFS-bootstrap deadlock, a mixed-
provider busy-spin (dropped the event-driven signal → pure poll), a not-yet-
initialized getIndexStatus crash, and once-per-window log/clock resets. 10 lock
tests (incl. the init-during-migration regression). Gates: typecheck 0, build 0,
test:unit 1753/1753.
8.0 shipped with no cold-graph guard (the 7.33.4 fix was never ported), so on a
cold open of a large brain where the native adjacency reports membership but its
source->target edges did not load, find({connected})/neighbors()/related() could
silently return [] for persisted edges.
Add the converged isReady() contract: GraphIndexProvider.isReady?(): boolean is
the honest cold-load readiness signal (true ONLY when edges are loaded), so brainy
gates on it instead of the lying membership-size() proxy. verifyGraphAdjacencyLive()
checks it — Strategy 1: false -> hydrate the id-mapper, rebuild from storage, re-check,
throw GraphIndexNotReadyError if still not ready; providers without isReady() fall
back to a global known-edge-sample probe (Strategy 2, not the queried anchor, so a
genuinely edgeless node still returns []). rebuildIndexesIfNeeded gates the graph
rebuild on it too, with the id-mapper hydrated before the adjacency rebuild on the
lazy cold-open path (the CTX-BR-RESTORE-REBUILD ordering, shared with restore()).
executeGraphSearch re-verifies before trusting an empty connected result and
re-collects on a heal. 6-case integration test + 1718 unit green.
Two production correctness defects fixed together so consumers can land
one upgrade.
BR-FIND-WHERE-ZERO — find() returned [] and stats() reported 0 entities
for any workspace whose data was written after the 7.20.0 column-store
refactor. Root cause: getStats() and the getIds() fallback still read
from the deleted sparse-index path. Separately, BaseStorage.getNounType()
was hardcoded to return 'thing', poisoning type-statistics.json with
every noun attributed to that bucket.
- MetadataIndex.getStats() reads from ColumnStore + idMapper.
- MetadataIndex.getIdsFromChunks() throws BrainyError(FIELD_NOT_INDEXED)
when neither store has the field. getIdsForFilter() catches per
clause, logs once, returns [].
- BaseStorage.nounTypeByIdCache populated in saveNounMetadata_internal
and consumed in saveNoun_internal. flushCounts() now persists the
Uint32Array counters too, so readers see fresh per-type counts.
- Self-heal at init: loadTypeStatistics() auto-rebuilds when the
poisoned signature is detected. rebuildTypeCounts() is now public for
use from `brainy inspect repair`.
- Dead state removed: dirtyChunks, dirtySparseIndices,
flushDirtyMetadata().
BR-DEFENSIVE-INTERFACE — 7.21.0 called supportsMultiProcessLocking()
unconditionally, crashing on older Cortex storage adapters that predate
the method. New hasStorageMethod(name) helper gates every new-method
call site. Older adapter triggers a one-line warning at init pointing
at the recommended plugin version.
Tests:
- new tests/integration/find-where-zero.test.ts (7 cases)
- new tests/integration/cortex-compat.test.ts (5 cases)
- multi-process-safety.test.ts updated to enforce correct counts
- 1329/1329 unit + 24/24 new integration tests passing
