feat: stable EntityIdMapper — rebuild() no longer renumbers UUID→int

Previously metadataIndex.rebuild() called idMapper.clear() which reset nextId
to 1 and renumbered every UUID by re-insertion order. Any consumer that had
persisted int-keyed data against the old map was silently invalidated — and
2.4.0's vector mmap store (#20), graph link compression (#21), and column-store
JS↔native interchange all need persisted int indices that survive a rebuild.

Remove the unconditional clear() in rebuild(). The rebuild already re-iterates
every entity via idMapper.getOrAssign(uuid), which returns the existing int
unchanged for known UUIDs. Stale UUID→int entries for entities no longer in
storage persist as harmless memory overhead; a dedicated prune step can be
added if it ever matters.

clearAllIndexData() — the explicit nuclear recovery path — keeps its existing
idMapper.clear() call (renumbering is intentional there), and now logs a
prodLog.warn making it explicit that any persisted int-keyed data is invalidated
and must be rebuilt from canonical sources.

Strengthened the EntityIdMapper class JSDoc to document the stability guarantee
as a contract — append-only getOrAssign, monotonic nextId, remove() leaves
permanent holes, rebuild() never renumbers, only clear() does.

Added tests/regression/entity-id-mapper-stability.test.ts pinning down the
five-point contract: (1) single-rebuild stability; (2) many-rebuild stability;
(3) post-rebuild adds get fresh monotonic ints; (4) removes leave permanent
holes — new entities never recycle; (5) clearAllIndexData() explicitly
renumbers (the documented destructive path).

Foundation for 2.4.0 #2-#4. Full test suite (62 files, 1417 tests) green.
This commit is contained in:
David Snelling 2026-05-28 09:45:22 -07:00
parent df7d739008
commit b2408cb5a6
3 changed files with 209 additions and 10 deletions

View file

@ -2635,13 +2635,20 @@ export class MetadataIndexManager implements MetadataIndexProvider {
this.verbCountsByTypeFixed.fill(0)
this.typeFieldAffinity.clear()
// Clear EntityIdMapper
// Clear EntityIdMapper. This is the explicit destructive path: the caller
// asked for nuclear recovery of a corrupted index, so renumbering UUIDs is
// intentional. Persisted int-keyed data (vector-mmap slots, graph
// link-compression encodings) is invalidated by this op — the warning
// below makes that explicit. Rebuild on its own does NOT clear the mapper.
await this.idMapper.clear()
// Clear chunk manager cache
this.chunkManager.clearCache()
prodLog.info(`✅ Cleared ${deletedCount} field indexes and all in-memory state`)
prodLog.warn('⚠️ EntityIdMapper was cleared — any persisted int-keyed data ' +
'(vector mmap slots, graph link-compression encodings, etc.) is now stale ' +
'and must be rebuilt from canonical sources.')
prodLog.info('⚠️ Run brain.index.rebuild() to recreate the index from entity data')
}
@ -3032,8 +3039,19 @@ export class MetadataIndexManager implements MetadataIndexProvider {
prodLog.info(`Cleared ${existingFields.length} field indexes from storage`)
}
// Clear EntityIdMapper to start fresh
await this.idMapper.clear()
// EntityIdMapper is intentionally NOT cleared here. Rebuild re-iterates
// every entity in storage and calls idMapper.getOrAssign(uuid), which
// returns the existing int for known UUIDs (no renumbering). This is the
// foundational stability guarantee — vector-mmap slot indices, graph
// link-compression encodings, and any other persisted int-keyed data
// remain valid across a rebuild. Previously this line reset nextId to 1
// and renumbered every UUID by re-insertion order, silently breaking
// any consumer that had persisted int-keyed data against the old map.
// Stale entries for UUIDs no longer in storage persist (harmless memory
// overhead); a dedicated prune step can be added if it ever matters.
// The destructive wipe is still available via clearAllIndexData() →
// idMapper.clear(), which is the explicit "recovery" path with the
// appropriate warning about invalidating persisted int-keyed data.
// Clear chunk manager cache
this.chunkManager.clearCache()