soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: code-point string collation in LSM SSTable, COW trees/refs, sorted queries

Browse source 
Replaces localeCompare / raw relational operators with compareCodePoints (UTF-8
byte order) in the remaining persisted or native-facing comparison sites:
- SSTable sourceId sort + zone-map range check + binary search (graph LSM)
- COW TreeObject + RefManager name sorts (reproducible content hashes and ref
  listings across environments)
- getSortedIdsForFilter (numeric-aware; code-point for strings) so the JS sort
  fallback matches the native column store exactly

Deterministic across OS/Node/ICU and byte-identical to the native engine. No
migration: COW serialize/deserialize preserve stored order, so existing trees
keep their hashes; only new writes adopt code-point order.
This commit is contained in:
David Snelling 2026-05-27 11:53:26 -07:00
parent 547721ae14
commit 7493d8e33f
4 changed files with 29 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

12
src/utils/metadataIndex.ts
View file

@ -7,6 +7,7 @@
import { StorageAdapter, resolveEntityField } from '../coreTypes.js'
import { ColumnStore } from '../indexes/columnStore/ColumnStore.js'
import { MetadataIndexCache, MetadataIndexCacheConfig } from './metadataIndexCache.js'
import { compareCodePoints } from './collation.js'
import { prodLog } from './logger.js'
import { getGlobalCache, UnifiedCache } from './unifiedCache.js'
import {
@ -2109,7 +2110,16 @@ export class MetadataIndexManager {
if (a.value == null) return order === 'asc' ? 1 : -1
if (b.value == null) return order === 'asc' ? -1 : 1
if (a.value === b.value) return 0
const comparison = a.value < b.value ? -1 : 1
// Numbers compare numerically; everything else by code-point (UTF-8 byte) order.
// This makes the JS fallback sort match cortex's native column store exactly
// (numeric i64/f64 vs code-point strings) and stay deterministic across
// environments, unlike the `<` operator's UTF-16 ordering for strings.
let comparison: number
if (typeof a.value === 'number' && typeof b.value === 'number') {
comparison = a.value < b.value ? -1 : 1
} else {
comparison = compareCodePoints(String(a.value), String(b.value))
}
return order === 'asc' ? comparison : -comparison
})