David Snelling
46583f2d9b
Adds a per-field sorted column store (Lucene doc values + roaring bitmap
architecture) that replaces the MetadataIndex sparse index internals for
both filtering and sorting. One system for all field types with exact
precision — no bucketing, no per-entity storage reads.
Column store: binary .cidx segment format, in-memory tail buffers,
LSM-style compaction, k-way merge sort, multi-value support (__words__).
All queries (filter, range, sort, filtered sort) route through the column
store when data is available, falling back to sparse index otherwise.
Key unlocks:
- find({ orderBy: 'createdAt' }) works WITHOUT a filter (previously threw)
- find({ orderBy: 'metadata.price' }) works for custom numeric fields
- Exact timestamp precision (no 1-minute bucketing)
- O(K log S) sort independent of total entity count
New files: src/indexes/columnStore/ (types, format, tail buffer, cursor,
manifest, coordinator — ~700 lines). 101 new unit tests covering binary
format round-trips, CRC validation, sort, filter, range, deletion,
multi-segment merge, persistence, and multi-value (words) fields.
Deleted: metadataIndex-automatic-bucketing.test.ts (bucketing behavior
eliminated by exact-precision column store). Sparse index write path
removed from addToIndex/removeFromIndex. Sparse index legacy code still
present as dead code pending cleanup in next commit.
38 lines
1 KiB
TypeScript
38 lines
1 KiB
TypeScript
/**
|
|
* @module columnStore
|
|
* @description Unified column store for filtering, sorting, range queries,
|
|
* and text search on any field type with exact precision at billion scale.
|
|
*
|
|
* Replaces MetadataIndex's sparse chunk/bloom filter/bucketing internals
|
|
* with per-field sorted columns. Design lineage: Lucene doc values + roaring
|
|
* bitmap acceleration.
|
|
*/
|
|
|
|
export { ColumnStore } from './ColumnStore.js'
|
|
export type { ColumnStoreConfig } from './ColumnStore.js'
|
|
export { ColumnTailBuffer } from './ColumnTailBuffer.js'
|
|
export { ColumnManifest } from './ColumnManifest.js'
|
|
export { ColumnSegmentCursor, TailBufferCursor } from './ColumnSegmentCursor.js'
|
|
export {
|
|
writeSegmentToBuffer,
|
|
readSegmentFromBuffer,
|
|
writeHeader,
|
|
readHeader,
|
|
crc32
|
|
} from './ColumnSegmentFormat.js'
|
|
export {
|
|
ValueType,
|
|
CIDX_MAGIC,
|
|
CIDX_VERSION,
|
|
HEADER_SIZE,
|
|
FOOTER_SIZE,
|
|
DEFAULT_FLUSH_THRESHOLD,
|
|
FLAG_MULTI_VALUE
|
|
} from './types.js'
|
|
export type {
|
|
ColumnStoreProvider,
|
|
SegmentHeader,
|
|
SegmentFooter,
|
|
SegmentMeta,
|
|
ManifestData
|
|
} from './types.js'
|