open-brainy/src
David Snelling 06f502ec4a feat: add direct storage access in write-only mode for efficient deduplication
Enables ID-based lookups in write-only mode without loading search indexes, solving the fundamental conflict between write-only optimization and deduplication needs.

Key Features:
- New allowDirectReads configuration option
- Direct storage methods: has(), exists(), getMetadata(), getBatch()
- Enhanced get() and getVerb() support in write-only mode
- Smart operation separation (storage vs. search operations)

Use Cases:
- Bluesky services: Avoid redundant profile API calls
- GitHub packages: Efficient user processing with existence checks
- General writer services: Smart deduplication without search overhead

Performance Benefits:
- 50-100% reduction in external API calls
- No search index memory usage
- Fast direct storage lookups
- Optimal for high-throughput data ingestion

Configuration:
const brainy = new BrainyData({
  writeOnly: true,        // Skip search index loading
  allowDirectReads: true  // Enable direct ID lookups
})

Includes comprehensive tests (26/26 passing), real-world demo, and complete README documentation with configuration examples.
2025-08-07 07:57:41 -07:00
..
augmentations feat: add intelligent verb scoring system for automatic relationship weighting 2025-08-06 17:47:11 -07:00
distributed feat: migrate system metadata from 'index' to '_system' directory with backward compatibility 2025-08-06 09:45:56 -07:00
errors **feat: implement robust error-handling and operation utilities for storage adapters** 2025-07-30 11:35:09 -07:00
examples refactor: clean up deprecated functions and unused code 2025-08-05 10:16:05 -07:00
hnsw fix: correct typo in README major updates section 2025-08-06 12:29:32 -07:00
mcp **docs: update README to streamline server functionality documentation** 2025-07-15 11:51:27 -07:00
storage feat: add automatic high-volume handling for S3 storage adapter 2025-08-07 06:11:45 -07:00
types feat\!: migrate from TensorFlow.js to Transformers.js with ONNX Runtime 2025-08-05 19:29:59 -07:00
utils feat: v0.49 - Filter discovery API, remove deprecated methods, improve performance 2025-08-06 14:39:33 -07:00
augmentationFactory.ts Initial commit 2025-06-24 11:41:30 -07:00
augmentationPipeline.ts fix(src/augmentationPipeline): remove unnecessary whitespace for formatting consistency 2025-06-30 09:39:25 -07:00
augmentationRegistry.ts Initial commit 2025-06-24 11:41:30 -07:00
augmentationRegistryLoader.ts Initial commit 2025-06-24 11:41:30 -07:00
brainyData.ts feat: add direct storage access in write-only mode for efficient deduplication 2025-08-07 07:57:41 -07:00
browserFramework.ts refactor: simplify build system and improve model loading flexibility 2025-08-05 16:09:30 -07:00
coreTypes.ts feat: v0.49 - Filter discovery API, remove deprecated methods, improve performance 2025-08-06 14:39:33 -07:00
demo.ts feat\!: migrate from TensorFlow.js to Transformers.js with ONNX Runtime 2025-08-05 19:29:59 -07:00
index.ts feat\!: migrate from TensorFlow.js to Transformers.js with ONNX Runtime 2025-08-05 19:29:59 -07:00
pipeline.ts Initial commit 2025-06-24 11:41:30 -07:00
sequentialPipeline.ts Initial commit 2025-06-24 11:41:30 -07:00
setup.ts feat\!: migrate from TensorFlow.js to Transformers.js with ONNX Runtime 2025-08-05 19:29:59 -07:00
unified.ts **docs: remove outdated statistics-related documentation and add standards** 2025-07-28 16:00:05 -07:00
worker.js **feat(cli, workers): introduce text encoding patches and worker improvements** 2025-07-04 14:42:33 -07:00
worker.ts **feat(tests): add tests for TextEncoder, TensorFlow.js, and fallback mechanisms** 2025-07-11 11:11:56 -07:00