Commit graph

1 commit

Author SHA1 Message Date
e389d67567 **docs: add scaling strategies for HNSW with large-scale data**
### Changes:
- Added a new `scalingStrategy.md` document explaining limitations and solutions for handling large-scale data with HNSW:
  - Highlighted in-memory limitations of standard HNSW implementations.
  - Detailed disk-based, distributed, and hybrid approaches for scaling to terabyte-scale datasets.
  - Provided real-world examples from systems like Qdrant, Milvus, FAISS, and DiskANN.
  - Emphasized trade-offs between memory usage, search accuracy, and performance when scaling.

### Purpose:
Documented approaches for managing large-scale data with HNSW to assist developers in implementing efficient, scalable solutions. The guide serves as a reference for overcoming memory limitations and adopting advanced techniques for handling terabyte-scale datasets.
2025-06-20 11:00:52 -07:00