Distributed HNSW Vector Storage for Predictable Search I/O
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional vector similarity search algorithms are not optimized for retrieval from slower distributed storage systems, leading to high memory consumption, scalability issues, and inefficient I/O overhead, making it impractical for low-latency applications and challenging to predict query behavior across different database engines.
Innovation Solution
A distributed proximity-based graph data structure is used to store and index vectors, utilizing a hierarchical navigable small world (HNSW) index across multiple devices, where vectors are grouped into leaf blocks stored across distributed storage, and a representative is selected for upper-level indexing, enabling efficient batch retrieval and parallel access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional vector similarity search algorithms are used, then memory consumption is reduced, but I/O efficiency deteriorates and query performance becomes unpredictable
Solution Approach 1:
The patent segments the vector index into a hierarchical structure with multiple levels (L0, L1, L2, etc.), where each level contains subsets of vectors. This segmentation allows the system to access only relevant portions of the index during search operations, improving I/O efficiency by avoiding full-index scans while managing memory consumption through selective loading of index levels.
Solution Approach 2:
The patent introduces a hierarchical dimension to the vector index structure, organizing vectors across multiple levels rather than a single flat structure. This dimensional organization enables efficient navigation from coarse-grained L0 level to fine-grained leaf levels, improving query performance predictability and I/O efficiency while allowing selective memory allocation across different hierarchy levels.
2Adaptability or versatility
If distributed storage systems are used, then scalability is improved, but I/O overhead increases
Solution Approach 1:
The patent pre-organizes vectors into a hierarchical index structure during index building, creating L0, L1, L2 levels and leaf blocks in advance. This preliminary organization enables efficient query execution by allowing the search algorithm to navigate the pre-built hierarchy and access only necessary data blocks, reducing I/O overhead during actual search operations while maintaining distributed storage scalability.
Solution Approach 2:
The patent introduces intermediate index levels (L1, L2, etc.) that act as mediators between the top-level L0 index and the actual vector data in leaf blocks. These intermediary structures enable efficient routing of search queries through the distributed storage system, reducing the number of I/O operations needed to locate relevant vectors while preserving scalability across distributed nodes.
3Speed
If hierarchical index structure is used, then query performance is improved, but device complexity increases
Solution Approach 1:
The patent applies local quality by making each level of the hierarchical index have specialized characteristics optimized for its function. L0 level uses coarse-grained filtering for quick initial screening, intermediate levels (L1, L2) provide progressive refinement, and leaf blocks contain actual vector data for final comparison. This localized optimization at each hierarchy level improves query performance while managing overall structure complexity through clear functional differentiation.
Data Source
AI summary
Systems and methods are provided for generating an index of a set of vectors as a distributed proximity-based graph data structure comprising representative vectors corresponding to subsets of the set of vectors, storing the set of vectors across a plurality of storage devices of a distributed storage system based on the distributed proximity-based graph data structure, and loading, in response to a vector query, a plurality of subsets of the set of vectors from the distributed storage system based on the distributed proximity-based graph data structure.