Hybrid Vector Indexing for Real-Time Search and Lower Storage Cost
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for writing and searching vector data in vector databases fail to balance real-time performance and storage costs, particularly for large-scale data scenarios.
Innovation Solution
Implement a composite index solution using a memory-based real-time read-write vector graph index (e.g., HNSW) and a disk-based low-cost vector graph index (e.g., DiskANN), where index update information is stored in a distributed file system, allowing conversion from a dynamic memory index to a static disk-based index.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a memory-based real-time read-write vector graph index (e.g., HNSW) is used, then real-time search performance is improved, but storage costs and memory consumption increase
Solution Approach 1:
The patent segments the index into two distinct parts: a memory-based HNSW index for real-time search operations and a disk-based DiskANN index for historical data storage. This segmentation allows each index type to be optimized for its specific function, with the HNSW index handling incremental real-time queries and the DiskANN index managing bulk historical data, thereby resolving the contradiction between real-time performance and storage costs
Solution Approach 2:
The patent introduces a temporal dimension to the index management by implementing periodic snapshots that convert the single-timepoint memory index into a multi-timepoint hybrid structure. This dimensional transformation allows the system to maintain real-time performance for current data while efficiently archiving historical data on disk, effectively addressing both performance and cost concerns
2Quantity of substance
If a disk-based low-cost vector graph index (e.g., DiskANN) is used, then storage costs are reduced, but real-time search performance deteriorates
Solution Approach 1:
The patent segments the index into two distinct parts: a memory-based HNSW index for real-time search operations and a disk-based DiskANN index for historical data storage. This segmentation allows each index type to be optimized for its specific function, with the HNSW index handling incremental real-time queries and the DiskANN index managing bulk historical data, thereby resolving the contradiction between real-time performance and storage costs
Solution Approach 2:
The patent merges the capabilities of memory-based and disk-based indexes into a unified hybrid search system. The search process combines results from both the HNSW index (for real-time performance) and the DiskANN index (for comprehensive historical coverage), allowing the system to leverage the strengths of both approaches while mitigating their individual weaknesses
3Reliability
If vector data is frequently updated in real-time, then data freshness is improved, but index construction and maintenance complexity increases
Solution Approach 1:
The patent segments the index management into two independent streams: incremental updates to the memory-based HNSW index for real-time data freshness, and periodic snapshots to the disk-based DiskANN index for historical preservation. This segmentation simplifies the complexity by allowing each index type to be updated independently according to its own optimization criteria, rather than requiring complex coordination between memory and disk operations
Solution Approach 2:
The patent implements periodic snapshots that occur at predetermined intervals or when data volume thresholds are reached. This periodic action converts the continuous complexity of real-time index synchronization into manageable discrete operations, where the memory index handles continuous real-time updates and the disk index is periodically refreshed, thereby reducing overall system complexity
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of this specification provide methods and apparatuses for writing and searching vector data in a vector database. The method for writing vector data into a vector database includes: receiving a write request for target vector data; adding index update information corresponding to the target vector data to a dynamic first index of a vector database in a memory table, where the first index is a memory-based real-time read-write vector graph index; when the memory table is refreshed to a disk, storing the index update information of the dynamic first index in the memory table into a distributed file system to obtain a static first index; and converting the static first index in the distributed file system into a second index, where the second index is a disk-based low-cost vector graph index. Both real-time performance and storage costs can be considered.