Hybrid Vector Database Indexing for Real-Time Search and Lower Storage Cost
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for writing and searching vector data in vector databases fail to balance real-time performance and storage costs, particularly in large-scale scenarios.
Innovation Solution
A composite index solution is employed, utilizing a memory-based real-time read-write vector graph index (HNSW) for incremental data and a disk-based low-cost vector graph index (DiskANN) for historical data, with index conversion and storage in a distributed file system to manage large-scale vector data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a memory-based real-time read-write vector graph index (HNSW) is used for writing vector data, then real-time performance is improved, but storage costs increase
Solution Approach 1:
The patent segments the index storage into two parts: a memory-based HNSW index for real-time write operations and a disk-based DiskANN index for historical data storage. This segmentation allows the system to use expensive memory storage only when necessary for real-time performance while using cheaper disk storage for historical data, thereby resolving the contradiction between real-time performance and storage costs.
Solution Approach 2:
The patent changes the storage medium parameter from purely memory-based to a hybrid memory-disk architecture. By adjusting the storage location parameter (memory vs. disk) based on data recency and access patterns, the system optimizes both real-time performance for recent data and storage cost efficiency for historical data.
2Quantity of substance
If a disk-based low-cost vector graph index (DiskANN) is used for storing historical data, then storage costs are reduced, but real-time search performance deteriorates
Solution Approach 1:
The patent segments the search operation into two paths: one using the memory-based HNSW index for real-time search performance and another using the disk-based DiskANN index for historical data search. By maintaining both index types and routing queries appropriately, the system achieves both cost efficiency and performance.
Solution Approach 2:
The patent performs preliminary action by pre-converting memory-based HNSW indexes to disk-based DiskANN indexes when data becomes historical. This conversion is done in advance so that when data moves from hot to cold storage, the appropriate index type is already in place, ensuring both cost savings and acceptable search performance.
3Adaptability or versatility
If vector data is stored in a distributed file system with index conversion, then storage scalability is improved, but system complexity increases
Solution Approach 1:
The patent introduces an intermediary conversion mechanism that automatically transforms HNSW indexes to DiskANN indexes when data is flushed from memory to disk. This intermediary process manages the complexity internally, allowing the distributed file system to scale efficiently while the complexity of index management is handled by the automated conversion layer.
Solution Approach 2:
The system implements self-service by automatically performing index conversion from HNSW to DiskANN format when data is moved to the distributed file system, without requiring manual intervention. This automation handles the complexity internally, enabling scalable storage while keeping the user interface simple.
Data Source
AI summary
The present disclosure provides methods and apparatuses for writing and searching vector data in a vector database. An example method includes receiving a write request for writing target vector data into a vector database, adding index update information corresponding to the target vector data to a first index of the vector database in a memory table, where the first index is a memory-based real-time read-write vector graph index, in response to flushing the memory table to a disk, writing the index update information of the first index in the memory table into a distributed file system, and converting the first index in the distributed file system into a second index, where the second index is a disk-based vector graph index.