Vector Database Graph Updates with Volatile-Memory Batching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector database management systems face challenges in efficiently managing and searching large-scale vector data, particularly due to the increasing demand for storage capacity in volatile memory as the dataset scales up.
Innovation Solution
A management method and database device that store a first vector database in a storage device and generate a second vector database in volatile memory, combining directed graphs from both to reduce the amount of data written in the storage device by adding multiple nodes in a single operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If vector data is stored in a storage device using DiskANN, then storage capacity is improved, but the amount of data written to the storage device increases, causing degradation due to frequent write cycles
Solution Approach 1:
The patent divides the vector database into two segments: a first vector database stored in the storage device and a second vector database stored in volatile memory. This segmentation allows frequent updates to be performed on the second database in volatile memory without affecting the stability of the first database in the storage device, thereby reducing write cycles to the storage device and preventing data degradation while maintaining storage capacity.
Solution Approach 2:
The second vector database in volatile memory acts as an intermediary between update operations and the first vector database in the storage device. Updates are first applied to the second database, and only consolidated updates are written to the first database, reducing the frequency of write operations to the storage device and minimizing data degradation from frequent writes.
2Quantity of substance
If multiple nodes are added to the vector database, then storage capacity is improved, but the amount of data written to the storage device increases
Solution Approach 1:
The patent performs preliminary actions by adding multiple nodes to the second vector database in volatile memory before consolidating and writing them to the first vector database in the storage device. This batch processing approach reduces the number of write operations to the storage device, minimizing data loss from frequent write cycles while achieving the desired storage capacity expansion.
3Speed
If vector database is stored in volatile memory, then search speed is improved, but storage capacity is limited
Solution Approach 1:
The patent implements a nested structure where the second vector database in volatile memory is nested within the overall vector database system that also includes the first vector database in the storage device. This nested architecture allows the system to utilize the fast access speed of volatile memory for frequently accessed data while leveraging the large storage capacity of the storage device for the complete dataset, achieving both speed and capacity goals.
Data Source
AI summary
A first vector database stored in a storage device includes a group of first information pieces each indicating one of a plurality of first vectors that correspond to a plurality of nodes of a first directed graph. A management method is capable of reducing the amount of data written in the storage device during updates. The method includes, while generating a second directed graph that includes two or more nodes corresponding to two or more second vectors, generating, in a volatile memory, a second vector database in which a group of third information pieces each indicating one of the two or more second vectors, is recorded. The method further includes combining the first directed graph with the second directed graph by updating one information piece of the group of first information pieces and the group of third information pieces, and storing the second vector database in the storage device.