Vector Retrieval Using Compression Graphs for Shared Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector retrieval solutions suffer from high delay and high throughput issues when applied in shared storage environments, leading to poor adaptation and user experience.
Innovation Solution
A vector retrieval method and apparatus that stores a graph index based on original and compression vectors in a first-level storage resource, where data access performance is better than in a second-level resource, allowing for decoupled nearest neighbor search processes, reducing synchronous data access operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If vector retrieval is performed using shared storage, then data availability and accessibility are improved, but retrieval delay increases and throughput decreases
Solution Approach 1:
The patent segments the storage system into a first-level storage resource (fast storage) and a second-level storage resource (shared storage). The graph index is divided into two parts: compression vectors stored in the first-level storage and original vectors stored in the second-level storage. This segmentation allows the system to leverage both the high availability of shared storage and the low latency of fast storage, resolving the contradiction between data availability and retrieval delay.
Solution Approach 2:
The patent introduces compression vectors as an intermediary between the query and the original vectors in shared storage. These compression vectors are stored in the first-level storage resource and serve as a fast access layer. The retrieval process first queries the compression vectors to identify candidate original vectors, then retrieves only those candidates from shared storage. This intermediary mechanism reduces the amount of data accessed from shared storage, thereby reducing retrieval delay while maintaining data availability.
2Adaptability or versatility
If all vectors are stored in shared storage to improve accessibility, then storage efficiency is improved, but data access performance deteriorates
Solution Approach 1:
The patent segments the vector data into two components stored in different storage resources: compression vectors in the first-level storage resource and original vectors in the second-level shared storage resource. This segmentation enables the system to maintain high accessibility through shared storage while achieving high data access performance through fast storage for the index structure.
Solution Approach 2:
The patent applies local quality by storing different types of data in storage resources matched to their access patterns. Compression vectors, which require frequent and fast access during query processing, are stored in the first-level storage resource with high access performance. Original vectors, which are accessed less frequently and can tolerate higher latency, are stored in the second-level shared storage resource. This localized optimization of storage quality resolves the contradiction between accessibility and access performance.
Data Source
AI summary
A computer-implemented method for vector retrieval includes, in response to a query vector, obtaining, from a first-level storage resource, at least one target compression vector having a highest similarity with the query vector based on a graph index constructed based on an original vector and a compression vector. After obtaining the at least one target compression vector, obtaining, from a second-level storage resource, at least one target original vector corresponding to the at least one target compression vector. Separately, a similarity between the query vector and the at least one target original vector is calculated; the at least one target original vector is sorted based on the calculated similarity; and, based on the sorted at least one target original vector, a vector retrieval result corresponding to the query vector is determined.