SSD Graph Index Partitioning for Vector Retrieval Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing vector graph index algorithm based on solid-state drives (SSDs) suffers from inefficiency due to reading redundant data, as each page read includes both retrieval and adjacent index nodes, leading to data waste and reduced performance in data retrieval processes.
Innovation Solution
The algorithm re-divides the graph index such that index nodes and their neighbor nodes are stored in the same partition on the SSD, allowing simultaneous reading of relevant data and processing of unprocessed data from previous accesses, thereby improving data reading and retrieval efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the graph index is divided into partitions on SSD with adjacent index nodes stored together, then data reading efficiency is improved by reading only necessary nodes, but the index reorganization complexity increases
Solution Approach 1:
The graph index is segmented into multiple partitions based on node access patterns and spatial relationships. Each partition contains index nodes that are likely to be accessed together, allowing the system to read only relevant partitions during retrieval operations rather than scanning entire pages, thus improving data reading efficiency while managing reorganization complexity through systematic segmentation strategies
Solution Approach 2:
The index reorganization is performed in advance during system initialization or idle periods, pre-arranging index nodes into optimal partitions before actual retrieval operations begin. This preliminary action separates the complex reorganization task from the time-critical retrieval process, allowing efficiency improvements without impacting operational complexity during active use
2Measurement precision
If index nodes and their neighbor nodes are stored in the same partition, then retrieval accuracy is improved by reducing redundant data, but the storage space utilization decreases
Solution Approach 1:
Redundant index nodes and their associated data are extracted from the storage structure and eliminated. By storing only the necessary neighbor nodes in each partition and removing duplicate or unnecessary copies, the system achieves higher retrieval accuracy through reduced redundant data while optimizing storage space utilization by eliminating wasted capacity
3Speed
If unprocessed data from previous accesses is processed synchronously, then overall retrieval speed is improved, but the processing complexity increases
Solution Approach 1:
The system maintains continuous processing by synchronously handling unprocessed data from previous accesses during idle periods or while waiting for I/O operations to complete. This continuous utilization of processing resources improves overall retrieval speed by eliminating idle time, while the processing complexity is managed through systematic queue management and prioritization strategies that organize synchronous tasks in a controlled manner
Data Source
AI summary
A retrieval apparatus, a method, and a storage medium are disclosed. The retrieval apparatus includes a memory and a processor, wherein the processor is configured to acquire a retrieval request, and the retrieval request includes the query vector; according to the query vector, generate and execute a first access request corresponding to the first memory according to the first graph index and the index node in the candidate pool; determine and process the data acquired by the first access request corresponding to the first memory, corresponding to the absence of redundant data in the previous storage pool, and store the result in the result pool; and output the data in the result pool corresponding to the candidate pool does not include unreachable index nodes. The retrieval apparatus improves the efficiency of the data reading process and further improves the efficiency of the retrieval process.