Vector Index Memory Allocation by Node Importance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector index storage systems are inefficient and do not effectively manage the importance of data nodes, leading to suboptimal system response times and resource utilization.
Innovation Solution
A method and device for optimizing vector index storage by generating a vector index structure, calculating node importance using a priority queue, and deleting nodes with low importance from memory, employing a weighted similarity calculation based on time and frequency of access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all vector indexes are stored in memory, then search quality is maintained, but memory usage increases and system response time deteriorates
Solution Approach 1:
The patent applies local quality by differentiating the storage treatment of vector indexes based on node importance. High-importance nodes (with importance > threshold) are stored in memory for fast access, while low-importance nodes are stored on disk. This selective storage strategy maintains search quality for critical data while optimizing memory usage, directly resolving the contradiction between maintaining search quality and reducing memory consumption.
2Quantity of substance
If vector indexes are deleted from memory to save space, then memory usage decreases, but system response time increases
Solution Approach 1:
The patent differentiates between high-importance and low-importance nodes, storing only the latter on disk while keeping the former in memory. This ensures that deletions from memory are limited to low-importance nodes, minimizing the impact on system response time while achieving memory optimization. The selective approach prevents uniform deletion that would uniformly degrade performance.
Solution Approach 2:
The patent performs preliminary calculation of node importance values before making storage decisions. By pre-calculating importance metrics (such as access frequency, recency, and data criticality) and establishing thresholds, the system determines in advance which nodes should be deleted from memory, avoiding runtime performance degradation and enabling proactive memory management.
3Quantity of substance
If node importance calculation is performed, then storage optimization improves, but computational complexity increases
Solution Approach 1:
The patent implements partial action by calculating node importance only for nodes that are candidates for deletion, rather than continuously recalculating importance for all nodes in the vector index. The system monitors memory usage and triggers importance calculation only when deletion is needed, reducing unnecessary computational overhead while still achieving storage optimization goals.
Data Source
AI summary
Provided are a server, method, and computer-readable recording medium for allocating a storage space for a vector index. The method includes generating a vector index structure of data points, generating a priority queue based on importance of nodes, calculating the importance of the nodes using the vector index structure, and deleting a vector index corresponding to a node having importance of a preset reference or less from a memory. According to this method, it is possible to improve the response of a system by optimizing the storage of vector indexes.