Neighbor Graph Vector Index Checkpointing for Consistent Reloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for persisting and restoring in-memory neighbor graph vector indexes, such as HNSW indexes, are CPU-intensive and result in inconsistent query results across nodes in a cluster, leading to significant processing overhead and suboptimal performance during instance restarts and index duplication.
Innovation Solution
Implementing disk checkpoints for HNSW indexes to create and restore neighbor graph vector indexes, using incremental or full checkpoints based on checkpoint factors, and employing checkpoint policies to minimize CPU usage and ensure consistent index duplication across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If in-memory neighbor graph vector indexes are restored using existing methods, then the index can be reloaded after instance restart, but the process is CPU-intensive and results in inconsistent query results across cluster nodes
Solution Approach 1:
The patent applies preliminary action by persisting the in-memory neighbor graph vector index to disk before instance shutdown. This allows the index to be restored from the persisted state rather than rebuilt from scratch, reducing CPU usage during restart and ensuring consistency across cluster nodes. The specific implementation involves writing the index structure and vector data to disk storage before shutdown, then reading it back during instance restart.
2Productivity
If in-memory neighbor graph vector indexes are persisted to disk, then index restoration is enabled, but the persisting process requires significant processing overhead
Solution Approach 1:
The system performs preliminary persistence of the index to disk before instance shutdown, capturing the index state while it is still in memory. This approach enables fast restoration by loading from the persisted state rather than rebuilding, significantly improving index reload speed while managing processing overhead through efficient persistence mechanisms.
Solution Approach 2:
The patent uses copying by creating a persistent copy of the in-memory index structure and vector data to disk storage. This copy can then be rapidly loaded during restoration without requiring the original in-memory data to be rebuilt, thereby improving reload speed while the copying process is optimized to minimize processing overhead.
3Reliability
If index duplication is performed across cluster nodes, then query consistency can be maintained, but redundant distance computations increase processing overhead
Solution Approach 1:
The patent applies preliminary action by persisting the index to disk before duplication to other cluster nodes. This allows receiving nodes to load the pre-persisted index state directly, avoiding redundant distance computations and processing overhead while maintaining query consistency across the cluster.
Solution Approach 2:
The system uses copying by transferring the persisted index data from the primary node to receiving nodes in the cluster. This approach maintains query consistency across all nodes while minimizing processing overhead, as the copied data is already in its final persisted state and does not require recalculation or redundant distance computations.
Data Source
AI summary
Techniques persist and restore in-memory neighbor graph vector indexes that include a vertex identifier to vector mapping and include a neighbor graph of vector neighbor vertices. At least one neighbor graph vector index checkpoint factor can be identified. A determination can be made as to whether to generate a full neighbor graph vector index checkpoint or an incremental neighbor graph vector index checkpoint based on the checkpoint factor.