HNSW Snapshot Indexing for Vector Versioning and Real-Time Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing vector databases face challenges in efficiently managing and processing high-volume vector data with real-time indexing, searching, and versioning, particularly in the context of evolving embedding models, leading to inefficiencies in response times and resource utilization.
Innovation Solution
The implementation of a system and method for creating and maintaining multiple snapshots of an HNSW index, utilizing a shared journal to track changes and combine query results across snapshots, along with flexible vector data types and storage formats that support versioning and efficient distance computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If multiple snapshots of HNSW index are created to enable versioning and real-time indexing, then adaptability and reliability are improved, but device complexity increases
Solution Approach 1:
The HNSW index is segmented into multiple snapshots, each representing a point-in-time version of the index. This allows the system to maintain multiple versions simultaneously without merging them into a single complex structure. Each snapshot is an independent, complete HNSW index that can be queried separately, enabling versioning while keeping individual snapshot management simple.
Solution Approach 2:
Instead of modifying a single index in place, the system creates copies of the HNSW index at different points in time. Each snapshot is a complete copy of the index state at a specific moment, allowing versioning without complex in-place updates. This copying approach simplifies management by treating each version as an independent entity.
2Productivity
If real-time indexing is implemented to handle evolving embedding models, then productivity is improved, but loss of time increases due to index rebuild operations
Solution Approach 1:
The system performs preliminary actions by creating snapshots before index rebuild operations are needed. When a new embedding model is deployed or data is significantly updated, a snapshot is created first to preserve the current state, then the index is rebuilt. This allows continuous operations while preparing for changes, reducing the time impact of rebuild operations.
Solution Approach 2:
The system dynamically switches between snapshots based on operational needs. When real-time indexing is required, the system can quickly switch to a recent snapshot rather than rebuilding the entire index. This dynamic approach allows the system to adapt to changing requirements without the time cost of full index rebuilds.
3Reliability
If shared journal is used to track changes across snapshots, then reliability is improved, but device complexity increases
Solution Approach 1:
The shared journal acts as an intermediary between the HNSW index and the versioning mechanism. Instead of embedding versioning logic throughout the entire index structure, changes are logged in a separate shared journal that tracks modifications across snapshots. This intermediary approach simplifies the overall system by centralizing change tracking in a dedicated structure.
4Adaptability or versatility
If multiple snapshots are maintained for versioning, then adaptability is improved, but loss of energy increases due to storing multiple index copies
Solution Approach 1:
The system implements a strategy of discarding old snapshots that are no longer needed and recovering storage resources for them. When a snapshot becomes sufficiently old or when the most recent snapshot contains all necessary data, older snapshots are discarded to free up storage space. This allows the system to maintain a manageable number of snapshots for versioning while minimizing energy consumption for storage.
Data Source
AI summary
Techniques for generating and leveraging a multi-snapshot HNSW index are provided. In one technique, multiple snapshots of an HNSW index that was generated based on a vector table comprising an initial set of vectors are stored. Each of snapshots has a different build time and is associated with a shared journal that stores a plurality of changes to a plurality of vectors in the vector table. A latest snapshot of the plurality of snapshots is identified. The shared journal is identified. It is determined, based on one or more criteria, whether to generate a new snapshot based on the latest snapshot and the shared journal. In another technique, in response to receiving a vector query that is associated with a timestamp, a particular snapshot is selected from the snapshots based on the timestamp. The vector query is executed against the particular snapshot and the shared journal.