Vector Indexing via HNSW Segmentation for Billion-Scale Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently indexing and retrieving embedding vectors at an above-billion scale for fast high-recall retrieval, particularly in facial recognition systems.
Innovation Solution
The method involves distributing batches of vectors to nodes, deduplicating vectors based on similarity scores, generating a vector index using a Hierarchical Navigable Small Worlds (HNSW) data structure, and iteratively refining the index until only one node remains, thereby creating a scalable and efficient indexing system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exhaustive search through entire database is performed, then retrieval completeness is improved, but computational efficiency deteriorates
Solution Approach 1:
The patent segments the database into multiple batches distributed across different nodes, and further segments the indexing process into multiple passes. Each pass processes a subset of data and builds upon the previous index, allowing the system to search through the entire database systematically without overwhelming computational resources in a single exhaustive search.
Solution Approach 2:
The patent performs preliminary deduplication of vectors before indexing, removing duplicate vectors and their associated metadata in advance. This preliminary action reduces the effective search space and computational burden during subsequent retrieval operations, maintaining completeness while improving efficiency.
2Productivity
If duplicate vectors are removed, then indexing efficiency is improved, but information completeness deteriorates
Solution Approach 1:
The patent implements a feedback mechanism where the deduplication process is monitored and adjusted. After each pass of deduplication, the system evaluates the impact on information completeness and adjusts the deduplication threshold or methodology accordingly, ensuring that only redundant information is removed while preserving essential data diversity.
Solution Approach 2:
The patent applies partial deduplication rather than complete deduplication. It removes only the most redundant duplicate vectors while retaining representative samples that maintain information completeness. This partial action approach balances indexing efficiency with information preservation by not being overly aggressive in removal.
3Measurement precision
If multi-pass processing is implemented, then indexing accuracy is improved, but processing time deteriorates
Solution Approach 1:
The patent segments the multi-pass processing into distinct phases, each handling specific aspects of indexing accuracy. Early passes focus on basic deduplication and structure building, while subsequent passes refine the index with more computationally intensive operations. This segmentation allows accuracy improvement without requiring all operations to complete simultaneously.
Solution Approach 2:
The patent performs preliminary indexing operations in earlier passes that establish the basic framework and structure. These preliminary actions complete faster and set up the foundation for subsequent refinement passes, reducing the overall time required compared to performing all accuracy-enhancing operations from the beginning.
4Adaptability or versatility
If vector distribution across nodes is performed, then system scalability is improved, but system complexity deteriorates
Solution Approach 1:
The patent segments the vector database into batches distributed across multiple nodes, with each node handling a specific subset of data. This segmentation enables horizontal scalability by allowing the system to add more nodes to handle larger datasets without requiring complete redesign of the indexing mechanism, while the distributed nature naturally manages complexity through localized processing.
Data Source
AI summary
This disclosure provides novel methods and systems for indexing embedding vectors representing disjoint classes at above-billion scale for fast high-recall retrieval. The disclosed methods and systems solved above-billion scale image (e.g., face image) search problems by building a new vector database, the new Nearest Neighbor Database (NNDB) based on a novel architecture for indexing vectors.