SONG ANN Search Decoupling for GPU Parallelism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graph-based approximate nearest neighbor (ANN) search methods struggle to effectively utilize powerful hardware tools like GPUs due to high execution dependencies, limiting their scalability and performance on modern processors.
Innovation Solution
The SONG (Search ON Graph) system decouples the graph-based ANN search into three stages: candidates locating, bulk distance computation, and data structures maintenance, utilizing open addressing hash tables, Bloom filters, and optimizations like bounded priority queues and selected insertion to parallelize distance computation and reduce memory consumption, enabling efficient processing on GPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If graph-based ANN search methods are used, then search accuracy is improved, but execution dependency increases preventing GPU adaptation
Solution Approach 1:
The patent segments the graph-based ANN search into three independent stages: candidate generation, candidate scoring, and result aggregation. Each stage can be executed independently on GPU threads without execution dependencies, enabling parallel processing while maintaining search accuracy through the preserved graph-based candidate selection logic.
Solution Approach 2:
The patent extracts the execution dependency bottleneck from the traditional graph-based search algorithm by separating candidate generation (which requires graph traversal) from candidate scoring (which can be parallelized). This extraction allows the scoring stage to be implemented as independent GPU kernels that do not rely on sequential graph traversal.
2Productivity
If traditional ANN search is used, then memory consumption is high, but processing speed is slow
Solution Approach 1:
The patent applies local quality by loading only the necessary graph index structures and candidate data into GPU memory rather than the entire dataset. The system selectively loads graph nodes and their neighbors into memory, reducing memory consumption while maintaining processing speed through efficient use of available GPU memory resources.
Solution Approach 2:
The patent implements a nested memory structure where the graph index is nested within the main dataset structure. The graph index nodes contain references to neighbor nodes, creating a hierarchical nesting that allows the system to process large datasets by only loading relevant portions into GPU memory at each search stage.
3Measurement precision
If graph-based methods are used, then search quality is improved, but hardware utilization is poor
Solution Approach 1:
The patent implements dynamic parallel execution where GPU threads are dynamically allocated to different search operations based on the data distribution and search parameters. The system dynamically adjusts the number of threads per block, the batch size for candidate scoring, and the memory allocation for graph index structures to maximize hardware utilization while preserving search quality.
Solution Approach 2:
The patent ensures continuous utilization of GPU hardware by implementing pipelined execution where candidate generation, scoring, and aggregation stages overlap in time. While one set of threads is generating candidates for the next search batch, other threads are scoring previous candidates and aggregating results, maintaining continuous hardware utilization without compromising search quality.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Approximate nearest neighbor (ANN) searching is a fundamental problem in computer science with numerous applications in area such as machine learning and data mining. For typical graph-based ANN methods, the searching method is executed iteratively, and the execution dependency prohibits graphics processor unit (GPU)/GPU-type processor adaptations. Presented herein are embodiments of a novel framework that decouples the searching on graph methodology into stages, in order to parallel the performance-crucial distance computation. Furthermore, in one or more embodiments, to obtain better parallelism on GPU-type components, also disclosed are novel ANN-specific optimization methods that eliminate dynamic memory allocations and trade computations for less memory consumption. Embodiments were empirically compared against other methods, and the results confirm the effectiveness.