Distributed K-Nearest Neighbor Partitioning for Lower Query Overhead
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for performing K-nearest neighbor (K-NN) queries in large spatial datasets are inefficient in distributed or parallel computing environments due to redundant processing, excessive data communication, and the need to repeatedly expand spatial extents, leading to increased computation and overheads.
Innovation Solution
Pre-processing the spatial dataset by partitioning the query space and calculating an object range for each partition, ensuring that the K candidate objects within this range are included, allowing efficient distribution of partitions across multiple processors for K-NN queries without extensive data communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the spatial extent is expanded repeatedly to retrieve more objects, then the completeness of K-NN results is improved, but the computation time and processing overhead increase significantly
Solution Approach 1:
The patent pre-computes and stores the spatial extent (bounding box) for each partition during the partitioning phase. This preliminary action eliminates the need for repeated spatial extent calculations and object re-retrieval operations during query processing, thereby resolving the contradiction between result completeness and computation time by preparing necessary spatial information in advance.
Solution Approach 2:
The patent divides the spatial dataset into multiple partitions, each with pre-calculated spatial extents. This segmentation allows the system to retrieve objects from specific partitions without repeatedly expanding the spatial extent across the entire dataset, thus improving query efficiency while maintaining result completeness.
2Productivity
If queries are processed in a distributed environment with multiple computing nodes, then the processing capacity is improved, but the data communication overhead between nodes increases
Solution Approach 1:
The patent segments both the query dataset and the object dataset into the same partitions distributed across multiple computing nodes. This segmentation strategy ensures that each node processes only its local partition data, minimizing the need for data communication between nodes and reducing communication overhead while maintaining high processing capacity.
Solution Approach 2:
Each computing node is assigned specific partitions with pre-calculated spatial extents, enabling local processing without requiring global data access. This local quality approach allows nodes to independently process queries using only their local partition data, significantly reducing inter-node communication requirements.
3Measurement precision
If the entire object set is passed to each computing node for processing, then the accuracy of K-NN results is improved, but the memory requirements and processing overhead become infeasible for very large datasets
Solution Approach 1:
The patent segments the object set into multiple partitions distributed across computing nodes, with each partition having a pre-calculated spatial extent. This segmentation allows each node to process only its local partition data, dramatically reducing memory requirements while maintaining result accuracy through the use of pre-computed spatial boundaries that ensure all relevant objects are included.
Solution Approach 2:
The spatial extents for each partition are pre-computed and stored before query processing. This preliminary action enables nodes to accurately determine which objects belong to each partition without requiring the entire object set in memory, thus maintaining result accuracy while reducing memory requirements.
Data Source
AI summary
Methods and systems for performing K-nearest neighbour (K-NN) queries on a spatial dataset in a distributed or parallel computing system are described herein, where the spatial data comprises query data and object data. The spatial dataset is pre-processed, whereby the query space (i.e., the spatial extent of the query data) is partitioned according to a particular partition scheme. For each partition, an area (defined as the object range of the partition) is calculated based on the geometry of the partition and a set of K candidate objects allocated to that partition, with all objects within the object range defining the extent of any subsequent K-NN queries performed on that partition. When the query dataset is subsequently partitioned using the same partition scheme to perform a K-NN query, the pre-processed object range information can be used to retrieve the objects to be queried for each partition.