Neighbor List Update Selection via Dynamic Thresholding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for constructing and updating k-nearest neighbor (kNN) graphs are inefficient, especially for high velocity and high volume data streams, leading to high construction costs and anomalies due to similarity measure metric distortion.
Innovation Solution
An orthogonal transform based indexing system is used to efficiently compute nearest neighbors, combined with a local search to update likely changed data objects, and an automatic selection mechanism for incrementally updating neighbor lists based on a probability distribution and threshold determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional methods are used to construct and update kNN graphs for high velocity and high volume data streams, then complete neighbor list updates can be performed, but the processing time and computational complexity become excessively high
Solution Approach 1:
The patent segments the neighbor list update process by dividing all data objects into two distinct groups: those whose neighbor lists require updating and those that do not. This segmentation is achieved by comparing similarity measures against a dynamically determined threshold, allowing the system to selectively update only the necessary portion of neighbor lists rather than performing complete updates across all data objects, thereby significantly reducing processing time while maintaining update reliability.
Solution Approach 2:
The patent applies partial action by performing updates only on the subset of data objects that meet the threshold criterion, rather than executing excessive complete updates on all data objects. The threshold-based selection mechanism ensures that updates are performed partially but precisely where needed, avoiding the time-consuming redundant updates while maintaining the reliability of neighbor list accuracy for affected objects.
2Measurement precision
If complete neighbor list updates are performed for all data objects, then accuracy is maintained, but the computational complexity and resource consumption increase significantly
Solution Approach 1:
The patent segments the computational workload by dividing data objects into two categories based on a threshold comparison: those requiring neighbor list updates and those that do not. This segmentation reduces computational complexity by focusing resources only on the necessary subset of objects while maintaining measurement precision for those objects through selective updating.
Solution Approach 2:
The patent introduces a dynamic threshold parameter that changes based on the distribution of similarity measures. By adjusting this parameter, the system optimizes the balance between maintaining accuracy (measurement precision) and reducing computational complexity. The threshold acts as a control parameter that determines the extent of updates needed, allowing the system to adapt to different data distributions and maintain precision while managing computational resources efficiently.
3Ease of manufacture
If traditional indexing methods are used for high-dimensional data, then simple implementation is achieved, but the system cannot maintain efficiency and accuracy with high-dimensional streaming data
Solution Approach 1:
The patent changes the parameter of threshold determination from static to dynamic, where the threshold is automatically determined based on the distribution of similarity measures in high-dimensional space. This parameter change enables the system to maintain efficiency and accuracy with high-dimensional streaming data by adapting to the specific characteristics of the data distribution, while the overall implementation remains relatively simple through the use of standard statistical methods.
Solution Approach 2:
The indexing system performs self-service by automatically determining the threshold value based on the similarity measure distribution without requiring manual intervention or complex configuration. This self-adjusting mechanism allows the system to efficiently handle high-dimensional data streams by adapting to their specific characteristics, maintaining both simplicity of implementation and high processing efficiency through autonomous parameter optimization.
Data Source
AI summary
Automatic selection of neighbor lists to be incrementally updated is disclosed. One example is a system including an indexing module to receive an incoming data stream, and retrieve neighbor lists for received data objects. An evaluator determines similarity measures between pairs of the received data objects. A threshold determination module determines distributions of order statistics based on the determined similarity measures and retrieved neighbor lists, and a threshold based on the distributions of order statistics. The evaluator determines additional similarity measures between a new data object in the data stream and the received data objects. A neighbor update module automatically selects a sub-plurality of the received data objects by comparing the additional similarity measures to the threshold, and determines, for each selected data object, if the respective retrieved neighbor list is to be incrementally updated based on neighborhood comparisons for the new data object and the selected data object.