Incremental kNN Neighbor List Updates for High-Volume Data Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for constructing and updating k-nearest neighbor (kNN) graphs are inefficient, especially for high velocity and high volume data streams, leading to increased costs and potential accuracy loss due to noise introduction from excessive similarity comparisons.
Innovation Solution
An orthogonal transform based indexing system is used to efficiently compute nearest neighbors, with a local search to update likely changed data objects, and a statistical distribution is employed to selectively update ranked neighbor lists based on a determined threshold, reducing computational time and noise.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing methods are used to construct and update kNN graphs for high velocity and high volume data streams, then comprehensive neighbor relationships can be maintained, but computational efficiency deteriorates and processing time increases
Solution Approach 1:
The patent segments the data stream processing into batches and uses incremental updates rather than complete re-construction. The kNN graph is updated incrementally as new data arrives, dividing the computational task into manageable segments that can be processed efficiently without sacrificing the comprehensiveness of neighbor relationships.
Solution Approach 2:
The patent performs preliminary indexing and organization of data objects before conducting similarity searches. By pre-processing and structuring the data in advance, the system reduces the computational burden during actual kNN graph construction and updates, improving overall processing efficiency while maintaining relationship accuracy.
2Measurement precision
If existing methods update all neighbor lists for every new data object, then accuracy is maintained, but computational cost and time increase significantly
Solution Approach 1:
The patent applies local quality by selectively updating only those neighbor lists that are likely to change based on the new data object's characteristics. Instead of uniformly updating all neighbor lists, the system identifies and updates only the relevant local subsets, maintaining accuracy where needed while reducing unnecessary computations elsewhere.
Solution Approach 2:
The patent employs partial action by updating a subset of neighbor lists rather than all of them. The system determines which neighbor lists require updates based on similarity thresholds and the new object's properties, performing just enough updates to maintain accuracy without the excessive computation of comprehensive updates.
3Measurement precision
If excessive similarity comparisons are performed to maintain accuracy, then neighbor relationships are precise, but noise is introduced and processing becomes inefficient
Solution Approach 1:
The patent changes the parameter of similarity comparison by using efficient similarity measures and thresholds that reduce unnecessary comparisons. By adjusting the similarity threshold parameter and using optimized comparison methods, the system maintains precise neighbor relationship identification while minimizing the introduction of noise from excessive or irrelevant comparisons.
4Quantity of substance
If conventional systems process high volume data streams, then data can be handled, but processing capacity is exceeded and performance degrades
Solution Approach 1:
The patent segments high volume data streams into manageable batches and processes them incrementally. By dividing the large data volume into smaller segments that can be handled within conventional system capacity, the system maintains processing efficiency while accommodating high data volumes through batched incremental updates.
Solution Approach 2:
The patent implements dynamic processing where the system adapts its processing rate and update frequency based on incoming data flow characteristics. This dynamic approach allows the system to handle variable data volumes efficiently, adjusting computational resources to match the actual processing needs without consistently exceeding capacity.
Data Source
AI summary
Incremental automatic update of ranked neighbor lists based on k-th nearest neighbors is disclosed. One example is a system including an indexing module to retrieve an incoming data stream, and retrieve ranked neighbor lists for received data objects. An evaluator determines similarity measures between the received data objects and their respective k-th nearest neighbors. A threshold determination module determines a statistical distribution based on the determined similarity measures, and a threshold based on the statistical distribution. The evaluator determines additional similarity measures between a new data object in the data stream and the received data objects. A neighbor update module automatically selects a sub-plurality of the received data objects by comparing the additional similarity measures to the threshold, and determines, for each selected data object, if the respective retrieved neighbor list is to be incrementally updated based on neighborhood comparisons for the new data object and the selected data object.