Orthogonal Transform Indexing for Incremental kNN Graph Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for constructing and updating k-nearest neighbor (kNN) graphs are inefficient, especially for high velocity and high volume data streams, due to high construction costs and anomalies caused by distance metric distortion in hashing techniques.
Innovation Solution
An orthogonal transform based indexing system is used to efficiently compute nearest neighbors in a kNN graph, applying a hash transform to associate data objects with ordered hash positions, and incrementally updating the graph by selecting neighbors based on neighborhood comparisons without direct similarity/distance calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If direct similarity/distance calculation methods are used to construct kNN graphs, then neighbor relationships can be accurately determined, but computational complexity increases to O(n^2) which is inefficient for high volume data streams
Solution Approach 1:
The patent introduces an orthogonal transform (e.g., Walsh-Hadamard transform) as an intermediary mechanism that maps data objects to hash positions. This transform serves as a mediator between the original data space and the neighbor search space, enabling efficient retrieval without direct pairwise distance calculations. The transform coefficients act as intermediaries that preserve neighbor relationships while enabling O(mn) complexity search.
Solution Approach 2:
The patent changes the parameter representation of data objects by applying orthogonal transforms. Instead of working with raw data vectors requiring O(n^2) comparisons, the system transforms data into a different parameter space (transform coefficients) where neighbor relationships can be determined more efficiently through hash-based indexing and comparison of transformed representations.
2Productivity
If hashing techniques are used to reduce comparison space, then processing speed improves, but distance metric distortion causes anomalies in neighbor determination
Solution Approach 1:
The patent replaces traditional hashing mechanisms (which rely on hash functions that can cause metric distortion) with an orthogonal transform-based indexing system. This substitution eliminates the mechanical hashing process that causes distance metric distortion, while maintaining the benefits of reduced comparison space through structured transform coefficients.
Solution Approach 2:
The patent changes from using hash function outputs to using orthogonal transform coefficients as the indexing parameter. This parameter change preserves the geometric relationships and distance metrics better than traditional hashing, as orthogonal transforms maintain energy conservation and structural properties that prevent metric distortion.
3Reliability
If incremental updates are performed for each new data object in streaming data, then the graph remains current, but construction costs increase considerably over time
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing orthogonal transform coefficients for incoming data objects as they arrive in the stream. This preliminary transformation and indexing allows for efficient incremental updates later, as new objects can be quickly integrated by comparing their pre-computed transform coefficients against the existing index structure, rather than performing full re-constructions.
Data Source
AI summary
Incremental update of a neighbor graph via an orthogonal transform based indexing is disclosed. One example is a system including a hash transform module to apply an orthogonal transform to a data object in a data stream, and to associate the data object with a collection of ordered hash positions. An indexing module retrieves an index of ordered key positions, where each key position is indicative of data objects in the data stream that have a hash position at the key position. A neighbor determination module determines a ranked collection of neighbors for the data object in a neighbor graph, where the ranking is based on the index. A graph update module incrementally updates the neighbor graph by including the data object as a neighbor for a selected sub-plurality of data objects in the ranked collection.